Multi-Agent Q-Learning with Joint State Value Approximation

This paper relieves the "curse of dimensionality" problem, which becomes intractable when scaling reinforcement learning to multi-agent systems. This problem is aggravated exponentially as the number of agents increases, resulting in large memory requirement and slowness in learning speed. For cooperative systems which are widely existed in multi-agent systems, this paper proposes a new multi-agent Q-learning algorithm based on the decomposing the joint state and joint action learning into two learning processes, which are learning individual action and the maximum value of the joint state approximately. The latter process considers others' actions to insure the joint action is optimal and supports the updating of the former one. The simulation results illustrate that the proposed algorithm can learn the optimal joint behavior with smaller memory and faster speed comparing with Friend-Q learning.