Improving scalability of multi-agent reinforcement learning with parameters sharing

Improving the scalability of a multi-agent system is one of the key challenges for applying reinforcement learning to learn an effective policy. Parameter sharing is a common approach used to improve the efficiency of learning by reducing the volume of policy network parameters that need to be updated. However, sharing parameters also reduces the variance between agents' policies, which further restricts the diversity of their behaviors. In this paper, we introduce a policy parameter sharing approach, it maintains a policy network for each agent, and only updates one of them. The differentiated behavior of agents is maintained by the policy, while sharing parameters are updated through a soft way. Experiments in foraging scenarios demonstrate that our method can effectively improve the performance and also the scalability of the multi-agent systems.