A hierarchical multi-agent allocation-action learning framework for multi-subtask games

Great progress has been made in the domain of multi-agent reinforcement learning in recent years. Most work concentrates on solving a single task by learning the cooperative behaviors of agents. However, many real-world problems are normally composed of a set of subtasks in which the execution order follows a certain procedure. Cooperative behaviors should be learned on the premise that agents are first allocated to those subtasks. In this paper, we propose a hierarchical framework for learning the dynamic allocation of agents among subtasks, as well as cooperative behaviors. We design networks corresponding to agents and subnetworks, respectively, which constitute the whole hierarchical network. For the upper layer, a novel allocation learning mechanism is devised to map an agent network to a subtask network. Each agent network could only be assigned to only one subtask network at each time step. For the lower layer, an action learning module is designed to compute appropriate actions for each agent with the allocation result. The agent networks together with the subtask networks are updated by a common reward obtained from the environment. To evaluate the effectiveness of our framework, we conduct experiments in two challenging environments, i.e., Google Research Football and SAVETHECITY. Empirical results show that our framework achieves much better performance than other recent methods.