Cooperative Multi-agent Reinforcement Learning with Hierachical Communication Architecture

Communication is an essential way for multi-agent system to coordinate. By sharing local observations and intentions via communication channel, agents can better deal with dynamic environment and thus make optimal decisions. However, restricted by the limited communication channel, agents have to leverage less communication resources to transmit more informative messages. In this article, we propose a two-level hierarchical multi-agent reinforcement learning algorithm which utilizes different timescales in different levels. Communication happens only between high levels at a coarser time scale to generate sub-goals which convey the intention of agents for the low level. And the low level is responsible for implementing these sub-goals by controlling primitive actions at every tick of environment. Sub-goal is the core of this hierachical communication architecture which requires the high level to communicate efficiently and provide guidance for the low level to coordinate. This hierarchical communication architecture conveys several benefits: 1) It coarsens the collaborative granularity and reduces the requirement of communication since communication happens only in high level at a larger scale; 2) It enables the high level to focus on the coordination of goals without paying attention to implementation, thus improves the efficiency of communication; and 3) It makes better control by dividing a complex multi-agent cooperative task into multiple single-agent tasks. In experiments, we apply our approach in vehicle collision avoidance tasks and achieve better performance than baselines.