FTPSG: Feature mixture transformer and potential-based subgoal generation for hierarchical multi-agent reinforcement learning

Hierarchical multi-agent reinforcement learning (HMAR) presents a promising approach for addressing complex multi-agent tasks. However, HMAR faces the challenge of identifying potential states or skills-subgoals that agents can efficiently solve. Our paper introduces a novel approach to subgoal generation within HMAR in response to learning signals in sparse delayed reward environments. We propose a Feature Mixture Transformer and Potential-based Subgoal Generation (FTPSG) as an efficient method for automatically generating promising subgoals by extracting and combining relevant features across past observations within a trajectory. Also, FTPSG utilizes a potential function to assess the probability of each subgoal leading agents to the ultimate goal. We design our potential function to rank these subgoals to achieve an actual goal and provide meaningful learning signals. Subgoals are then grouped based on their potential, prioritizing those with high potential as more crucial. This grouping enables agents to concentrate on the most important subgoals initially. We investigate the effectiveness of our proposed method across various multi-agent tasks, and the results consistently show that FTPSG outperforms state-of-the-art methods across all evaluated tasks. These findings affirm FTPSG's promising role in subgoal generation within the HMAR framework.