Hierarchical Reinforcement Learning Based on Continuous Subgoal Space

Multi-agents are designed at different temporal levels to decomposed a complex task into several simple ones in hierarchical reinforcement learning. At the beginning of neural network training, the great changes of the low-level policy would cause the unstable transitions of the high-level one. In this paper, we propose a hierarchical policy combined with PPO and DDPG to deal with the simultaneous training of multi-agents. To create an end-to-end policy, neural networks are employed to extract scene features in both low-level and high-level policies. In the meanwhile, a novel internal reward function is designed to enhance the goal achieving ability of low-level policy. A lightweight and fast gridworld Gym environment, MiniGrid, is used to test its validity. We found that the hierarchical policy is able to explore and plan without dense rewards. This attribute has a considerable influence on the study of robot navigation, especially in large and complex environment.