HCS-R-HER: Hierarchical reinforcement learning based on cross subtasks rainbow hindsight experience replay

Sparse reward feedback from the environment is the main challenge for learning goal-oriented tasks based on reinforcement learning. The lack of sufficient exploration also leads to the inability of the agent to robustly learn strategies, especially for hierarchical task control of continuous action space continuum robots, which are more difficult to explore. In this paper, we propose a hierarchical reinforcement learning framework, HCS-R-HER, to accelerate learning by reusing empirical data across subtasks. It uses an upper-level controller, meta-controller, to integrate the underlying targets, and a set of lower-level controllers, controllers, responsible for performing atomic operations. The Oracle perspective mechanism can skip the process of unfinished subtasks, which helps speed up the learning of the meta-controller. The CS-R-HER framework is used to improve the sparsity of the data and accelerate the learning of controllers. Our approach can solve complex tasks or hierarchical tasks more effectively, especially for continuum robot motion environments in continuous action space. Our method is the first time to apply HER to data augmentation for hierarchical tasks and to implement a framework where multiple subgoals are learned together.