Toward Scalable and Efficient Hierarchical Deep Reinforcement Learning for 5G RAN Slicing

As an emerging and promising network paradigm, network slicing creates multiple logical networks on shared infrastructure to provide services with customized Quality-of-Service (QoS) for heterogeneous devices and applications. However, network complexity and service heterogeneity pose a huge challenge in achieving optimal performance and ensuring stringent QoS requirements. In this paper, we design a hierarchical deep reinforcement learning based 5G radio access network slicing framework to achieve scalable and efficient resource allocation. By decomposing the resource allocation problem into a slice-level task and several user-level tasks, the proposed framework tackles each task with an agent, thus conquering insufficient exploration and achieving scalable management. Knowledge transfer and progressive learning are employed to improve training efficiency and stability, respectively. We apply collaborative training to eliminate distribution mismatch by refining value approximators and policies of agents alternately. Extensive experiments show that the proposed framework can learn effective resource allocation policies stably and efficiently and outperform other methods in network utility maximization and QoS assurance, which improves the network utility by 25% and 8% compared with the random strategy and the ADMM strategy, respectively. Furthermore, we validate that our framework is more robust to changes in network traffic conditions including network congestion.