User Association and Power Allocation for User-Centric Smart-Duplex Networks via Tree-Structured Deep Reinforcement Learning

This article considers a smart-duplex (SD) powered user-centric ultra dense networks (UC-UDNs), where each user is served cooperatively by multiple access points (APs) adopting the de-cellular concept to achieve desired Quality-of-Service (QoS). The average QoS satisfaction ratio maximization problem for the considered SD UC-UDN is formulated as a Markov decision process (MDP) with large discrete action space by designing the user association and power allocation. To reduce the action space, user association and power allocation are modeled as a two-layer tree, and selecting an action for each user is equivalent to finding the path from the root to one leaf of the constructed tree. Then, a multiagent tree-structured policy gradient (MATSPG)-based deep reinforcement learning (DRL) algorithm is proposed to solve the MDP problem, whose training process is shown to be equivalent to that of the two-layer neural networks. Next, the time and space complexity of searching one action in the proposed MATSPG are also proved to be lower than the conventional DRL algorithms. Finally, simulations show that the proposed MATSPG algorithm significantly improves the average QoS satisfaction ratio than the conventional multiagent deep deterministic policy gradient and multiagent deep Q-network methods in typical scenarios.