Deep Reinforcement Learning Based Scheduling Scheme for the NR-U/WiGig Coexistence in Unlicensed mmWave Bands

This paper considers the coexistence of the New Radio-based access to unlicensed spectrum (NR-U) network and the Wireless Gigabit (WiGig) network in unlicensed millimeter-wave (mmWave) bands. We aim to design a new scheduling scheme for the NR-U network to maximize its total data rate while satisfying the quality of service (QoS) requirement for each user equipment (UE). Specifically, we first formulate this problem into the constrained Markov decision process (CMDP) framework. Then the Lagrangian duality method is applied to relax the hard constraints in CMDP into the soft constraints. To address the multi-constraint issue, we put forth a new deep reinforcement learning (DRL) algorithm that incorporates the constraints into the DRL framework, referred to as adaptive multi-constraint deep Q-network (AMC-DQN). A prominent advantage of AMC-DQN is that it enables the NR-U network to access the shared spectrum without acquiring prior information about theWiGig network. Simulation results show that compared with the omnidirectional listen-before-talk (omniLBT) and directional LBT (dirLBT), the AMC-DQN based scheduling scheme yields the total data rate gain of the NR-U network by 158% and 38%, respectively. The results also demonstrate the ability of AMC-DQN to satisfy the QoS requirements of different UEs. Furthermore, AMC-DQN brings less interference to the WiGig network in comparison to baselines.