Deep Reinforcement Learning for IRS-assisted Secure NOMA Transmissions Against Eavesdroppers

Physical layer security issues have attracted significant attention in wireless networks to protect information leakage from illegitimate eavesdroppers. In this paper, we focus on an intelligent reflecting surface (IRS)-assisted secure non-orthogonal multiple access (NOMA) uplink system. Multiple users intend to transmit sensitive data to an access point (AP) considering the existence of a nearby eavesdropper (Eve). The IRS can be used to enhance the NOMA users' sum rates while concurrently weakening the Eve's channel condition, suppressing information leakage to the Eve without resorting to a cooperative jammer or the power injection of artificial noise in the system. The users' scheduling, the IRS's passive beamforming, and the AP's receive beamforming are jointly optimized to maximize the secure rate of the IRS-assisted NOMA system. We develop a hierarchical deep reinforcement learning (DRL) framework to iteratively search for an optimal solution considering the combinatorial nature of the NOMA users' scheduling and the high-dimensional beamforming design. Firstly, we search for the NOMA users' scheduling strategy by using the PPO-based DRL algorithm. Given the NOMA scheduling strategy, we then optimize the active and passive beamforming strategies by the alternating optimization (AO) algorithm. The inner optimization helps evaluate the quality of the scheduling strategy and thus guides the outer-PPO algorithm to update a better scheduling strategy. Simulation results demonstrate the superiority of the proposed scheme over existing benchmarks, resulting in significant gains in secure rate.