Reinforcement learning-based autonomous attacker to uncover computer network vulnerabilities

In today’s intricate information technology landscape, the escalating complexity of computer networks is accompanied by a myriad of malicious threats seeking to compromise network components. To address these security challenges, we propose an approach that synergizes reinforcement learning and deep neural networks. Our method involves training autonomous cyber-agents to strategically attack network nodes, aiming to expose vulnerabilities and extract confidential information. We employ various off-policy deep reinforcement learning algorithms, including deep Q-network (DQN), double DQN, and dueling DQN, to train and evaluate these agents within two enterprise simulation networks provided by Microsoft. The simulations, modeled as Markov games between attack and defense, exclude human intervention. Results demonstrate that agents trained by double DQN and dueling DQN surpass baseline agents trained using traditional reinforcement learning and DQN methods. This approach not only enhances our understanding of network vulnerabilities but also lays the groundwork for future efforts to fortify computer network defense and security. © The Author(s), under exclusive licence to Springer-Verlag London Ltd., part of Springer Nature 2024.