Risk-aware deep reinforcement learning for mapless navigation of unmanned surface vehicles in uncertain and congested environments

This paper addresses the navigation problem of Unmanned Surface Vehicles (USVs) in uncertain and congested environments. While previous research has extensively explored USV navigation, most approaches assume that the environmental maps and obstacle locations are pre-known to the USVs. In this paper, we focus on a sensor-level navigation approach that utilizes real-time LiDAR data integrated with deep reinforcement learning (DRL) for decision-making. To tackle sparse reward challenges, we propose a potential-based reward-shaping (PRS) module to regulate navigation behavior, and this module helps to improve the training efficiency of the twin delayed deep deterministic policy gradient (TD3) algorithm. Moreover, we introduce a risk evaluation and correction (REC) module to mitigate potential risks. This module employs a risk evaluation network to enhance the agent's risk awareness and an action-level correction mechanism to avoid unsafe behavior. The proposed approach is validated through ablation studies and comparative experiments in OpenAI Gym-based environments and simulated island regions of Zhoushan. The results indicate that the proposed approach significantly improves training efficiency while maintaining consistency and robustness in unknown and congested marine environments.