Coordinated multi-agent hierarchical deep reinforcement learning to solve multi-trip vehicle routing problems with soft time windows

Vehicle Routing Problem (VRP) is a widespread problem in the transportation field, which challenges the intelligent level of vehicle decisions. Multi-Trip Vehicle Routing Problem with Time Windows (MTVRPTW), as a further evolved problem of VRP considering multiple departures from one depot and temporal constraint of visiting nodes, has developed into one of the critical issues in the scheduling of logistics, bus transit, railway, and aviation. Traditionally, MTVRPTW is solved by the heuristic algorithm, which is generally time-consuming and of non-steady results. Reinforcement learning (RL) and multi-agent framework have become popular in solving VRP to get better performance. However, the lack of variant dimensions in searching space and knowledge exchange between agents inhibit the further improvement of algorithms. Therefore, a Coordinated Multi-agent Hierarchical Deep Reinforcement Learning (CMA-HDRL) method is proposed in this study to enhance the overall solution quality and convergence rate by constructing a three-layered structure (time, communication, and global layers), which is particularly designed to handle the state space explosion and improve the collaboration between agents. The results show that the proposed method can significantly outperform the general genetic algorithm (GA), RL, multi-agent algorithm, and hierarchical algorithm, not only from the effectiveness on the cost consisting of travel time and penalty time but also from the operation robustness.