Multi-agent Actor-Critic Reinforcement Learning Based In-network Load Balance

Load balancing is a difficult online decision-making problem in the current network. Recently, with the development of the programmable data-plane, it is feasible to perform flexibly load balance directly inside the network. This in-network load balance scheme can quickly adapt to the volatility of network traffic. However, previous in-network solutions are largely relying on the manual process. Inspired by recent successes in applying machine learning in online control, automating the in-network load balance process is thus appealing. But as a distributed control system, it behooves us to ask the critical question: "Can the distributed switches learn globally optimal scheduling policy and still be deployed in a distributed fashion to allow rapid reaction in real-time?" To tackle this question, we adopt a centralized learning and distributed execution framework and propose a multi-agent actor-critic reinforcement learning algorithm in this paper. The centralized "critic" is reinforced with the global network state and joint actions of all agents to ease the training process whilst distributed switches can take actions relaying on their local observations. In addition, a baseline scheme is introduced to solve the credit assignment problem in the multi-agent system. The extensive simulations are conducted to evaluate our proposed algorithm in comparison to state-of-the-art schemes.