Bias Estimation Correction in Multi-Agent Reinforcement Learning for Mixed Cooperative-Competitive Environments

Multi-agent reinforcement learning (MARL) is a domain that is being actively researched in the current times. The ability of MARL algorithms in finding promising solutions to problems while having limited prior knowledge of the environment has found application in traffic control, unmanned vehicles, routing, and more. Despite the algorithmic advancements over the years, the issue of bias estimation still persists in MARL methods. Multi-agent twin delayed deep deterministic (MATD3) method solves for bias overestimation problem that arises in Multi-agent deep deterministic policy gradient (MADDPG). MATD3 does so by using a double-critic architecture. MATD3 calculates the target values and updates the critic networks by taking a minimum of the Q-values estimated by the target Q-networks. The method solves the overestimation problem but in the process introduces an underestimation. A method is proposed in this paper to correct for both bias over-estimation and under-estimation problem by incorporating a triple critic architecture. The target values are then calculated using a weighted sum of the minimum and the median of the target Q-values. The proposed method is tested in the multi-agent particle environment (MPE). Experimental results show that our method, apart from correcting for the bias estimation, outperforms MATD3 and MADDPG methods in terms of overall returns. We also perform experiments to test the effect of a weighted critic update on our proposed method and have found propitious results. © 2023, The Author(s), under exclusive licence to Springer Nature Singapore Pte Ltd.