Alleviating Imbalance in Synchronous Distributed Training of Deep Neural Networks

Deep neural networks achieve huge success in computer vision, natural language process and other fields. As larger networks and larger datasets bring better performance, the training time grows rapidly. Distributed training uses parallelization strategy to accelerate the training of deep neural network. In the existing distributed training framework of deep neural network, the parameter server architecture has poor scalability and is not suitable for large-scale distributed training. Ring/HD All Reduce algorithm perform well in scalability under ideal circumstances. However, in many realistic scenarios, imbalance will occur frequently in the distributed training system, which is mainly the imbalance of compute ability and network. Due to the structure characteristics of synchronous distributed training and All Reduce algorithm, imbalance will bring serious impact on performance. At present, there is a lack of research on both compute ability imbalance and network imbalance, especially dynamic imbalance. To alleviate this problem, we analyze All Reduce algorithm first, and propose RingDL, which optimizes the performance of distributed training system by alleviating imbalance through customizable dynamic workloads strategy and link strategy. The experimental results show that RingDL significantly reduces idle time in computing phase and synchronous phase, improving the overall performance.