PAARD: Proximity-Aware All-Reduce Communication for Dragonfly Networks

The all-reduce operation is one of the most commonly used collective communication operations, which is widely used in the research and engineering fields of high-performance computing(HPC) and distributed machine learning(DML). Previous optimization work for all-reduce operation is to design new algorithms only for different message size and different number of processors, and ignores the optimization that can be achieved by considering the topology. Dragonfly is a popular topology for current and future high-speed interconnection network. The hierarchical characteristics of dragonfly topology can be utilized to effectively reduce the hardware overhead while ensuring low end-to-end transmission latency. In this paper we propose PAARD, Proximity-Aware All-Reduce Communication on Dragonfly Networks. According to the characteristics of dragonfly topology, PAARD proposes an end-to-end solution to alleviate the congestion which could remarkably boost the performance. We carefully design the algorithm of PAARD to ensure desirable performance with acceptable overhead. To illustrate the effectiveness of PAARD, we analyze the performance of PAARD with the state-of-the-art algorithm, Halving-doubling(HD) algorithm and Ring algorithm. The simulation results demonstrate that in our design the completion time can be reduced up by 75.73 % for HD algorithm and 98.63% for Ring algorithm.