Performance Characterization of MPI_Allreduce in Cloud Data Center Networks

Advancements in hardware architecture and system design have enabled the transformational pivot towards cloud-computing. The rising cost of on-premise, vertical scaling, and maintenance combined with the rise of workload heterogeneity have fueled this paradigm shift. Furthermore, increasingly demanding use-cases for storage and High-Performance Computing (HPC) as well as the emergence of new workloads such as machine learning and big-data have motivated research into network traffic analysis, resource disaggregation, job scheduling, and orchestration in a bid to reduce total cost while maintaining high performance. A prime focus of this research is the communication performance of collectives, which comprise a significant portion of the communication of many of the aforementioned workloads. In this paper, we characterize the performance of MPI_Allreduce, which is used extensively in HPC and Deep-Learning (DL) training workloads in cloud environments. We demonstrate several key insights including that the Ring algorithm performs better than the Rabenseifner algorithm under conditions of high congestion and packet drops. We further illustrate detailed performance analysis under different assumptions and provide recommendations on how to address issues that can manifest.