Recent Efficiency Gains in Deep Learning: Performance, Power, and Sustainability

Deep learning (DL) continues to develop at a rapid pace with improvements coming from both hardware and software sides. In this work we evaluate the strides made over the last 2 years, during which a new generation of GPU accelerators has been introduced and significant algorithmic progress has been made. We find a dramatic improvement in runtime and power usage for a standard AI training workload. Specifically, a 4x improvement in runtime and energy consumption is demonstrated. The improvements are about equally split between hardware and algorithms. Additionally, we examine further ways to improve AI training power consumption on data center servers and identify 3 system level tunings that make most difference. These yield up to 20% more energy savings without any changes to the user code. Implications for the field and ways to make DL more energy-efficient going forward are also discussed. (Abstract)