Opportunities and Challenges Of Machine Learning Accelerators In Production

The rise of deep learning has resulted in tremendous demand for compute power, with the FLOPS required for leading machine learning (ML) research doubling roughly every 3.5 months since 2012 [1]. This increase in demand for compute has coincided with the end of Moore's Law [2]. As a result, major industry players such as NVIDIA, Intel, and Google have invested in ML accelerators that are purpose built for deep learning workloads. ML accelerators present many opportunities and challenges in production environments. This paper discusses some high level observations from experience internally at Google.