Distributed Deep Neural Network Deployment for Smart Devices from the Edge to the Cloud

Traditionally, deep learning acceleration mostly focuses on the trade-off between accuracy and training time but seldom addresses the deployment over hierarchical 5G networks to maximize the inference throughput. By contrast, computing offloading research emphasizes whether to offload the tasks to the cloud to reduce computing time and achieve a lower response time, and thus, the optimal deployment to maximize throughput has not been explored. In this paper, we explore Distributed Deep Neural Network Deployment Problem with Constrained Completion Time (TREND-WANT) to solve the deployment problem considering both response time and inference throughput. Due to the intractability of TREND-WANT, we first design a new algorithm, named Stage-Time-Aware Layer Deployment Algorithm (STEED), to maximize the throughput. Afterward, an extension termed STEED with Adaptable Completion Time (STEED-ADAPT) is developed to tailor the solution to achieve a lower responsible time. Simulation results manifest our algorithms outperform the traditional methods by at least 200%.