Hardware accelerator systems for embedded systems

This chapter describes various engineering considerations and constraints to deploy neural network applications in embedded systems and presents a variety of processing solutions to accelerate neural network computations in the embedded hardware. Deep learning on embedded systems has potentially many advantages for security, privacy, latency, energy, power, etc. However, deploying the deep neural networks in embedded systems imposes numerous hardware challenges on the resource-limited embedded edge devices. Embedded systems for deep learning typically target on providing rapid inferences, and thus latency rather than throughput in general becomes the primary objective for the executions of embedded hardware. The central point of hardware acceleration in embedded systems is to place neural network computations closer to I/Os and sensors to provide fast inferences. With continued advances in processor technologies, embedded edge devices evolve to become capable of handling compute-intensive workloads at low power. Such a trend propels integrating the hardware acceleration of deep neural networks into the embedded systems. There is not a universal solution for all different kinds of embedded systems. Different embedded processing solutions can be employed to accelerate the neural network applications depending on their performance requirements, operating conditions (e.g., network connectivity, power and thermal constraints), costs, etc. These considerations leave a wide range of hardware options for the embedded systems. The embedded hardware to accelerate neural network applications ranges from single-board devices such as Google Edge TPU to high-performance processors such as Intel Xeon and AMD EPYC CPUs, NVIDIA GPUs with Tensor Cores. All of these hardware choices for the neural network acceleration provide distinct features and computational capabilities in the embedded systems.