A Compression-Driven Training Framework for Embedded Deep Neural Networks

Deep neural networks (DNNs) are brain-inspired machine learning methods designed to recognize key patterns from raw data. State-of-the-art DNNs, even the simplest ones, require a huge amount of memory to store and retrieve data during computation. This prevents a practical mapping onto platforms with very limited resources, like those deployed in the end-nodes of the Internet-of-Things (IoT). The aim of this paper is to describe an efficient compression-driven training framework for embedded DNNs. The learning algorithm, which consists of a modified version of the Stochastic Gradient Descent, limits the original training space in a (-sigma, 0, sigma) ternarized subspace, with sigma a hyperparameter learned layer-wise. Tested on medium- and large-scale DNNs, both Convolutional Neural Networks (CNNs) and Recurrent Neural Networks (RNNs), the obtained sigma-ary DNNs enable an efficient use of sparse-matrix representations, hence high compression rate (up to 77x for CNN and 95x for RNNs), at the cost of a limited accuracy loss.