A NOVEL LAYERWISE PRUNING METHOD FOR MODEL REDUCTION OF FULLY CONNECTED DEEP NEURAL NETWORKS

Deep neural networks (DNN) are powerful models for many pattern recognition tasks, yet they tend to have many layers and many neurons resulting in a high computational complexity. This limits their application to high-performance computing platfonns. In order to evaluate a trained DNN on a lowerperformance computing platfonn like a mobile or embedded device, model reduction techniques which shrink the network size and reduce the number of parameters without considerable perfonnance degradation perfonnance are highly desirable. In this paper, we start with a trained fully connected DNN and show how to reduce the network complexity by a novellayerwise pruning method. We show that if some neurons are pruned and the remaining parameters (weights and biases) are adapted correspondingly to correct the errors introduced by pruning, the model reduction can be done almost without performance loss. The main contribution of our pruning method is a closed-fonn solution that only makes use of the first and second order moments of the layer outputs and, therefore, only needs unlabeled data. Using three benchmark datasets, we compare our pruning method with the low-rank approximation approach.