A L1-constrained DNN Acceleration Algorithm with ADMM

When facing high dimension and low sample sized data, such as image classification problem using manually labeled realistic images, deep learning network suffers from overfitting and may fail to fit additional data or predict future observations. L-1-constrained Deep Neural Network (DNN) is widely used to avoid overfitting problem and has achieved breakthroughs in applications. However, the non-linear and non-convex characters of L-1-constrained DNN are obstacles to rapidly converge to global optimum. In this paper, a non-gradient-based approach, called B-ADMM, is proposed to train L-1-constrained DNN. It decomposes the objective function into a bunch of mini-steps and updates parameters alternatively. To avoid local minima, B-ADMM solves each sub-step seperately and achieves global optimal solution. Experiments show that the convergence of B-ADMM is faster than SGD, Adam and RIVISprop while accuracy maintains to be the same. With a restrained L-1-regularizer, the network also achieves a sparse architecture.