Fast learning in Deep Neural Networks

The paper aims at speeding up Deep Neural Networks (DNN) since this is one of the major bottlenecks in deep learning. This has been achieved by parameterizing the weight matrix using low rank factorization and periodic functions. By parameterization, the weight matrix is split into two matrices of smaller size of rank K with periodic functions. A shrinkage parameter has been introduced which helps in reducing the number of parameters and thus helps in increasing the speed to a great extent. Performance of the proposed parameterization is compared with standard DNN, DNN based on weight factorization alone and on periodic-bounded weights. This has been demonstrated on benchmark datasets MNIST and MNIST variants. (C) 2015 Elsevier B.V. All rights reserved.