Sparse Representations in Deep Learning for Noise-Robust Digit Classification

Many sparse regularization methods for encouraging succinct hierarchical features of deep architectures have been proposed, but there is still a lack of studies that compare them. We present a comparison of several sparse regularization methods in deep learning with respect to the performance of a noisy digit classification task under varying size of training samples. We also propose a deep hybrid architecture built from a particular combination of sparse auto-encoders and Restricted Boltzmann Machines. The results show that the sparse architectures can produce better classification performance under noisy test samples than the dense architectures in most cases. In addition, the deep hybrid architectures can solve the digit classification task more effectively with a small size of training samples.