Discriminative Feature Extraction with Deep Neural Networks

We propose a framework for optimizing Deep Neural Networks (DNN) with the objective of learning low-dimensional discriminative features from high-dimensional complex patterns. In a two-stage process that effectively implements a Nonlinear Discriminant Analysis (NDA), we first pretrain a DNN using stochastic optimization, partly supervised and unsupervised. This stage involves layer-wise training and stacking of single Restricted Boltzmann Machines (RBM). The second stage performs fine-tuning of the DNN using a modified back-propagation algorithm that directly optimizes a Fisher criterion in the feature space spanned by the units of the last hidden-layer of the network. Our experimental results show that the features learned by a DNN using the proposed framework greatly facilitate classification, even when the discriminative features constitute a substantial dimension reduction.