Unsupervised Pre-training on Improving the Performance of Neural Network in Regression

The paper aims to empirically analyse the performance of the prediction capability of Artificial Neural Network by applying a pre-training mechanism. The pre-training used here is same as the training of Deep Belief Network where the network is formed by stacking Restricted Boltzmann Machine one above the other successively. A different set of experiments are performed to understand in what scenario pre-trained ANN performed better than randomly initialised ANN. The results of experiments showed that pre-trained model performed better than randomly initialised ANN in terms of generalised error, computational units required and most importantly robust to change in hyperparameters such as learning rate and model architecture. The only cost is in additional time involved in the pre-training phase. Further, the learned knowledge in pre-training, which is stored as weights in ANN, are analysed using Hinton diagram. The analysis could provide clear picture of the pre-training that learned some of the hidden characteristics of the data.