An Extreme Learning Machine Based Pretraining Method for Multi-Layer Neural Networks

One approach in training a deep neural network to perform effectively is to do unsupervised pretraining on each layer, followed by fine-tuning the whole network. A common way is to train an unsupervised model of neural network such as restricted Boltzmann machines or autoencoders and stack them on top of another. Although these unsupervised pretraining approaches yield good performance, relying on back-propagation, due to iterative learning process, they still suffer from a long pretraining time. Extreme learning machine (ELM) is an analytical training approach which is extremely fast and gives a solution with a good generalization performance. In this paper, we apply a new ELM based unsupervised learning, named backward ELM based autoencoder (BELM-AE), to pretrain each layer of a neural network before using a back-propagation based learning algorithm to fine-tune the whole network. Experimental results show that the new pretraining method requires significantly shorter training time and also yields better testing performance on various datasets.