Observational Learning Algorithm for an ensemble of neural networks

We propose Observational Learning Algorithm (OLA), an ensemble learning algorithm with T and O steps alternating. In the T-step, an ensemble of networks is trained with a training data set. In the O-step, 'virtual' data are generated in which each target pattern is determined by observing the member networks' output for the input pattern. These virtual data are added to the training data and the two steps are repeatedly executed. The virtual data was found to play the role of a regularisation term as well as that of temporary hints having the auxiliary information regarding the target function extracted from the ensemble. From numerical experiments involving both regression and classification problems, the OLA was shown to provide better generalisation performance than simple committee, boosting and bagging approaches, when insufficient and noisy training data are given. We examined the characteristics of the OLA in terms of ensemble diversity and robustness to noise variance. The OLA was found to balance between ensemble diversity and the average error of individual networks, and to be robust to the variance of noise distribution. Also, OLA was applied to five real world problems from the UCI repository, and its performance was compared with bagging and boosting methods.