Improved binary classification performance using an information theoretic criterion

Feedforward neural networks trained to solve classification problems define an approximation of the conditional probabilities P(C-i\x) if the output units correspond to categories C-i. The present paper shows that if a least mean squared error cost function is minimised during training phase, the resulting approximation of the P(C-i\x)s is poor in the ranges of the input variable x where the conditional probabilities take on very low values. The use of the Kullback-Leibler distance measure is proposed to overcome this limitation; a cost function derived from this information theoretic measure is defined and a computationally light training procedure is derived in the case of binary classification problems. The effectiveness of the proposed procedure is verified by means of comparative experiments.