Generalized Correntropy Induced Loss Function for Deep Learning

Through multiple levels of abstraction, deep learning takes advantage of multiple layers models to find the complicated structure and learn the high level representations of data. In recent years, deep learning has made great progress in object detection, speech recognition, and many other domains. The robustness of learning systems with deep architectures is however rarely studied and needs further investigation. Especially, the mean square error(MSE), which is commonly used as optimization cost function in deep learning, is sensitive to outliers(or impulsive noises). To combat the harmful influences caused by outliers which are pervasive in many real world data, it is indispensable to improve the robustness in deep learning. In this paper, a robust deep learning method based on generalized correntropy is proposed and named generaliezed correntropy induced loss function(GC-loss) based SAE(GC-SAE). Generalized correntropy as a nonlinear measure of similarity is robust to outliers and can approximate different norms(from l(0) to l(2)) of data. By using generalized Gaussian density(GGD) function as its kernel, generalized correntropy achieves a more flexible shape and shows a better robustness for non-Gaussian noise when compared with the original correntropy with Gaussian kernel. The good robustness of the proposed method is confirmed by the experiments on MNIST benchmark dataset.