Impact of action unit detection in automatic emotion recognition

In this paper, we investigate the interest of action unit (AU) detection for automatic emotion recognition. We propose and compare two emotion detectors: the first works directly on a high-dimensional feature space and the second projects facial image in the low-dimensional space of AU intensities before recognizing emotion. In both approaches, facial images are coded by local Gabor binary pattern (LGBP) histogram differences. These features reduce the sensitivity to subject identity by computing differences between two LGBP histograms: one computed on an expressive image and the other synthesized and approaching the one we would compute on a neutral face of the same subject. As classifiers, we test support vector machines with different kernels. A new kernel is proposed, the histogram difference intersection kernel that increases classification performances. This kernel is well suited when exploiting the proposed histogram differences. Thorough experiments on three challenging databases (respectively, the Cohn-Kanade, MMI and Bosphorus databases) show the accuracy of our AU and emotion detectors. They lead to significant conclusions on three critical issues: (1) the interest of combining different training databases labeled by different AU coders, (2) the influence of each AU according to its type and detection accuracy on emotion recognition and (3) the sensitivity to identity variations.