Multichannel convolutional neural network for human emotion recognition from in-the-wild facial expressions

Facial emotions reflect the person’s moods and show the human affective state that is correlative with non-verbal intentions and behaviors. Despite the advances on computer vision techniques, capturing automatically the facial expressions in-the-wild remains a very difficult task. In this context, we propose a multichannel convolutional neural network based on the quality and the strengths of three well-known pre-trained models, namely VGG19, GoogleNet, and ResNet101. Indeed, the complementarity of the features extracted from the three models is exploited in order to form a more robust feature vector. During the training phase, a freezing weight is applied for each architecture. Then, the layers containing the most relevant information are marked, and the final feature descriptor for emotion prediction is thereafter defined by concatenating the obtained vectors. In fact, the three architectures have showed their efficiency severally in term of emotion recognition, and notably they do not err in the same images. The final vector, obtained by concatenating the features extracted from the different models, is fed to a support vector machine classifier in order to predict the final emotions. Extensive experiments have been conducted on four challenging datasets (JAFFE, CK+, FER2013 and SFEW_2.0) covering in-the-wild as well as in-the-laboratory facial expressions. The obtained results show that the suggested method is not only more accurate compared to each pre-trained CNN model but it also outperforms relevant state-of-the-art methods.