Video-Based Emotion Recognition using Face Frontalization and Deep Spatiotemporal Feature

In this paper, we present the method used for the Multimodal Emotion Recognition Challenge (MEC) 2017 in the category of video-based emotion recognition. Our approach is based on two core ideas. First, to solve the problem of head-pose variations in video, we use the face frontalization approach, which is generally used in the field of face recognition, to synthesize the front view of the face in each frame through aligning the face to a 3D frontal model while preserving the facial expression information. Second, we use C3D, a deep 3-dimensional convolutional network that can model the appearance and motion of videos simultaneously, to extract spatiotemporal facial features from frontalized face sequences. We also use facial geometric features as a supplement. We tried different combinations of prediction scores output by softmax and linear SVM classifiers for different features to predict emotion. We tested our method on the Chinese Natural Audio-Visual Emotion Database (CHEAVD) 2.0. The experimental results show that our method achieves impressive results in terms of both accuracy and macro average precision, which significantly outperform the baseline.