Facial micro-expression recognition based on the fusion of deep learning and enhanced optical flow

Micro-expression is a kind of split-second subtle expression which could not be controlled by the autonomic nervous system. Micro-expression indicates that a person is hiding his truly emotion consciously. Because the micro-expression is closely interrelated with lie detection, micro-expression recognition has various potential applications in many domains, such as the public security, the clinical medicine, the investigation and the interrogation. Because recognizing the micro-expression through human observation is very difficult, researchers focus on the automatic micro-expression recognition. This research proposed a novel algorithm for automatic micro-expression recognition which combined a deep multi-task convolutional network for detecting the facial landmarks and a fused deep convolutional network for estimating the optical flow features of the micro-expression. Firstly, this research employed the deep multi-task convolutional network to detect facial landmarks with the manifold related tasks and divided the facial region by utilizing these facial landmarks. Furthermore, a fused convolutional network was applied for extracting the optical flow features from the facial regions which contain the muscle changes when the micro-expression presents. Finally the enhanced optical flow was applied for refining the information of the features and these refined optical flow features were classified by Support Vector Machine classifier for recognizing the micro-expression. The result of experiments on two spontaneous micro-expression database demonstrated that the method proposed in this paper achieved good performance in micro-expression recognition.