Facial expression recognition based on Electroencephalogram and facial landmark localization

BACKGROUND: Facial expression recognition plays an essential role in affective computing, mental illness diagnosis and rehabilitation. Therefore, facial expression recognition has attracted more and more attention over the years. OBJECTIVE: The goal of this paper was to improve the accuracy of the Electroencephalogram (EEG)-based facial expression recognition. METHODS: In this paper, we proposed a fusion facial expression recognition method based on EEG and facial landmark localization. The EEG signal processing and facial landmark localization are the two key parts. The raw EEG signals is pre-processed by discrete wavelet transform (DWT). The energy feature vector is composed of energy features of the reconstructed signal. For facial landmark localization, images of the subjects' facial expression are processed by facial landmark localization, and the facial features are calculated by landmarks of essence. In this research, we fused the energy feature vector and facial feature vector, and classified the fusion feature vector with the support vector machine (SVM). RESULTS: From the experiments, we found that the accuracy of facial expression recognition was increased 4.16% by fusion method (86.94 +/- 4.35%) than EEG-based facial expression recognition (82.78 +/- 5.78%). CONCLUSION: The proposed method obtain a higher accuracy and a stronger generalization capability.