Multimodal emotion recognition for the fusion of speech and EEG signals

To construct an effective emotion recognition system, the emotions of joy, sadness, anger and neutrality are induced by sound stimulation, and the corresponding speech and EEG signals are collected. First, this paper extracts the nonlinear geometric feature and nonlinear attribute feature of EEG and speech signals by phase space reconstruction respectively, and the emotion recognition is realized by combining the basic features. Then, a feature fusion algorithm based on the Restricted Boltzmann Machine is constructed to realize multimodal emotion recognition from the perspective of feature fusion. Finally, a multimodal emotion recognition system is constructed through decision fusion by using the quadratic decision algorithm. The results show that the overall recognition rate of the multimodal emotion recognition system constructed by feature fusion is 1.08% and 2.75% higher than that of speech signals and that of EEG signals respectively, and that the overall recognition rate of the multimodal emotion recognition system constructed by decision fusion is 6.52% and 8.19% higher than that of speech signals and that of EEG signals respectively. The overall recognition effect of the multimodal emotion recognition system based on decision fusion is better than that of feature fusion. A more effective emotion recognition system can be constructed by combining the emotional data of different channels such as speech signals and EEG signals. © 2019, The Editorial Board of Journal of Xidian University. All right reserved.