Driver Emotion Recognition Involving Multimodal Signals: Electrophysiological Response, Nasal-Tip Temperature, and Vehicle Behavior

Accurate driver emotion recognition is one of the key challenges in the construction of an intelligent vehicle safety assistant system. In this paper, we conduct a driving simulator study on driver emotion recognition. Taking the car-following scene as an example, the multimodal parameters of a driver in the five emotional states of neutral, joy, fear, sadness, and anger are obtained from the emotion induction experiment and the simulated driving experiment. Wavelet denoising and debase processing are used to reduce the influence of signal noise and the individual differences between drivers. The statistical domain and the time-frequency domain features of the electrophysiological response signals, nasal-tip temperature signals, and vehicle behavior signals are analyzed. The factor analysis method is used to extract and reduce the feature parameters, and the driver's emotion recognition model is established based on machine learning methods such as random forest (RF), K-nearest-neighbor (KNN), and extreme gradient boosting (XGBoost). Through the verification and the comparison of different modalities and different modality combinations with different machine learning methods, the RF model, based on the feature combination of three types of modal data, has the best model recognition effect. The research results can provide a theoretical basis for driver emotion recognition of intelligent vehicles and have positive significance for promoting the development of human-computer interaction (HCI) systems of intelligent vehicles and improving road traffic safety.