Fusion of forehead EEG with machine vision for real-time fatigue detection in an automatic processing pipeline

Driving fatigue is a leading contributor to traffic accidents and fatalities. For automatic detection of fatigue, multimodal data fusion is a potential key technique, especially the merging of electroencephalogram (EEG) and eye movement. Although EEG can produce objective measures of fatigue with very high temporal resolution, an unfriendly multichannel system limits its practical application while portable wearables and machine vision receive much attention since they are pervasive and user friendly. Hence, this study aims to construct a novel pipeline by using machine vision technique to improve the quality of driver fatigue detection based on forehead EEG. By coupling the Karolinska Sleepiness Scale (KSS) and percentage of eyelid closure (PERCLOS), the precise and reliable dataset for reflecting drivers' fatigue levels were obtained. Moreover, major artifact contamination related to blink activity for frontal-channel EEG was removed by a synchronous video-based eyeblink event marker. In addition, the scale-invariant feature transform (SIFT) features of eyelid keypoints was applied to fuse with the EEG-driven features. Sixteen subjects participated in a realistic driving simulation experiment. Both machine and deep learning methods were used to implement the intra-subject and inter-subject cross validation. It demonstrated that our proposed method can achieve a significant performance. The present work showed the usefulness of forehead EEG and eye images that was originally merged for detecting fatigue. It provides a new strategy of using forehead EEG combined with machine vision to design a potential and automatic pipeline for driver fatigue detection.