Convolutional Neural Network for Hybrid fNIRS-EEG Mental Workload Classification

The classification of workload memory tasks based on fNIRS and EEG signals requires solving high-dimensional pattern classification problems with a relatively small number of training patterns. In the use of conventional machine learning algorithms, feature selection is a fundamental difficulty given the large number of possible features and the small amount of available data. In this study, we bypass the challenges of feature selection and investigate the use of Convolutional Neural Networks (CNNs) for classifying workload memory tasks. CNNs are well suited for learning from the raw data without any a priori feature selection. CNNs take as input two-dimensional images, which differ in structure from the neural time series obtained on the scalp surface using EEG and fNIRS. Therefore, both the existing CNN architectures and fNIRS-EEG input must be adapted to allow fNIRS-EEG input to a CNN. In this work, we describe this adaptation, evaluate the performance of CNN classification of mental workload tasks. This study makes use of an open-source meta-dataset collected at the Technische Universitat Berlin; including simultaneous EEG and fNIRS recordings of 26 healthy participants during n-back tests. A CNN with three convolution layers and two fully connected layers is adapted to suit the given dataset. ReLU and ELU activation functions are employed to take advantage of their better dampening property in the vanishing gradient problem, fast convergence, and higher accuracy. The results achieved with the two activation functions are compared to select the best performing function. The proposed CNN approach achieves a considerable average improvement relative to conventional methods such as Support Vector Machines. The results across differences in time window length, activation functions, and other hyperparameters are benchmarked for each task. The best result is obtained with a three-second window and the ELU activation function, for which the CNN yields 89% correct classification, while the SVM achieves only 82% correct classification.