Decoding Action Observation Using Complex Brain Networks from Simultaneously Recorded EEG-fNIRS Signals

In the current study, a novel brain-machine interaction was proposed, which incorporates action observation decoding into the traditional control circuit of a brain-machine interface. In this new brain-machine interaction, the machine can actively decode the user's action observation and stop immediately if it detects that the user does not understand the intention of the action correctly. We measured brain activation using electroencephalography (EEG)-functional near-infrared spectroscopy (fNIRS) bimodal measurement while 16 healthy participants observed three action tasks: drinking, moving a cup, and action with unclear intention. Complex brain networks were constructed for EEG and fNIRS data separately, and four network measures were chosen as features for classification. The obtained results revealed that the classification of three action observation tasks achieved accuracy of 72.3% using EEG-fNIRS confusion features, which was higher than that using fNIRS features (52.7%) or EEG features (68.6%) alone. Thus, the current findings suggested that our proposed method could provide a promising direction for brain-machine interface systems design.