Deep facial spatiotemporal network for engagement prediction in online learning

Recently, online learning has been gradually accepted and approbated by the public. In this context, an effective prediction of students’ engagement can help teachers obtain timely feedback and make adaptive adjustments to meet learners’ needs. In this paper, we present a novel model called the Deep Facial Spatiotemporal Network (DFSTN) for engagement prediction. The model contains two modules: the pretrained SE-ResNet-50 (SENet), which is used for extracting facial spatial features, and the Long Short Term Memory (LSTM) Network with Global Attention (GALN), which is employed to generate an attentional hidden state. The training strategy of the model is different with changes of the performance metric. The DFSTN can capture facial spatial and temporal information, which is helpful for sensing the fine-grained engaged state and improving the engagement prediction performance. We evaluate the methods on the Dataset for Affective States in E-Environments (DAiSEE) and obtain an accuracy of 58.84% in four-class classification and a Mean Square Error (MSE) of 0.0422. The results show that our method outperforms many existing works in engagement prediction on DAiSEE. Additionally, the robustness of our method is also exhibited by experiments on the EmotiW-EP dataset.