Student engagement detection in online environment using computer vision and multi-dimensional feature fusion

In the post-COVID-19 era, online learning has changed from an emergency teaching method to a new, normalized one. However, online learning is often plagued by low participation and high dropout compared to offline learning. A critical way to address these issues is the accurate detection of student engagement, which will help teachers promptly assess learners’ status. Image data are one of the most straightforward ways to reflect student engagement levels. However, traditional engagement detection methods with images either rely on manual analysis or interfere with student behavior, which leads to a need for more objectivity in final engagement levels. This paper proposes a system that utilizes images obtained from individual webcams in online classrooms. Based on the techniques of multi-dimensional feature fusion and multimodal analysis, this system can rapidly detect and output students’ classroom engagement levels which provides real-time support for teachers to adjust their teaching methods during the teaching process, aiming to enhance students’ engagement in online courses. In the feature extraction module, VGG16 is utilized to recognize students’ facial expressions, ResNet-101 is designed to estimate head pose in each image, and Mediapipe is applied to estimate facial landmarks that reflect eye–mouth behavior. Subsequently, a BP neural network is constructed to fuse these multi-dimensional features and output the engagement level in each image in the data fusion module. The present method is evaluated on the wacv2016 data set and achieves an accuracy of 62.03%, outperforming the single-dimensional method. It is also applied in online courses to demonstrate its validity in the actual scenario further. Pearson Correlation between engagement levels calculated by our multi-dimensional fusion method and NSSE–China survey scores filled out by students is 0.714. It indicates that the method can enable real-time monitoring of students’ classroom engagement with similar results to traditional questionnaires with little human resources and time.