A Comprehensive Approach to Early Detection of Workplace Stress with Multi-Modal Analysis and Explainable AI

This study introduces a novel framework for stress detection, leveraging the synergy of physiological signals and facial expressions through advanced machine learning techniques. Employing a suite of models including Long Short-Term Memory (LSTM) networks, Support Vector Machines (SVM), and Convolutional Neural Networks (CNNs) such as VGG16 and Custom CNN models, we undertake a comprehensive analysis across varied data durations. Our findings highlights the superiority of LSTM networks, which consistently outperform SVMs across metrics, particularly excelling in longer data sequences with notable improvements of 4 % in average in test accuracy, precision, recall, and F1 scores. This highlights the critical advantage of deep learning in capturing complex temporal patterns inherent in stress manifestations. Moreover, our exploration reveals that VGG16 surpasses custom CNNs, achieving a remarkable test accuracy of 87%, thereby setting a new standard in stress detection through facial expression analysis. This research not only advances the state of the art in stress classification but also underscores the transformative potential of multimodal data integration in understanding stress. By demonstrating significant improvements over existing methods [1, 4, 19], this work paves the way for innovative, AI-driven approaches to stress management, emphasizing the critical role of multimodal representations in enhancing the accuracy and reliability of stress detection systems. Also, we have harnessed the three Explainable AI tools (XAI), i.e., SHAP, LIME and Permutation Importance to illuminate the decision-making processes of complex AI models, aiding in the detection and reduction of biases. Through this pioneering effort, we contribute to the broader endeavor of improving mental health and well-being with technology.