On automatic job interview assessement possibilities using facial expressions analysis

The purpose of this study is to show the possibility of using facial expression analysis for remote job interviews automated assessment. The design of the study involves comparing the job interviews automatic evaluation results with the corresponding estimates of HR specialists. Automatic job interviews evaluation was based on mimic activity (MA) indicators developed by the authors. The general methodology of the study is based on the links between emotions and facial expressions (Rosenberg, Ekman, 2020) and further transition from the analysis of emotions to the understanding of personal meanings, from the understanding of personal meanings to the assessment of a person answering interview questions (A. N. Leontiev, 1976, D. A. Leontiev, 1999, Asmolov, 2007). The special methodology is based on the approach developed by M.S. Baev and A.N. Gusev for the FACS AUs analysis in video recordings for emotional states evaluation. Employees of large Russian companies (413 men and 242 women, average age - 42.6 years) remotely underwent a structured 6-16 questions video interview. Results. 4038 videos were selected for the analysis. MA analysis was performed using the EmoRadar WR 5.0 software. Individual AUs, basic emotions and MA patterns had been detected. Based on the criteria proposed by HR specialists, six rules for MA automatic analysis of were developed. These rules were based on the combinations of different AUs patterns and characterized the respondent behavior while answering an interview question as corresponding (inconsistent) with the expert's expectation. 60 videos were selected for comparative analysis. These videos were evaluated by six experts on the following scales: engagement, stress, confidence, lightness, activity, strength. The results obtained indicate a good agreement between the results of the video recordings automated MA based classification and expert assessments. Implications for practice. Thus, the possibility of practical use of the original technology of automatic MA analysis for evaluating of a job video interview has been confirmed. Our findings expand the instrumental capabilities of HR specialists when working with a large amounts of video interviews.