Queen's Simulation Assessment Tool Development and Validation of an Assessment Tool for Resuscitation Objective Structured Clinical Examination Stations in Emergency Medicine

Introduction: The use of high-fidelity simulation is emerging as an effective approach to competency-based assessment in medical education. We aimed to develop and validate a modifiable anchored global assessment scoring tool for simulation-based Objective Structured Clinical Examinations (OSCEs) of resuscitation competence in postgraduate emergency medicine (EM) trainees. Methods: The Queen's Simulation Assessment Tool was developed using a modified Delphi technique with a panel of EM physicians. Ten standardized resuscitation OSCE scenarios were administered to EM trainees, and their video-recorded performances were scored by 3 independent and blinded EM attending physicians using the Queen's Simulation Assessment Tool. Correlational analyses and analysis of variance were applied to measure the discriminatory capabilities and interrater reliability of each scenario. A fully crossed generalizability study was conducted for each examination. Results: Emergency medicine postgraduate trainees at Queen's University (n = 19-25 per station) participated in the study over 3 years. Interrater reliability showed acceptable levels of agreement for each scenario (mean Spearman rho = 0.75 [0.63-0.87]; mean interclass correlation coefficient, 0.69 [0.58-0.87]). Discriminatory validity was strong, with senior residents outperforming junior residents in all but 1 of the 10 scenarios. Generalizability studies found the trainee and trainee by scenario interactions as the largest contributors to variance, with G coefficients ranging from 0.67 to 0.84. Resident trainees reported comfort being assessed in the simulation environment (3.8/5) and found the simulation-based examination valuable to their learning (4.6/5). Conclusions: This study describes the development and validation of a novel modifiable anchored global assessment scoring tool for simulation-based OSCE assessment of resuscitation competence in postgraduate EM trainees.