Reliability and validity of a scoring instrument for clinical performance during Pediatric Advanced Life Support simulation scenarios

Aim: To assess the reliability and validity of scoring instruments designed to measure clinical performance during simulated resuscitations requiring the use of Pediatric Advanced Life Support (PALS) algorithms. Methods: Pediatric residents were invited to participate in an educational trial involving simulated resuscitations that employ PALS algorithms. Each subject participated in I session comprised of four scenarios (asystole, dysrhythmia, respiratory at-rest, shock). Video-recorded sessions were independently reviewed and scored by four raters using instruments designed to Measure performance in terms of timing, sequence, and quality. Validity was assessed by two-factor analysis of variance with postgraduate year (PGY-1 versus PGY-2) as all independent variable. Reliability was assessed by calculation of overall inter-rater reliability (IRR) as well as a generalizability study to estimate variance components of individual measurement facets (scenarios, raters) and associated interactions. Results: 20 subjects were scored by four raters. Based on a two-factor ANOVA, PGY-2s Outperformed PGY-1s (p < 0.05): significant differences in difficulty existed between the four scenarios, with dysrhythmia scores being the lowest. Overall IRR was high (0.81) and most variance Could be attributed to subject (17%), scenario (13%), and the interaction between Subject and scenario (52%): variance attributable to rater was minimal (1.4%). Conclusions: The instruments assessed in this Study measure clinical performance during PALS scenarios in a reliable and valid manner. Measurement error Could be minimized further through the use of additional scenarios but additional raters, fora given scenario, Would not improve reliability. Further Studies should assess validity of measurement with respect to actual clinical performance during resuscitations. (C) 2009 Elsevier Ireland Ltd. All rights reserved.