Decreasing variability in faculty ratings of student case presentations: a faculty development intervention focusing on reflective practice

Background: Faculty perceptions of medical student knowledge are often based on case presentations, but their assessment of student performance can vary significantly based on content, presentation skill, and interpersonal characteristics. Purpose: To determine if a brief faculty development intervention could significantly decrease (>= 20%)faculty members' variability in rating student case presentations in the ambulatory setting. Methods: During the 1998-99 academic year, we videotaped 10 third-year medical students during one well-child presentation. Seven general pediatric faculty rated the videotapes using a validated scoring instrument measuring content: History of Present Illness (HPI), Past Medical History, and Physical Exam (PE); and communication skill: Precision of Language, Fluency, and Economy. Baseline ratings were from June to September 1999 and then repeated in October to -November 2000 following a workshop where faculty reflected on the rating process and decreasing rating variance. The change in mean faculty ratings of the student's overall performance and for each item and the change in the mean difference in standard deviation before and after the workshop were analyzed. Results: Internal consistency of the scoring instrument using Cronbach's alpha was 0.88. The mean faculty rating for overall presentation performance and the PE did not significantly change after the workshop, whereas for all other items it did. The mean standard deviation of faculty ratings decreased significantly for the overall performance and all other items except HPI. Conclusion: Despite using a validated scoring instrument to rate student case presentations significant variability exists among faculty, with rating inflation likely. A brief faculty development intervention using "reflection -on-action" was associated with more congruent ratings in the short term.