Effects of comprehensive educational reforms on academic success in a diverse student body

CONTEXT Calls for medical curriculum reform and increased student diversity in the USA have seen mixed success: performance outcomes following curriculum revisions have been inconsistent and national matriculation of under-represented minority (URM) students has not met aspirations. Published innovations in curricula, academic support and pipeline programmes usually describe isolated interventions that fail to affect curriculum-level outcomes. METHODS United States Medical Licensing Examination (USMLE) Step 1 performance and graduation rates were analysed for three classes of medical students before (matriculated 1995-1997, n = 517) and after (matriculated 2003-2005, n = 597) implementing broad-based reforms in our education system. The changes in pipeline recruitment and preparation programmes, instructional methods, assessment systems, academic support and board preparation were based on sound educational principles and best practices. RESULTS Post-reform classes were diverse with respect to ethnicity (25.8% URM students), gender (51.8% female), and Medical College Admissions Test (MCAT) score (range 20-40; 24.1% scored < 25). Mean +/- standard deviation MCAT scores were minimally changed (from 27.2 +/- 4.7 to 27.8 +/- 3.6). The Step 1 failure rate decreased by 69.3% and mean score increased by 14.0 points (effect size: d = 0.67) overall. Improvements were greater among women (failure rate decreased by 78.9%, mean score increased by 15.6 points; d = 0.76) and URM students (failure rate decreased by 76.5%, mean score increased by 14.6 points; d = 0.74), especially African-American students (failure rate decreased by 93.6%, mean score increased by 20.8 points; d = 1.12). Step 1 scores increased across the entire MCAT range. Four- and 5-year graduation rates increased by 7.1% and 5.8%, respectively. CONCLUSIONS The effect sizes in these performance improvements surpassed those previously reported for isolated interventions in curriculum and student support. This success is likely to have resulted from the broad-based, mutually reinforcing nature of reforms in multiple components of the education system. The results suggest that a narrow reductionist view of educational programme reform is less likely to result in improved educational outcomes than a system perspective that addresses the coordinated functioning of multiple aspects of the academic enterprise.