The effect of patient care order sets on medical resident education: a prospective before-after study

Background: Patient care order sets are increasingly being used to optimize care. While studies have evaluated the impact of order sets on provider performance and patient outcomes, their impact on postgraduate medical trainee knowledge remains unknown. We sought to evaluate the impact of order sets on respirology knowledge, order-writing skills, and self-reported learning. Methods: We conducted a prospective before-after study. Postgraduate trainees completing a Respirology rotation at a quaternary-care hospital 6 months before (no order set period) and 12 months after (order set period) order set introduction. Guideline-based admission order sets with educational prompts detailing recommended management of cystic fibrosis and chronic obstructive pulmonary disease were implemented on the respirology ward. Each resident completed a test before and after the rotation assessing knowledge and order-writing. Residents in the order set period additionally completed a questionnaire regarding the impact of order set use on their learning. Analysis: The primary outcome, the difference between pre and post rotation scores was compared between residents in the no order set period and residents in the order set period, using univariate linear regression. Test validity was assessed with a 2-sample t-test, analysis of variance and Pearson's correlation coefficient. Self-reported impact of order set use were descriptively analyzed, and written responses were collated and coded. Results: Investigators consecutively recruited 11 subjects before and 28 subjects after order set implementation. Residents in the order set period had a greater improvement in post-rotation test scores than residents in the no order set period (p = 0.04); after adjustment for baseline scores, this was not significant (p = 0.3). The questionnaire demonstrated excellent convergent, discriminant and construct validity. Residents reported that order sets improved their knowledge and skills and provided a systematic approach to care. Conclusions: Order sets do not appear to impair resident education, and may impart a benefit. This will require validation in larger studies and across diseases.