Evaluating the impact of a classroom simulator training on graduate teaching assistants' instructional practices and undergraduate student learning

In this study, we evaluate the impact of rehearsing teaching skills in a mixed-reality classroom simulator on graduate teaching assistants' (GTAs) instructional practices as well as undergraduate student learning outcomes. The simulator training is intended to provide GTAs opportunities to deliberately practice essential pedagogical skills that support active learning, specifically in the context of the combined tutorial and laboratory sections of an algebra-based introductory physics sequence. Over three semesters, GTAs participated in different numbers of simulator rehearsal sessions: no simulator training, one session, and four sessions. We conducted 109 classroom observations for 23 GTAs, using a modified version of the Laboratory Observation Protocol for Undergraduate STEM (LOPUS); we also documented the frequencies of questioning-related skills (e.g., cold calling) implemented by the GTAs. Undergraduate student learning outcomes were measured by pre- and posttests of the Force Concept Inventory (FCI) and Conceptual Survey of Electricity and Magnetism (CSEM). To classify and characterize GTAs' instructional practices, we conducted a hierarchical cluster analysis and found three instructional styles: the small-group facilitator, the whole-class facilitator, and the waiter. The results suggest that four-session simulator training throughout a semester supported GTAs (i) to shift away from the style of the waiter toward the whole-class facilitator, and (ii) to implement posing questions and cold calling techniques. While new GTAs were found to have more interactive behaviors than experienced GTAs in the semester with no simulator training, we found that four-session simulator training supported both new and experienced GTAs to use more interactive instructional styles and to implement questioning-related skills more frequently. Although the results demonstrate the effectiveness of simulator training, our analysis also indicates areas for improvement. GTAs tended to shift away from the style of the small-group facilitator toward the whole-class facilitator when they participated in four-session training, and the weekly implementations of questioning-related skills decreased over the course of a semester despite an increased total implementation. In addition, student learning outcomes in different semesters (with different numbers of simulator rehearsal sessions) did not show a statistically significant difference. However, GTAs' instructional styles were correlated with student performance on FCI posttest with a small effect size when controlling for FCI pretest scores and lecture instructors; no correlation was found between GTAs' instructional style and student performance on the CSEM posttest. We conclude with a discussion of factors that may have led to the success of the simulator training as well as strategies to further enhance the effectiveness of the simulator training.