Student learning in secondary school mathematics classrooms: On the validity of student reports in international large-scale studies

Public availability, high data quality, and sample characteristics make data from international large-scale assessments (ILSA) attractive for secondary analyses, and more and more researcher use them to analyze teaching and learning in students' classrooms. However, for several reasons ILSA data is limited in its scope, and conclusions on educational effectiveness are restricted. This study specifically intends to evaluate the potential danger of overestimating the effectiveness of indicators of teaching and learning based on ILSA data. We take advantage of two extensions to the German sample for the Programme for International Student Assessment (PISA) 2012 study: The sample was enhanced a) by adding students from entire 9th grade classrooms and b) by a repeated measurement, re-testing all participants one year later in 10th grade. This study analyzed eight indicators of students' reports on teaching and learning in their mathematics classrooms. Regression analyses in multi-level structural equation models (doubly-latent models) compared results of cross-sectional analyses (controlling for individual background and classroom level characteristics) with a longitudinal extension (controlling for prior achievement). Results only partially confirm common practice to use ILSA data to study educational effectiveness of classroom level processes. Cross-sectional analyses overestimate effectiveness results, but three out of four indicators that are cross-sectionally related to student achievement are also related to students' achievement development in grade 10. Assuming that previous studies modeled and interpreted the data correctly, the importance of their results in debates on educational effectiveness is thus underlined.