Measurement issues in the alignment of standards and assessments: A case study

This study examined the impact of rater agreement on decisions concerning the alignment between the Golden State Examination in High School Mathematics (California Department of Education, 2001a) and the University of California (UC) Statement on Competencies in Mathematics Expected of Entering College Students (Academic Senate of the California State University, 1997). The UC faculty and high school mathematics teachers (N = 20) rated the mathematics items of the Golden State Examination relative to the expectations identified in the UC competency statement, identifying item features related to content and dimensionality. Raters assigned values for a primary topic, a secondary topic, item/topic centrality, and depth of knowledge. Agreement within these criteria was the basis of the assessment of alignment. Results showed considerable variability in judgments across raters and different pictures of alignment depending on the particular subset of raters providing the ratings. A few differences emerged between rater types.