Evaluating the quality of ratings obtained from standard-setting judges

The purpose of this study is to describe a new approach based on Rasch measurement theory for examining the quality of ratings obtained from standard-setting judges. This new approach is based on the adaptation and modification of several indicators of rating errors. To explore the usefulness of the Rasch measurement theory, the ratings of the nine judges who took part in the standard-setting process on a certification examination for nurses were examined. Ratings were obtained for 213 multiple-choice items on this examination. The data suggest that the judges vary significantly in their views of the essential items required for certification. Furthermore, the data suggest that there is statistically significant variability in the judged essentiality of these items.