Evaluation of ratings: Psychometric quality assurance via many-facet Rasch measurement

Building on the many-facet Rasch measurement model (Linacre, 1989; Linacre & Wright, 2002), this paper presents a general framework of statistical procedures suitable for a detailed analysis of the psychometric quality of rating data collected in various kinds of applied settings (e.g., performance assessment). Major goals are: (a) measuring severity (or leniency) of raters, ability of examinees, difficulty of tasks and items (or criteria) in a single frame of reference, (b) deriving fair measures of examinee ability by taking rater severity, task and item difficulty into account, (c) assessing the degree of rater consistency, (d) detecting other rater effects (e.g., central tendency and halo effects), (e) analyzing interaction effects and differential facet functioning. Perspectives for the development and application of rating systems being as objective, precise, and fair as possible are discussed.