Score resolution: An investigation of the reliability and validity of resolved scores

When raters assign different scores to a performance task, a method for resolving rating differences is required to report a single score to the examinee. Recent studies indicate that decisions about examinees, such as pass/fail decisions, differ across resolution methods. Previous studies also investigated the interrater reliability of operational scores formed through different resolution methods, however, reliability might have been overestimated because adjudication was conditional on raters' initial disagreement. This study used a replication design involving autonomous teams of raters to investigate the reliability associated with three forms of resolution: averaging the original raters' scores, averaging the original raters' scores with an adjudicator's score, and matching the adjudicator's score with the closest original score. This study Also examined validity coefficients for resolved scores and two types of criterion scores. Findings include (a) interrater reliability was slightly higher for the resolution method that averages the scores of the original raters and the adjudicator and (b) the lowest validity coefficients were associated most frequently with the method that matches the adjudicator's score with the closest original score.