Assessing and testing interrater agreement on a single target using multi-item rating scales

An index for assessing interrater agreement with respect to a single target using a multi-item rating scale is developed. The variance of rarer mean scale scores is used as the numerator of the agreement index. Four variants of a disattenuated agreement index that vary in the random response term used as the denominator are examined. A variant of r'(wg( J )) using perfectly collinear, uniformly distributed responses as the baseline is recommended as a useful index of disattenuated agreement. A test of the equality of variances is recommended for deleting items with a level of interrater agreement significantly different from zero but tower than that of the remaining items.