The effect of clustered objective probability scoring with truncation (T-COPS) on reliability

Building on de Finetti (1965) and Shuford et al. (1966), Leclercq (1983) maintained that any method of scoring multiple-choice tests should take into account the existence of varying degrees of correctness of "correct" responses. This paper introduces Clustered Objective Probability Scoring (COPS) procedure and a modified version (T-COPS) as such methods.. COPS combines 1/0 response-correctness matrices and polytomous confidence-level matrices in such a way that potentially contaminating personality factors will have no bearing on the outcome. In T-COPS, one additional step is taken at the last phase to truncate scores lower than chance level. Five data sets produced by Japanese university students responding to several types of multiple-choice English reading tests were scored by the number-correct method, COPS, and T-COPS. The focus was the extent to which COPS and T-COPS would improve upon reliability and mean item discrimination of the original dichotomous data. The results showed that COPS indeed boosted both properties substantially, and T-COPS achieved still further improvement upon COPS. The effect of employing T-COPS on reliability was equivalent to lengthening the original dichotomously-scored test from four to five times.