Prevalence and Magnitude of Paradoxical Results in Multidimensional Item Response Theory

Multidimensional Item Response Theory (MIRT) has been proposed as a means to model the relation between examinee abilities and test responses. Three recent articles proved that when MIRT is used in ability estimation, an examinee's score could theoretically decrease due to a correct answer or increase due to an incorrect answer. The current article examines the extent to which such ''paradoxical results'' can arise in practice. In an operational test designed to measure two dimensions, a substantial percentage of paradoxical results occurred when using a MIRT model with a prior correlation of 0 between abilities. Assuming a positive correlation between abilities reduced the prevalence of paradoxical results but did not eliminate them entirely. Associated issues in test fairness are discussed.