Utility of the Mantel-Haenszel procedure for detecting differential item functioning in small samples

Sample-size restrictions limit the contingency table approaches based on asymptotic distributions, such as the Mantel-Haenszel (MH) procedure, for detecting differential item functioning (DIF) in many practical applications. Within this framework, the present study investigated the power and Type I error performance of empirical and inferential criteria for DIF detection in small samples. Sample sizes (50/50, 100/50, 200/50, and 100/100 for the reference and focal groups, respectively), ability distributions (equal and unequal), and amount of DIF (moderate and high) were manipulated. The results show the advantages of employing the MH chi-square statistic using high levels of significance (alpha = .20) as opposed to the empirical criteria (cutoffs for categorizing DIF based on the magnitude of the MH common odds ratio estimator and the standardized p-difference statistic). Some considerations concerning Type I and Type II errors are made.