Estimating intercoder reliability: a structural equation modeling approach

Intercoder reliability is usually estimated with a summary index, and yet the limitations concerning the indexing approach have been well noted. This study critically reviewed all the existing major modeling approaches to estimating intercoder reliability, and empirically tested and further compared these approaches. It was found that latent variable modeling, also called the second-generation SEM, generally perform better than log-linear modeling, and is able to explain the paradox haunting some indices, and to spot the sources of disagreement among coders. Implications were discussed at last.