Applying Multivariate Generalizability Theory to Psychological Assessments

Multivariate generalizability theory (GT) represents a comprehensive framework for quantifying score consistency, separating multiple sources contributing to measurement error, correcting correlation coefficients for such error, assessing subscale viability, and determining the best ways to change measurement procedures at different levels of score aggregation. Despite such desirable attributes, multivariate GT has rarely been applied when measuring psychological constructs and far less often than univariate techniques that are subsumed within that framework. Our purpose in this tutorial is to describe multivariate GT in a simple way and illustrate how it expands and complements univariate procedures. We begin with a review of univariate GT designs and illustrate how such designs serve as subcomponents of corresponding multivariate designs. Our empirical examples focus primarily on subscale and composite scores for objectively scored measures, but guidelines are provided for applying the same techniques to subjectively scored performance and clinical assessments. We also compare multivariate GT indices of score consistency and measurement error to those obtained using alternative GT-based procedures and across different software packages for analyzing multivariate GT designs. Our online supplemental materials include instruction, code, and output for common multivariate GT designs analyzed using mGENOVA and the gtheory, glmmTMB, lavaan, and related packages in R.