Reliability From α to ω: A Tutorial

Reliability is a fundamental problem for measurement in all of science. Although defined in multiple ways, and estimated in even more ways, the basic concepts seem straightforward and need to be understood by practitioners as well as methodologists. Reliability theory is not just for the psychometrician estimating latent variables, it is for everyone who wants to make inferences from measures of individuals or of groups. For the case of a single test administration, we consider multiple measures of reliability, ranging from the worst (beta) to average (alpha, lambda(3)) to best ( lambda(4)) split half reliabilities, and consider why model-based estimates (omega(h), omega(t)) should be reported. We also address the utility of test-retest and alternate form reliabilities. The advantages of immediate versus delayed retests to decompose observed score variance into specific, state, and trait scores are discussed. But reliability is not just for test scores, it is also important when evaluating the use of ratings. Estimates that may be applied to continuous data include a set of intraclass correlations while discrete categorical data needs to take advantage of the family of kappa statistics. Examples of these various reliability estimates are given using state and trait measures of anxiety given with different delays and under different conditions. An online supplemental materials is provided with more detail and elaboration. The online supplemental materials is also used to demonstrate applications of open source software to examples of real data, and comparisons are made between the many types of reliability.