Improving Content Validity Evaluation of Assessment Instruments Through Formal Content Validity Analysis

Content validity is defined as the degree to which elements of an assessment instrument are relevant to and representative of the target construct. The available methods for content validity evaluation typically focus on the extent to which a set of items are relevant to the target construct, but do not afford precise evaluation of items' behavior, nor their exhaustiveness with respect to the elements of the target construct. Formal content validity analysis (FCVA) is a new procedure combining methods and techniques from various areas of psychological assessment, such as (a) constructing Boolean classification matrices to formalize relationships among an assessment instrument's items and target construct elements, and (b) computing interrater agreement indices. We discuss how FCVA can be extended through the implementation of a Bayesian procedure to improve the interrater agreement indices' accuracy (Bayesian formal content validity analysis [B-FCVA]). With respect to extant methods, FCVA and B-FCVA can provide a great amount of information about content validity while not demanding much more work for authors and experts. Translational Abstract Validity is a crucial and multifaceted aspect of research and clinical practice. A measure of a psychological construct is valid if quantitative variations of the construct (e.g., anxiety) are reflected in quantitative variations of the measure of the construct (e.g., a participant's score on a test designed to measure anxiety). Despite the importance of valid measures of psychological constructs, methods for evaluating the content validity of assessment instruments have received relatively little attention. Here, we present formal content validity analysis (FCVA), a new procedure to evaluate the content validity of assessment instruments, and Bayesian formal content validity analysis (B-FCVA), which extends FCVA by embedding it with a recently developed Bayesian method for the correction of interrater agreement indices. FCVA and B-FCVA enable assessment instruments to be constructed whose target constructs are investigated in an exhaustive, nonredundant, and unambiguous manner. This may have positive implications for the accuracy and validity of clinical and theoretical inferences based on test scores. This article presents a theoretical discussion of FCVA and B-FCVA and an illustrative practical example of application of B-FCVA. All the data and the codes used for the computations are available online on OSF (see the Appendix).