Missing Data in Experiments: Challenges and Solutions

Translational Abstract Researchers rarely manage to collect every piece of information about each participant in their study. For instance, participants sometimes refuse to answer questions that they consider sensitive (e.g., income, political orientation, sexual practices) or quit the study before completing it. If ignored or handled inappropriately, this phenomenon referred to as "missingness" generally compromises researchers' ability to make causal inferences based on their experiments. Specifically, missingness biases researchers' estimates of the effect size of the treatment. In this tutorial, we review the different ways in which missingness impacts the results of experimental studies and provide researchers with concrete steps for addressing each type of missingness they may encounter. For mild cases of missingness, we recommend using a method called inverse probability weighting (IPW). For severe instances of missingness, we recommend that researchers recontact a sample of participants with missing values to fill the gaps. This method, which involves recollecting data, is called double sampling and bounds. For both methods, we provide lines of R code that researchers may use in their own analyses. Missing data is a common feature of experimental datasets. Standard methods used by psychology researchers to handle missingness rely on unrealistic assumptions, invalidate random assignment procedures, and bias estimates of effect sizes. We describe different classes of missing data typically encountered in experimental datasets, and we discuss how each of them impacts researchers' causal inferences. In this tutorial, we provide concrete guidelines for handling each class of missingness, focusing on 2 methods that make realistic assumptions: (a) inverse probability weighting (IPW) for mild instances of missingness, and (b) double sampling and bounds for severe instances of missingness. After reviewing the reasons why these methods increase the accuracy of researchers' estimates of effect sizes, we provide lines of R code that researchers may use in their own analyses.