Causal Definitions Versus Casual Estimation: Reply to Valente et al. (2022)

In this response to Valente et al. (2022), I am discussing the plausibility and applicability of the proposed mediation model and its causal effects estimation for single case experimental designs (SCEDs). I will focus on the underlying assumptions that the authors use to identify the causal effects. These assumptions include the particularly problematic assumption of sequential ignorability or no-unmeasured confounders. First, I will discuss the plausibility of the assumption in general and then particularly for SCEDs by providing an analytic argument and a reanalysis of the empirical example in Valente et al. (2022). Second, I will provide a simulation that reproduces the design by Valente et al. (2022) with the exception that, for a more realistic depiction of empirical data, an unmeasured confounder affects the mediator and outcome variables. The results of this simulation study indicate that even minor violations will lead to Type I error rates up to 100% and coverage rates as low as 0% for the defined causal direct and indirect effects. Third, using historical data on the effect of birth control on stork population and birth rates, I will show that mediation models like the proposed method can lead to surprising artifacts. These artifacts can hardly be identified with statistically means including methods such as sensitivity analyses. In this response to Valente et al. (2022), I am discussing the plausibility and applicability of the proposed mediation model and its causal effects estimation for single case experimental designs (SCEDs). SCEDs produce time series data for single persons using repeated measures of the same variable(s). Mediation models try to identify intermediate variables which are thought of being part of a causal chain leading from the intervention to the relevant outcome variable. One of the main problems of the approach-which is similar to the majority of mediation models-is the use of a no-unmeasured-confounder assumption. This assumption is problematic because unmeasured confounders may exist in virtually all applications. I will illustrate with two empirical examples and a small simulation study why a violation of the assumption will lead to severe problems when using this mediation model, that is, the detection of statistical artifacts instead of actual intermediate variables. I will further show that it is very complicated to detect how strongly violated the assumption is and that applied users may have difficulties in judging the validity of their results when using the proposed mediation model.