The "Goldilocks Zone": (Too) Many Confidence Intervals In Tests of Mediation Just Exclude Zero

Questionable research practices (QRPs) can occur whenever one result is favored over another, and tests of mediation are no exception. Given mediation's ubiquity and importance to both theory and practice, QRPs in tests of mediation pose a serious threat to the advancement of psychology. We investigate this issue through the introduction of a straightforward means of detecting the presence and magnitude of QRPs in tests of mediation and validate this methodology with a series of sensitivity tests and simulations. We then apply this method to 2,569 tests of mediation published in five leading psychology journals in 2018 and 2019. We find that despite most hypothesized tests of mediation being likely underpowered. most (76%) were nevertheless supported. Furthermore, confidence intervals (CIs) that just barely exclude zero are 3.6 to 4.4 times as prevalent as those CI's that just barely include zero. We also find a number of study- and test-level factors, such as whether the test of mediation was hypothesized, explain both whether the CI excluded zero (odds ratio [OR] = 17.87,p < .001) as well as the CI's proximity to zero (b = .27, p < .001). In addition, other factors, most notably sample size, do predict the CT's proximity to zero (gamma = .00, p < .001), but surprisingly do not predict the CT's exclusion of zero (OR = .99. p = .803). We conclude with actionable QRP curtailment strategies so that both, academics and practitioners, can have greater and well-founded confidence in tests of mediation in psychological research.