RETHINKING REPLICATION IN EMPIRICAL LEGAL RESEARCH

A large number of systematic replication attempts in the social sciences have failed to support the claims in the original studies. These surprising results have inspired a body of metascientific research aimed at understanding these failures to replicate and ensuring future research is credible. In this article, we relate these new insights from metascience to empirical legal research. Specifically, a recent effort to replicate three influential empirical legal studies published in law journals found results that diverged from the originals. The replicators suggested their results were caused by the changing social context and did not explicitly consider whether the original effects were overstated or false positives. We re-analysed the data from the replications to attempt to confirm their results (i.e., computational reproducibility) with mixed success. When possible, we combined the data from the replications (i.e., meta-analysis) to leverage the greater precision that comes with large sample sizes. In one case, we found an effect where the replicators did not. Overall, however, our re-analysis and review of the broader social scientific context suggests that empirical legal research suffers from the same challenges plaguing other fields - small sample sizes and undisclosed flexibility have produced untrustworthy results.