Multiple Confidence Intervals and Surprisal Intervals to Avoid Significance Fallacy

Overconfidence in statistical results in medicine is fueled by improper practices and historical biases afflicting the concept of statistical significance. In particular, the dichotomization of significance (i.e., significant vs. not significant), blending of Fisherian and Neyman-Pearson approaches, magnitude and nullification fallacies, and other fundamental misunderstandings distort the purpose of statistical investigations entirely, impacting their ability to inform public health decisions or other fields of science in general. For these reasons, the international statistical community has attempted to propose various alternatives or different interpretative modes. However, as of today, such misuses still prevail. In this regard, the present paper discusses the use of multiple confidence (or, more aptly, compatibility) intervals to address these issues at their core. Additionally, an extension of the concept of confidence interval, called surprisal interval (S-interval), is proposed in the realm of statistical surprisal. The aforementioned is based on comparing the statistical surprise to an easily interpretable phenomenon, such as obtaining S consecutive heads when flipping a fair coin. This allows for a complete departure from the notions of statistical significance and confidence, which carry with them longstanding misconceptions.