Application of the Reverse Fragility Index to Statistically Nonsignificant Randomized Clinical Trial Results

Question In clinical trials with statistically nonsignificant primary end point results, what is the minimum number of events that must be changed to move the result from nonsignificant to statistically significant (ie, the reverse fragility index)? Findings In this cross-sectional study of 167 randomized clinical trials with statistically nonsignificant results, the median reverse fragility index at a threshold of P = .05 was 8. A median of 8 events were required to change to enable a nonsignificant primary end point to become statistically significant. Meaning Results of this cross-sectional study suggest that the reverse fragility index, along with effect sizes and associated 95% CIs, may provide a useful context for interpreting null clinical trial results. Importance Interpreting randomized clinical trials (RCTs) and their clinical relevance is challenging when P values are either marginally above or below the P = .05 threshold. Objective To use the concept of reverse fragility index (RFI) to provide a measure of confidence in the neutrality of RCT results when assessed from the clinical perspective. Design, Setting, and Participants In this cross-sectional study, a MEDLINE search was conducted for RCTs published from January 1, 2013, to December 31, 2018, in JAMA, the New England Journal of Medicine (NEJM), and The Lancet. Eligible studies were phase 3 and 4 trials with 1:1 randomization and statistically nonsignificant binary primary end points. Data analysis was performed from August 1, 2019, to August 31, 2019. Exposures Single vs multicenter enrollment, total number of events, private vs government funding, placebo vs active control, and time to event vs frequency data. Main Outcomes and Measures The primary outcome was the median RFI with interquartile range (IQR) at the P = .05 threshold. Secondary outcomes were the number of RCTs in which the number of participants lost to follow-up was greater than the RFI; the median RFI with IQR at different P value thresholds; the median reverse fragility quotient with IQR; and the correlation between sample sizes, number of events, and P values of the RCT and RFI. Results Of the 167 RCTs included, 76 (46%) were published in the NEJM, 50 (30%) in JAMA, and 41 (24%) in The Lancet. The median (IQR) sample size was 970 (470-3427) participants, and the median (IQR) number of events was 251 (105-570). The median (IQR) RFI at the P = .05 threshold was 8 (5-13). Fifty-seven RCTs (34%) had an RFI of 5 or lower, and in 68 RCTs (41%) the number of participants lost to follow-up was greater than the RFI. Trials with P values ranging from P = .06 to P = .10 had a median (IQR) RFI of 3 (2-4). When compared, median (IQR) RFIs were not statistically significant for single-center vs multicenter enrollment (5 [4-13] vs 8 [5-13]; P = .41), private vs government-funded studies (9 [5-13] vs 8 [5-13]; P = .34), and time-to-event primary end points vs frequency data (9 [5-14] vs 7 [4-13]; P = .43). The median (IQR) RFI at the P = .01 threshold was 12 (7-19) and at the P = .005 threshold was 14 (9-21). Conclusions and Relevance This cross-sectional study found that a relatively small number of events (median of 8) had to change to move the primary end point of an RCT from nonsignificant to statistically significant. These findings emphasize the nuance required when interpreting trial results that did not meet prespecified significance thresholds. This cross-sectional study describes application of the reverse fragility index in establishing the clinical relevance of published clinical trials with null results.