Interpretation of time-to-event outcomes in randomized trials: an online randomized experiment

Background: Multiple features in the presentation of randomized controlled trial (RCT) results are known to influence comprehension and interpretation. We aimed to compare interpretation of cancer RCTs with time-to-event outcomes when the reported treatment effect measure is the hazard ratio (HR), difference in restricted mean survival times (RMSTD), or both (HR+RMSTD). We also assessed the prevalence of misinterpretation of the HR. Methods: We carried out a randomized experiment. We selected 15 cancer RCTs with statistically significant treatment effects for the primary outcome. We masked each abstract and created three versions reporting either the HR, RMSTD, or HR+RMSTD. We randomized corresponding authors of RCTs and medical residents and fellows to one of 15 abstracts and one of 3 versions. We asked how beneficial the experimental treatment was (0-10 Likert scale). All participants answered a multiple-choice question about interpretation of the HR. Participants were unaware of the study purpose. Results: We randomly allocated 160 participants to evaluate an abstract reporting the HR, 154 to the RMSTD, and 155 to both HR+RMSTD. The mean Likert score was statistically significantly lower in the RMSTD group when compared with the HR group (mean difference -0.8, 95% confidence interval, -1.3 to -0.4, P < 0.01) and when compared with the HR+RMSTD group (difference -0.6, -1.1 to -0.1, P = 0.05). In all, 47.2% (42.7%-51.8%) of participants misinterpreted the HR, with 40% equating it with a reduction in absolute risk. Conclusion: Misinterpretation of the HR is common. Participants judged experimental treatments to be less beneficial when presented with RMSTD when compared with HR. We recommend that authors present RMST-based measures alongside the HR in reports of RCT results.