Design analysis indicates Potential overestimation of treatment effects in randomized controlled trials supporting Food and Drug Administration cancer drug approvals

Objective: Statistical significance drives interpretation of randomized controlled trials (RCTs). We examined the type S error risk-claiming a new drug is falsely beneficial-and exaggeration ratio-how estimated effects differ from true effects-to re-emphasize direction and magnitude of treatment effects. Study Design and Setting: We systematically reviewed RCTs supporting Food and Drug Administration (FDA) approval of cancer drugs between 2007 and 2016. We extracted data for overall survival (OS), progression-free survival (PFS), and response outcomes from FDA reviews. We estimated type S error risks and exaggeration ratios by considering replicated RCTs of equal size and a range of true effects. Results: We analyzed 42 RCTs for 39 approved drugs. Across 38 RCTs reporting OS, the median type S error risk was 0.00% (Q1-Q3, 0.00-0.01%) and 3.56% (0.40-6.74%), for true hazard ratios of 0.7 and 0.9, respectively, indicating confidence in effect direction. The corresponding exaggeration ratios were 1.09 (1.01-1.11) and 1.30 (1.13-1.42), indicating median overestimations of 9% and 30%. Similar results held for PFS and response outcomes. Conclusions: The type S error risk and exaggeration ratio provide additional insights into the replicability of RCTs. Our analyses also quantify the winner's curse, in which pivotal RCTs tend toward overoptimism. (C) 2018 Elsevier Inc. All rights reserved.