Evidence reversals in primary care research: a study of randomized controlled trials

Background Evidence-Based Medicine is built on the premise that clinicians can be more confident when their decisions are grounded in high-quality evidence. Furthermore, evidence from studies involving patient-oriented outcomes is preferred when making decisions about tests or treatments. Ideally, the findings of relevant and valid trials should be stable over time, that is, unlikely to be reversed in subsequent research. Objective To evaluate the stability of evidence from trials relevant to primary healthcare and to identify study characteristics associated with their reversal. Methods We studied synopses of randomized controlled trials (RCTs) published from 2002 to 2005 as "Daily POEMs" (Patient Oriented Evidence that Matters). The initial evidence (E-1) from these POEMs (2002-2005) was compared with the updated evidence (E-2) on that same topic in a summary resource (DynaMed 2019). Two physician-raters independently categorized each POEM-RCT as (i) reversed when E-1 not equal E-2, or as (ii) not reversed, when E-1 = E-2. For all "Evidence Reversals" (E-1 not equal E-2), we assessed the direction of change in the evidence. Results We evaluated 408 POEMs on RCTs. Of those, 35 (9%; 95% confidence interval [6-12]) were identified as reversed, 359 (88%) were identified as not reversed, and 14 (3%) were indeterminate. On average, this represents about 2 evidence reversals per annum for POEMs about RCTs. Conclusions Over 12-17 years, 9% of RCTs summarized as POEMs are reversed. Information alerting services that apply strict criteria for relevance and validity of clinical information are likely to identify RCTs whose findings are stable over time. Lay Summary We studied the extent to which evidence from randomized controlled trials (RCTs) relevant to primary care is contradicted in subsequent research. When it was, we identified this event as an evidence reversal. In addition, we sought to identify characteristics of RCTs associated with their reversal. From 408 RCTs published during the period 2002-2005, study characteristics such as sample size were identified and extracted. Subsequently, we compared the evidence reported in each of these RCTs with the evidence on that same topic in an online summary resource in 2019. This allowed us to classify each RCT in one of the following 3 categories: evidence confirmed, reversed, or uncertain if this evidence is confirmed or reversed. Over 12-17 years of follow-up time, the findings of about 9 in 10 RCTs summarized as POEMs are stable. We found no statistically significant associations between trial characteristics and their subsequent reversal. This low rate of evidence reversal is good news for the RCTs that are used to inform decision-making.