Evaluation of a Natural Language Processing Approach to Identify Diagnostic Errors and Analysis of Safety Learning System Case Review Data: Retrospective Cohort Study

Background: Diagnostic errors are an underappreciated cause of preventable mortality in hospitals and pose a risk for severepatient harm and increase hospital length of stay. Objective: This study aims to explore the potential of machine learning and natural language processing techniques in improvingdiagnostic safety surveillance. We conducted a rigorous evaluation of the feasibility and potential to use electronic health recordsclinical notes and existing case review data. Methods: Safety Learning System case review data from 1 large health system composed of 10 hospitals in the mid-Atlanticregion of the United States from February 2016 to September 2021 were analyzed. The case review outcome included opportunitiesfor improvement including diagnostic opportunities for improvement. To supplement case review data, electronic health recordclinical notes were extracted and analyzed. A simple logistic regression model along with 3 forms of logistic regression models(ie, Least Absolute Shrinkage and Selection Operator, Ridge, and Elastic Net) with regularization functions was trained on thisdata to compare classification performances in classifying patients who experienced diagnostic errors during hospitalization.Further, statistical tests were conducted to find significant differences between female and male patients who experienced diagnosticerrors.Results: In total, 126 (7.4%) patients (of 1704) had been identified by case reviewers as having experienced at least 1 diagnosticerror. Patients who had experienced diagnostic error were grouped by sex: 59 (7.1%) of the 830 women and 67 (7.7%) of the 874men. Among the patients who experienced a diagnostic error, female patients were older (median 72, IQR 66-80 vs median 67,IQR 57-76; P=.02), had higher rates of being admitted through general or internal medicine (69.5% vs 47.8%; P=.01), lowerrates of cardiovascular-related admitted diagnosis (11.9% vs 28.4%; P=.02), and lower rates of being admitted through neurology department (2.3% vs 13.4%; P=.04). The Ridge model achieved the highest area under the receiver operating characteristic curve(0.885), specificity (0.797), positive predictive value (PPV; 0.24), and F1-score (0.369) in classifying patients who were at higherrisk of diagnostic errors among hospitalized patients .Conclusions: Our findings demonstrate that natural language processing can be a potential solution to more effectively identifyingand selecting potential diagnostic error cases for review and therefore reducing the case review burden