Development of an algorithm to aid triage decisions for intensive care unit admission: a clinical vignette and retrospective cohort study

Background: Intensive care unit (ICU) admission triage is performed routinely and is often based solely on clinical judgment, which could mask biases. A computerized algorithm to aid ICU triage decisions was developed to classify patients into the Society of Critical Care Medicine's prioritization system. In this study, we sought to evaluate the reliability and validity of this algorithm. Methods: Nine senior physicians evaluated forty clinical vignettes based on real patients. The reference standard was defined as the priorities ascribed by two investigators with full access to patients' records. Agreement of algorithm-based priorities with the reference standard and with intuitive priorities provided by the physicians were evaluated. Correlations between algorithm prioritization and physicians' judgment of the appropriateness of ICU admissions in scarcity and nonscarcity settings were also evaluated. Validity was further assessed by retrospectively applying this algorithm to 603 patients with requests for ICU admission for association with clinical outcomes. Results: Agreement between algorithm-based priorities and the reference standard was substantial, with a median kappa of 0.72 (interquartile range [IQR] 0.52-0.77). Algorithm-based priorities demonstrated higher interrater reliability (overall kappa 0.61, 95 % confidence interval [CI] 0.57-0.65; median percentage agreement 0.64, IQR 0.59-0.70) than physicians' intuitive prioritization (overall kappa 0.51, 95 % CI 0.47-0.55; median percentage agreement 0.49, IQR 0.44-0.56) (p = 0.001). Algorithm-based priorities were also associated with physicians' judgment of appropriateness of ICU admission (priorities 1, 2, 3, and 4 vignettes would be admitted to the last ICU bed in 83.7 %, 61.2 %, 45.2 %, and 16.8 % of the scenarios, respectively; p < 0.001) and with actual ICU admission, palliative care consultation, and hospital mortality in the retrospective cohort. Conclusions: This ICU admission triage algorithm demonstrated good reliability and validity. However, more studies are needed to evaluate a difference in benefit of ICU admission justifying the admission of one priority stratum over the others.