Evaluation of machine learning models as decision aids for anesthesiologists

Machine Learning (ML) models have been developed to predict perioperative clinical parameters. The objective of this study was to determine if ML models can serve as decision aids to improve anesthesiologists' prediction of peak intraoperative glucose values and postoperative opioid requirements. A web-based tool was used to present actual surgical case and patient information to 10 practicing anesthesiologists. They were asked to predict peak glucose levels and post-operative opioid requirements for 100 surgical patients with and without presenting ML model estimations of peak glucose and opioid requirements. The accuracies of the anesthesiologists' estimates with and without ML estimates as reference were compared. A questionnaire was also sent to the participating anesthesiologists to obtain their feedback on ML decision support. The accuracy of peak glucose level estimates by the anesthesiologists increased from 79.0 +/- 13.7% without ML assistance to 84.7 +/- 11.5% (< 0.001) when ML estimates were provided as reference. The accuracy of opioid requirement estimates increased from 18% without ML assistance to 42% (p < 0.001) when ML estimates were provided as reference. When ML estimates were provided, predictions of peak glucose improved for 8 out of the 10 anesthesiologists, while predictions of opioid requirements improved for 7 of the 10 anesthesiologists. Feedback questionnaire responses revealed that the anesthesiologist primarily used the ML estimates as reference to modify their clinical judgement. ML models can improve anesthesiologists' estimation of clinical parameters. ML predictions primarily served as reference information that modified an anesthesiologist's clinical estimate.