Evaluating automated machine learning platforms for use in healthcare

Objective To describe development and application of a checklist of criteria for selecting an automated machine learning (Auto ML) platform for use in creating clinical ML models.Materials and Methods Evaluation criteria for selecting an Auto ML platform suited to ML needs of a local health district were developed in 3 steps: (1) identification of key requirements, (2) a market scan, and (3) an assessment process with desired outcomes.Results The final checklist comprising 21 functional and 6 non-functional criteria was applied to vendor submissions in selecting a platform for creating a ML heparin dosing model as a use case.Discussion A team of clinicians, data scientists, and key stakeholders developed a checklist which can be adapted to ML needs of healthcare organizations, the use case providing a relevant example.Conclusion An evaluative checklist was developed for selecting Auto ML platforms which requires validation in larger multi-site studies. Machine learning (ML) is a form of artificial intelligence whereby computers learn associations within large complex datasets and encode these into a statistical model that can then be applied to new datasets in generating predictions or classifications. In healthcare, such models can assist clinicians in making diagnostic, therapeutic, and prognostic decisions. However, developing and testing such models for different use cases take time and effort on the part of data scientists, collaborating clinicians, and informatics teams who may not have extensive data and model processing capacity. Auto ML platforms are designed to rapidly build and validate ML models by automating complex, time-consuming tasks involved in data processing and model training. Numerous Auto ML platforms now available from both open source and commercial vendors necessitate guidance in how to choose the one most appropriate to organizational needs. Using systematic methods, a multidisciplinary team formulated an evaluation checklist for objectively appraising different Auto ML platforms in making a final selection. The checklist was assessed for its utility by its application to a practical use case of a dosing model for an intravenous antithrombotic with unpredictable therapeutic effects. The checklist may prove useful to other users and can accommodate organizational procurement requirements.