A model-based hybrid soft actor-critic deep reinforcement learning algorithm for optimal ventilator settings

A ventilator is a device that mechanically assists in pumping air into the lungs, which is a life-saving supportive therapy in an intensive care unit (ICU). In clinical scenarios, each patient has unique physiological circumstances and specific respiratory diseases, thus requiring individualized ventilator settings. Long-term supervision by experienced clini-cians is essential to perform the task of precisely adjusting ventilator parameters and mak-ing timely modifications. Moreover, a tiny clinical error can result in severe lung injury, induce multi-system organ dysfunction, and increase mortality. To reduce the workload of clinicians and prevent medical errors, machine learning (ML), or more specifically, rein-forcement learning (RL) methods, have been developed to automatically adjust the venti-lator's parameters and select optimal strategies. However, the ventilator settings contain both continuous (e.g., frequency) and discrete parameters (e.g., ventilation mode), making it challenging for conventional RL-based approaches to handle such problems. Meanwhile, it is necessary to develop models with high data efficiency to overcome medical data insuf-ficiency. In this paper, we propose a model-based hybrid soft actor-critic (MHSAC) algo-rithm that is developed based on the classic soft actor-critic (SAC) and model-based policy optimization (MBPO) framework. This algorithm can learn both continuous and dis-crete policies according to the current and predictive state of patient's physiological infor-mation with high data efficiency. Results reveal that our proposed model significantly outperforms the baseline models, achieving superior efficiency and high accuracy in the OpenAI Gym simulation environment. Our proposed model is capable of resolving mixed action space problems, enhancing data efficiency, and accelerating convergence, which can generate practical optimal ventilator settings, minimize possible medical errors, and provide clinical decision support.(c) 2022 Elsevier Inc. All rights reserved.