Contrastive Multitask Transformer for Hospital Mortality and Length-of-Stay Prediction

Motivated by the performance on clinical prediction tasks of a transformer-based model (STraTS), we propose a multitask training scheme to exploit information in multiple labels with the goal of improving generalisation, alongside a novel contrastive training scheme. We couple the existing STraTS architecture, which processes temporal data without imputation, with our contrastive training block which optimises embeddings based on the primary diagnosis of the patient, aiming to improve representations of patients who have similar physiological and lab measurements, but different outcomes. We find that multitask training improves the baseline results in three of four prediction tasks, and contrastive pretraining performs similarly to a forecasting pretext task. Finally, we test knowledge transfer to a general ward setting by finetuning a pretrained model on a separate dataset of vital signs. Contrastive pretraining exhibits the highest AUPRC for this challenging task, whilst maintaining a competetive F1 score against the baseline.