The Use of Clinical Language Models Pretrained on Institutional EHR Data for Downstream Tasks

Clinical language models have attracted considerable attention in recent years because of their potential to improve healthcare workflows a nd o ptimise p atient c are. While many pre-trained clinical language models have been published, there are no models specifically f or t he T hai c linical context, where English terminologies are used for diseases, procedures and medications, and Thai language is used for clinical notes. This study investigated the pretraining of different language model architectures, namely RoBERTa, GPT-2 and T5, on the EHR data of Songklanagarind Hospital in Thailand, which includes over 80 million documents. We also investigated the applications of the pretrained model to three downstream clinical tasks: tuberculosis case finding, B IRADS c ategory c lassification an d intraocular pressure extraction. The results indicate that our domain-specific language models performed better than the general-purpose language model, mBERT, and required fewer training examples to achieve the same performance. The study encourages the use of clinical language models to streamline clinical workflows, support clinical research and assist hospital auditing.