A full-document analysis of the semantic relation between European Public Assessment Reports and EMA guidelines using a BERT language model

In the European Union, the Committee for Medicinal Products for Human Use of the European Medicines Agency (EMA) develop guidelines to guide drug development, supporting development of efficacious and safe medicines. A European Public Assessment Report (EPAR) is published for every medicine application that has been granted or refused marketing authorisation within the EU. In this work, we study the use of text embeddings and similarity metrics to investigate the semantic similarity between EPARs and EMA guidelines. All 1024 EPARs for initial marketing authorisations from 2008 to 2022 was compared to the 669 current EMA scientific guidelines. Documents were converted to plain text and split into overlapping chunks, generating 265,757 EPAR and 27,649 guideline text chunks. Using a Sentence BERT language model, the chunks were transformed into embeddings and fed into an in-house piecewise matching algorithm to estimate the full-document semantic distance. In an analysis of the document distance scores and product characteristics using a linear regression model, EPARs of anti-virals for systemic use (ATC code J05) and antihemorrhagic medicines (B02) present with statistically significant lower overall semantic distance to guidelines compared to other therapeutic areas, also when adjusting for product age and EPAR length. In conclusion, we believe our approach provides meaningful insight into the interplay between EMA scientific guidelines and the assessment made during regulatory review, and could potentially be used to answer more specific questions such as which therapeutic areas could benefit from additional regulatory guidance.