Performance of ChatGPT in the In-Training Examination for Anesthesiology and Pain Medicine Residents in South Korea: Observational Study

Background: ChatGPT has been tested in health care, including the US Medical Licensing Examination and specialty exams,showing near-passing results. Its performance in the field of anesthesiology has been assessed using English board examination questions; however, its effectiveness in Korea remains unexplored. Objective: This study investigated the problem-solving performance of ChatGPT in the fields of anesthesiology and painmedicine in the Korean language context, highlighted advancements in artificial intelligence (AI), and explored its potentialapplications in medical education. Methods: We investigated the performance (number of correct answers/number of questions) of GPT-4, GPT-3.5, and CLOVAX in the fields of anesthesiology and pain medicine, using in-training examinations that have been administered to Koreananesthesiology residents over the past 5 years, with an annual composition of 100 questions. Questions containing images,diagrams, or photographs were excluded from the analysis. Furthermore, to assess the performance differences of the GPT acrossdifferent languages, we conducted a comparative analysis of the GPT-4's problem-solving proficiency using both the originalKorean texts and their English translations. Results: A total of 398 questions were analyzed. GPT-4 (67.8%) demonstrated a significantly better overall performance thanGPT-3.5 (37.2%) and CLOVA-X (36.7%). However, GPT-3.5 and CLOVA X did not show significant differences in their overallperformance. Additionally, the GPT-4 showed superior performance on questions translated into English, indicating a languageprocessing discrepancy (English: 75.4% vs Korean: 67.8%; difference 7.5%; 95% CI 3.1%-11.9%; P=.001). Conclusions: This study underscores the potential of AI tools, such as ChatGPT, in medical education and practice but emphasizesthe need for cautious application and further refinement, especially in non-English medical contexts. The findings suggest thatalthough AI advancements are promising, they require careful evaluation and development to ensure acceptable performanceacross diverse linguistic and professional settings.