Evaluation of objective tools and artificial intelligence in robotic surgery technical skills assessment: a systematic review

Background There is a need to standardize training in robotic surgery, including objective assessment for accreditation. This systematic review aimed to identify objective tools for technical skills assessment, providing evaluation statuses to guide research and inform implementation into training curricula.Methods A systematic literature search was conducted in accordance with the PRISMA guidelines. Ovid Embase/Medline, PubMed and Web of Science were searched. Inclusion criterion: robotic surgery technical skills tools. Exclusion criteria: non-technical, laparoscopy or open skills only. Manual tools and automated performance metrics (APMs) were analysed using Messick's concept of validity and the Oxford Centre of Evidence-Based Medicine (OCEBM) Levels of Evidence and Recommendation (LoR). A bespoke tool analysed artificial intelligence (AI) studies. The Modified Downs-Black checklist was used to assess risk of bias.Results Two hundred and forty-seven studies were analysed, identifying: 8 global rating scales, 26 procedure-/task-specific tools, 3 main error-based methods, 10 simulators, 28 studies analysing APMs and 53 AI studies. Global Evaluative Assessment of Robotic Skills and the da Vinci Skills Simulator were the most evaluated tools at LoR 1 (OCEBM). Three procedure-specific tools, 3 error-based methods and 1 non-simulator APMs reached LoR 2. AI models estimated outcomes (skill or clinical), demonstrating superior accuracy rates in the laboratory with 60 per cent of methods reporting accuracies over 90 per cent, compared to real surgery ranging from 67 to 100 per cent.Conclusions Manual and automated assessment tools for robotic surgery are not well validated and require further evaluation before use in accreditation processes. PROSPERO: registration ID CRD42022304901Conclusions Manual and automated assessment tools for robotic surgery are not well validated and require further evaluation before use in accreditation processes. PROSPERO: registration ID CRD42022304901 This systematic review provides a comprehensive evaluation of current objective, manual, automated and artificial intelligence (AI) methods used in robotic technical skills assessment. Many lack full evaluation and AI is in its conceptual stages. Background Robotic surgery is increasingly used worldwide to treat many different diseases. The robot is controlled by a surgeon, which may give them greater precision and better outcomes for patients. However, surgeons' robotic skills should be assessed properly, to make sure patients are safe, to improve feedback and for exam assessments for certification to indicate competency. This should be done by experts, using assessment tools that have been agreed upon and proven to work.Aim This review's aim was to find and explain which training and examination tools are best for assessing surgeons' robotic skills and to find out what gaps remain requiring future research.Method This review searched for all available studies looking at assessment tools in robotic surgery and summarized their findings using several different methods.Findings and conclusion Two hundred and forty-seven studies were looked at, finding many assessment tools. Further research is needed for operation-specific and automatic assessment tools before they should be used in the clinical setting.