The robot doesn't lie: real-life validation of robotic performance metrics

Background Degree of resident participation in a case is often used as a surrogate marker for operative autonomy, an essential element of surgical resident training. Previous studies have demonstrated a considerable disagreement between the perceptions of attending surgeons and trainees when it comes to estimating operative participation. The Da Vinci Surgical System dual console interface allows machine generated measurements of trainee's active participation, which has the potential to obviate the need for labor intensive direct observation of surgical procedures. However, the robotic metrics require validation. We present a comparison of operative participation as perceived by the resident, faculty, trained research staff observer (gold standard), and robotic machine generated data. Methods A total of 28 consecutive robotic inguinal hernia repair procedures were observed by research staff. Operative time, percent active time for the resident, and number of handoffs between the resident and attending were recorded by trained research staff in the operating room and the Da Vinci Surgical System. Attending and resident evaluations of operative performance and perceptions of percent active time for the resident were collected using standardized forms and compared with the research staff observed values and the robot-generated console data. Wilcoxon two-sample tests and Pearson Correlation coefficients statistical analysis were performed. Results Robotic inguinal hernia repair cases had a mean operative time of 91.3 (30) minutes and an attending-rated mean difficulty of 3.1 (1.26) out of 5. Residents were recorded to be the active surgeon 71.8% (17.7) of the total case time by research staff. There was a strong correlation (r = 0.77) in number of handoffs between faculty and trainee as recorded by the research staff and robot (4.28 (2.01) vs. 5.8 (3.04) respectively). The robotic machine generated data demonstrated the highest degree of association when compared to the gold standard (research staff observed data), with r = 0.98, p < 0.0001. Lower levels of association were seen with resident reported (r = 0.66) perceptions and faculty-reported (r = 0.55) perceptions of resident active operative time. Conclusions Our findings suggest that robot-generated performance metrics are an extremely accurate and reliable measure of intraoperative resident participation indicated by a very strong correlation with the data recorded by research staff's direct observation of the case. Residents demonstrated a more accurate awareness of their degree of participation compared with faculty surgeons. With high accuracy and ease of use, robotic surgical system performance metrics have the potential to be a valuable tool in surgical training and skill assessment.