Validation of an objective structured assessment of technical skill for surgical residents

Purpose. This study examined the concurrent validity of the Objective Structured Assessment of Technical Skill (OSATS), a new test of technical skill for general surgery residents. Method. Twelve residents (six in their senior, or fifth, year and six in their junior, or third, year) at the University of Toronto in 1994-95 were ranked within level of training according to their OSATS marks and by surgical faculty. Correspondence between OSATS and faculty rankings was assessed using Spearman rank-order correlation coefficients. Results. The correlations between test scores and faculty rankings were generally high for the senior residents but low for the junior residents. Conclusion. Scores on the OSATS accurately reflect the independent opinions of faculty regarding the technical skills of senior residents, suggesting that it is a valid measure of technical skill for these individuals. The scores did not, however, reproduce faculty rankings of the junior residents. Whether this was a failing of the OSATS or the faculty rankings requires further study.