Reliability of Results Produced Through Objectively Structured Assessment of Technical Skills (OSATS) for Endotracheal Intubation (ETI)

Objective: To estimate the reliability of an objectively structured assessment of technical skills (OSATS) in patients for Endotracheal lntubation (ETI), using task-specific checklists during administration of general anaesthesia. Study Design: Co-relational reliability study. Place and Duration of Study: Department of Anaesthesia and Medical Education, Pakistan Institute of Medical Sciences, Islamabad, from September to November 2010. Methodology: Ten first year residents of the department of anaesthesia who had successfully completed a one-day course of ETI on mannequins and had performed 25 ETI in actual patients under supervision, were asked to perform ETI on anaesthetized patients in the operation theater after consent. The procedure was directly observed by two expert observers (ER1 and ER2) and one inexperienced observer (novice, NR) using a validated task oriented checklist. Each trainee repeated the procedure on another patient and was reassessed using the same checklist. Inter-rater for test and retest, and expert to novice reliabilities along with internal consistency were calculated. Results: The inter-rater reliability was 0.92 in test and was 0.98 in retest (Ebel's calculation). Spearman Rank Correlation Coefficient (SRCC) for ER 1 and NR was 0.76 in test and 0.89 in retest. Cronbach's alpha for internal consistency was 0.93 for first test and 0.97 for retost. Conclusion: The OSATS for ETI in real patients demonstrated excellent inter-rater test reliability, inter-rater retest reliability, expert-novice reliability and internal consistency. This instrument, therefore, proves promising as a tool for competency-based evaluations.