Assessing the inter-rater and intra-rater reliability of the Physical Therapy Competence Assessment for Airway Suctioning (PT-CAAS)

Background and Purpose: The Physical Therapy Competence Assessment for Airway Suctioning (PT-CAAS) is a recently developed measure to assess the clinical competence of physiotherapists who perform airway suctioning with adults. The purpose of this study was to assess the inter-rater and intra-rater reliability of the PT-CAAS. Methods: Scoring rules were developed through expert consultation. Reliability was then assessed using nine videos of suctioning performed in a simulated learning environment. A repeated measures design was used, with two replicate sets of measurements made by each participant for all videos. Data were analyzed using a repeated measures model for the concurrent assessment of inter-rater and intrarater reliability. Participants were physiotherapists with suctioning experience. Results: Twenty physiotherapists completed initial scoring and re-scoring for all nine videos; their data were included in the analysis. Intraclass correlation coefficients (ICCs) for inter-rater reliability ranged from 0.569 [lower one-sided 95% confidence interval (CI): 0.395; standard error of measurement (SEM): 0.963] for infection control to 0.759 (lower one-sided 95% CI: 0.612; SEM: 0.722) for post-suctioning assessment and care. The inter-rater ICC for overall performance was 0.752 (lower one-sided 95% CI: 0.602; SEM: 0.660). ICCs for intra-rater reliability ranged from 0.759 (lower one-sided 95% CI: 0.197; SEM 0.721) for infection control to 0.860 (lower one-sided 95% CI: 0.544; SEM: 0.550) for post-suctioning assessment and care. The intra-rater ICC for overall performance was 0.867 (lower one-sided 95% CI: 0.559; SEM: 0.483). Discussion: Evidence of moderate to good inter-rater and good intra-rater reliability was found; however, the results should be interpreted with caution given the wide Cis and relatively large SEMs. Improved assessor training and assessments of reliability using a larger sample size are recommended.