Improving rater calibration in aviation: A case study

The assessment of pilot performance relies on systematic observation and assessment by a single trained rater or instructor/evaluator (I/E). Because of the importance of aviation safety, it is imperative that the rating and evaluation processes used by these I/Es yield reliable and valid data. This article describes a case study focused on improving the reliability and validity of crew assessment at a commercial carrier. The process for improving reliability involved the evaluation of current training methods, the construction and evaluation of 5 metrics for assessing inter-rater reliability, and a standardized process for using these metrics to train I/Es. A separate set of interventions was developed and implemented to improve structural validity. Data collected from 2 fleets in this airline during a 3-year period indicate aspects of reliability and validity that should be the focal points of continuing I/E training.