Measuring general aviation pilot judgment using a situational judgment technique

This paper reports on 2 studies that were conducted to develop and to evaluate a situational judgment test (SJT) for general aviation (GA) pilots. An initial 51-item test was constructed in which each item described an in-flight situation that required a decision on the part of the pilot and 4 alternative solutions to the situation. Subject matter experts were used to generate a scoring key from the alternatives that the experts recommended for a typical GA pilot. In Study 1, the test was administered in paper-and-pencil format to 246 participants. Results from that first study showed that the test had acceptable psychometric properties in terms of internal consistency and distribution. In Study 2, the test was administered to 467 participants over the Internet. Analyses resulted in the reduction of the test to 39 items, with an internal consistency reliability (coefficient alpha) of .747. Construct validation was accomplished through correlation with a measure of the number of times the participant had been involved in an accident or other hazardous aviation event. Pilots who had higher (better) scores on the SJT were found to have experienced fewer hazardous events, which was taken as supporting the construct validity of the test. Evaluation of mode of administration (paper-and-pencil vs. Internet) showed that the 2 forms were equivalent. These results suggest that the SJT has potential for use in the assessment of judgment or aeronautical decision making by GA pilots, and might be useful in the evaluation of training. The implications of the findings, regarding equivalence of the 2 administration formats, are discussed.