Innovations in Competence Assessment: Design and Initial Validation of the Vignette Matching Assessment Tool (VMAT)

This study responds to an urgent need for innovative approaches to competence assessment in the face of strong evidence that several current instruments may be vulnerable to systematic rating biases. Building on previous research, the current study aimed to (a) design a full catalogue of vignettes for the Vignette Matching Assessment Tool (VMAT); (b) establish a matrix of calibration scores for the catalogue of vignettes by having experts rate each of the vignettes; (c) evaluate its psychometric properties by comparing competence ratings derived from the VMAT with those from a conventional instrument, the Clinical Psychology Practicum Competencies Rating Scale (C Psi PRS); and (d) assess whether the VMAT reduced halo and leniency biases. An initial catalogue of 52 vignettes was recalibrated by a large sample of experts (N = 45) to establish a final catalogue of 41 standardized vignettes across 10 domains and multiple stages of competence. The VMAT used the matrix of calibrated vignettes to help supervisors anchor competence trajectories attained by trainees for each of 10 competency domains. The results provide initial support for the validation of the VMAT. The VMAT attenuated halo effects, and most supervisors endorsed the VMAT as their preferred instrument. Compared to the C-PRS, cutoff scores on the VMAT identified a larger number of trainees with serious competence problems (n = 3 vs. n = 1). The current study has important implications for competence assessments and benchmarking within psychology and valuable cross-disciplinary applications.