Comparative Judgment as a reliable and valid assessment method in resume screening: a comparison between experts and novices

In practice and in earlier empirical research, it is indicated that resume screening does not always provide suitable candidates for a vacancy. This might be due to several issues: one assessor carries out the screening, resulting in cognitive distortions influencing the selection process; the assessment does not focus on all aspects of the selection, nor does it allow for certain criteria to weigh more heavily than others; and/or the assessor is insufficiently trained to carry out the resume screening. The current study captures these issues by offering an alternative assessment method (Comparative Judgment) in which the inter-rater reliability and construct validity of the resume screening is studied: several assessors with different levels of expertise assess resumes comparatively. In this study, resumes from 42 candidates applying for an existing vacancy were used. These resumes were directly compared by experienced (N = 7; experts), and less experienced assessors (N = 57; students). Results show that the comparative judgements of experienced assessors are linked to valid and reliable resume screening. The inter-rater reliability of the student assessments was lower than that of the experts. Even though the final rank ordering of the resumes correlated, students often relied on irrelevant aspects in the resumes.