Rating accuracy, leniency, and rater perceptions when using the RPM and BARS

Researchers have argued that social-comparative rating formats hold important psychometric advantages over traditional absolute ratings. We asked 152 participants to observe and assess the videotaped performance of individuals completing a task using a social-comparative (Relative Percentile Method; RPM) and absolute rating (Behaviorally Anchored Rating Scale; BARS) formats. After collecting expert ratings on the same set of videos, we calculated accuracy indices and leniency. Additionally, we collected rater perceptions of accuracy and fairness for both formats. Our study revealed that the BARS was perceived as more accurate and fairer than the RPM. However, the RPM was found to be better at combating rater leniency. We discuss the implications of these findings. Some scholars suggest social-comparative ratings offer psychometric advantages over traditional absolute ratings, improving their value in real-world organizations. Our study scrutinizes whether the Relative Percentile Method (RPM) yields more accurate and less lenient ratings than traditional Behaviorally Anchored Rating Scales (BARS). We probed whether raters perceived the RPM as fairer and more accurate than the BARS. We conducted a lab experiment where raters evaluated the performance of a series of targets using the RPM and BARS and calculated accuracy and leniency metrics. We found that raters perceived the BARS as fairer and more accurate than the RPM. We present evidence that there are trade-offs to using social-comparative ratings in the workplace.