Exploring differential item functioning in a 360-degree assessment: Rater source and method of delivery

This research used logistic regression to model item responses from a popular 360-degree assessment-for-development survey. The model used method of survey delivery and rater group to identify items exhibiting differential item functioning (DIF). The methods of survey delivery were pencil-and-paper and online by Web page. The rater groups were self, boss, peer, and direct report, The sample consisted of 374 survey families where a survey family was a set of four surveys: self boss, peer, and direct report, Half the survey families were pencil-and-paper administration; half were online administration. The flagging procedure used effect size from Wald chi-square statistics. Results indicated little evidence that DIF existed due to method of survey delivery, lending additional support for the use of the Internet to deliver 360-degree surveys. Approximately 10% of items exhibited DIF attributable to rater group, though in even, instance the magnitude of the DIF was small, suggesting that the impact of DIF on the 360-degree feedback could be slight. There was no evidence of an interaction between method of delivery and rater group. The maximum likelihood parameter estimates indicated DIF resulted from either hierarchical complexity or contingency, theory. This research suggested that such forms of DIF could be naturally occurring phenomena in 360-degree assessment because both contingency and complexity can influence the perceptions by raters of a manager's performance.