Testing for Between-Group Differences in Within-Group Interrater Agreement

Users of r(wg) may find situations where they want to compare the levels of within-group agreement across two or more groups. F tests are presented that compare two values of r(wg), r(wg(J)), or r(wg)*((J)), and recommendations are made for comparing pairwise group values for three or more groups with a control for alpha inflation. More robust and flexible methods based on O'Brien's test are then recommended when sufficient raw data are available. The meaning of findings that agreement statistics are either the same or different across groups is discussed in light of simultaneous comparisons of mean differences. Examples are shown to demonstrate the robust methods using employee responses to questions regarding climate perceptions of organizational support for employee development activities. Other scenarios where the F test may prove useful are discussed, and directions for future research are provided.