Perceived quality and methodology in graduate department ratings: Sociology, political science, and economics

Graduate department ratings help to shape the power and privilege of a graduate department at its home university, as well as with respect to other departments in its discipline. Ideally, such ratings would reflect the perceived quality of a department. Yet frequent critiques suggest that the ratings fall far short of this ideal. The authors analyzed subjective ratings in three related disciplines-sociology, political science, and economics-and from two rating sources: the National Research Council and U.S. News and World Report. They hypothesized three major components to the ratings: (1) perceived departmental quality, (2) systematic error owing to the method of data collection, and (3) random measurement error. Using structural equation modeling, they found that 20 percent to 30 percent of the variation in ratings derives from systematic error, most of the rest comes from perceived quality, and the smallest component is random measurement error. The authors also found that variables, such as citations per capita, departmental size, public/private status, and geographic location, influence ratings through their impact on perceived quality and/or the systematic errors that enter the ratings.