Measuring the quality of research-A critical analysis of the Forschungsrating of the German Wissenschaftsrat

This paper deals with the Forschungsrating (RWR) by the German Wissenschaftsrat (WR) in sociology. The WR's objective in conducting this pilot study was to test a practice which allows a sophisticated assessment of the research facilities in Germany. Basically, two theses are exhibited towards the RWR in this paper: First, it seems possible that the published results are neither objective, nor reliable or valid because of implicit assessment procedures by the reviewers. Second we ask, whether a simple model of scientific quality, composed of a few quantitative indicators, yields similar results compared with the RWR. To empirically address these aspects, data of the RWR and our own standardized survey on the reputation of German research facilities is analyzed. Furthermore we conducted interviews with members of the group of reviewers. The results show that problems concerning the objectivity of analysis cannot be excluded. Our general conclusion about the reliability of the measurement, limited to the inner consistency of the quantitative indicators, is positive. A satisfactory reproduction of the RWR's results solely based on quantitative data cannot be achieved. This is interpreted as a sign of minor validity. It is concluded that the amount and quality of data collected in the RWR, as well as the standardization of the rating process need to be improved.