Validity and reliability of subjective social indicators - The effect of different measures of association

In an international research project, a meta analysis of multitrait-multimethod matrices was used to evaluate the effects of characteristics of survey measurement instruments on the quality of survey data. The analyses in this project were done on the basis of polychoric/polyserial correlations. The result of such a study can be used to correct for measurement errors if the correlation in the substantive study is also a polychoric/polyserial correlation. It is, however, doubtful if these quality estimates of survey measurement instruments are appropriate for studies based on the most frequently used Pearson correlations. In this paper, the general approach will be discussed first. After that, the results with respect to validity and reliability estimates, obtained using two different measures of association, will be compared: the polychoric/polyserial correlations and the Pearson correlations. This study suggests that the differences between estimates of the reliability for scales with different numbers of categories are quite large, while all other effects remain approximately the same. The reasons for the differences will also be given. All the results are based on experiments containing questions with respect to satisfaction with life and aspects of life. Thus the study provides estimates of validity and reliability on measures of satisfaction with life and aspects of life which can be used in practice for the prediction and optimization of data quality and for correction of the data with regard to measurement error.