Seemingly minor changes to a questionnaire can make a big difference to mean scores: a cautionary tale

Background It is not unusual for researchers to make apparently minor modifications to existing instruments without checking if this alters psychometric properties. Method Equivalent items on child mental health from two different versions of the Rutter parent questionnaire were compared: items from the standard version and from a modified version. The parents of 400 children aged 5-7 years were randomised into two groups: each group completed one version of the Rutter as well as an independent measure of psychopathology (the Strengths and Difficulties Questionnaire; SDQ). Results The mean psychopathology scores of the two groups were comparable according to the SDQ but differed markedly between the two Rutter versions, principally because of changes in the response categories. Nevertheless, the validity of the two versions of the Rutter was similar as judged from Rutter-SDQ correlations. Conclusion Seemingly minor changes in the wording of a measure can have a major impact on mean scores, thereby making it harder to compare or combine the results of studies using the original and the modified measure.