An experiment on measuring belief: asking the same question in different ways

It is common knowledge that asking a question in different ways can produce different responses. But do different objective response alternatives for the same question produce different results? A questionnaire stating a proposition about the risk of an accident in a nuclear power plant was given to 478 relatively well-educated Japanese representing an even distribution of males and females and divided across various cities, ages, salaries, and occupations. Each respondent was asked to respond using each of four objective methods: (1) simple true or false, (2) true or false with probability weightings, (3) true or false or "no idea", and (4) surely true, probably true, ambivalent, probably false, and surely false. Response alternatives in method 3 are a simple version of the categories used in Dempster-Shafer belief theory and those of method 4 are a simple example of fuzzy sets. Not surprising to the statistician, results showed large numerical differences between forced-choice T–F frequencies and the "subjective probability" T–F response averages (i.e., between methods 1 and 2). Methods 3 and 4 could be shown to yield comparable results to method 2 by means of a simple transformation for allocating the central uncertainty weightings to either true or false. Chi-square tests showed no significant differences between subcategories within each of the attributes of City of Residence, Gender, Age, Salary, and Occupation.