Comparison of Item Formats: Agreement vs. Item-Specific Endpoints

The current study was an investigation of the extent to which items constructed with an agreement format were affected by acquiescence bias relative to responses to a matched set of items designed with item-specific formats. If an acquiescence bias existed, the expectation was that the magnitude of responses to agreement items would be consistently greater than those to the matched item-specific versions. Two hundred respondents completed a survey in which they described and rated a recent interaction with an auto insurance website. Half of the respondents provided ratings using items with an agreement format (Version 1 of the survey), and the other half provided ratings on items matched for content but with item-specific endpoint anchors rather than generic disagree/agree endpoints (Version 2 of the survey). The two groups of respondents were matched on their ratings of overall experience, which was the same for both groups. Of 14 comparisons of matched agreement and item-specific formats, 12 (86%) were not statistically significant. For the other two items, the means of the agreement versions were significantly lower than those for the item-specific versions (p < 0.05), which is the opposite of the expected result if agreement items were affected by an acquiescence bias. Thus, despite published concerns about the potential effect of acquiescence bias on responses to items using an agreement format, the results of this study found no evidence for such bias, at least in the context of user experience research. Both agreement and item-specific formats appeared to work equally well.