Look Who's Talking: Evaluating the Utility of Interventions During an Interactive Think-Aloud

We report the results of a study comparing two concurrent think-aloud approaches for usability testing: the classic think-aloud (CTA) and an interactive think-aloud (ITA). The think-alouds were compared in respect of task performance and usability problem data. We also analyse the utility of the interventions used within the ITA in eliciting useful participant utterances. The most useful interventions were those focused on seeking explanations and opinions; these generated more utterances about user difficulties. Requests for clarifications, particularly about actions, resulted in fewer useful utterances: participants responded with simple procedural descriptions. In comparing the CTA and ITA, we found no differences in the number of successfully completed tasks, but the ITA did elongate the test session. The ITA led to the detection of more usability problems overall, and a greater number of causal explanations. However, the ITA produced more low-severity problems than the CTA.