Estimating Uncertainty in Task-Oriented Dialogue

Situated multimodal systems that instruct humans need to handle user uncertainties, as expressed in behaviour, and plan their actions accordingly. Speakers' decision to reformulate or repair previous utterances depends greatly on the listeners' signals of uncertainty. In this paper, we estimate uncertainty in a situated guided task, as leveraged in nonverbal cues expressed by the listener, and predict that the speaker will reformulate their utterance. We use a corpus where people instruct how to assemble furniture, and extract their multimodal features. While uncertainty is in cases verbally expressed, most instances are expressed non-verbally, which indicates the importance of multimodal approaches. In this work, we present a model for uncertainty estimation. Our findings indicate that uncertainty estimation from nonverbal cues works well, and can exceed human annotator performance when verbal features cannot be perceived.