Direct and Wordgraph-Based Confidence Measures in Dialogue Annotation with N-Gram Transducers

Dialogue annotation is a necessary step for the development of dialogue systems, specially for data-based dialogue strategies. Manual annotation is hard and time-consuming, and automatic techniques can be used to obtain a draft annotation and speed up the process. The presentation of the draft annotation with confidence levels on the correctness of every part of the hypothesis can make even faster the supervision process. In this paper we propose two methods to calculate confidence measures for an automatic dialogue annotation model, and test them for the annotation of a task-oriented human-computer corpus on railway information. The results show that our proposals have a similar behaviour and that they are a good starting point for incorporating confidence measures in the dialogue annotation process.