Evaluating Coherence in Open Domain Conversational Systems

We propose a method for evaluating coherence between user utterances and those generated from open domain conversational systems. Our aim is to make it possible for such systems to ascertain whether utterances generated from them are appropriate to the context before generation so that possible breakdown in conversation arising from inappropriate utterances can be avoided. In our method, we train a classifier that distinguishes a pair of a user utterance and that generated from a system coherent or incoherent by using various pieces of information related to dialogue exchange, such as dialogue acts, question types, and predicate-argument structures. Experimental results show that our method significantly outperforms the baseline, confirming its effectiveness.