Improving and Understanding Clarifying Question Generation in Conversational Search

Conversational information-seeking systems (CISs), such as chatbots and virtual personal assistants, encounter difficulty when processing ambiguous user requests (URs) and generate an accurate response, especially when multiple search results match the given request. As a result, machine-generated clarifying questions (CQs) can be used to refine the user's intent and provide a more precise answer to the initial request. In this paper, we introduce a CIS that can identify the need for clarification in URs and, when necessary, generating appropriate CQs based on the most relevant answers generated from web search results leveraging our novel modular approach, otherwise, the system directly provides the most likely answer to the user. Experimental results on our enhanced version of the ClariQ dataset show the effectiveness of generating relevant and varied CQs, as evaluated by automatic evaluation metrics for fluency and informativeness, BLEURT and Distinct-N. Additionally, our experimental results are comparable to or outperform previous approaches in terms of traditional NLG evaluation metrics, such as BLEU, ROUGE, and METEOR. Finally, we conducted a user study assessing the CQs on five key aspects: grammaticality, on-topic, specificity, new information, and narrow down, which revealed their adequacy. Moreover, our comprehensive analysis identified correlations among these quality aspects.