Mixed-initiative Dialog Management for Speech-based Interaction with Graphical User Interfaces

Controlling graphical user interfaces (GUI) by speech is slow, but proves useful for disabled persons with limitations in operating mouse and keyboard. We present conversation-and-control, a new approach for using speech as input modality for GUIs, which facilitates direct manipulation of widget functions by spoken commands. Our approach is based on a command language, which provides a unique command for each specific widget function. For managing the interaction we propose a mixed-initiative dialog model, which can be generated from widget properties. Using heuristics for inferring the meaning of a recognition result and having the ability to ask clarification questions, our approach avoids the rejection of recognition errors. We hypothesized that conversation-and-control allows for shorter task completion times than conventional command-and-control approaches, due to a reduction of the average number of required commands. The results of a user experiment, which we present and discuss, indicate a 16.8% reduction of task completion time achieved by our approach.