Speech recognition for command entry in multimodal interaction

Two experiments investigated the cognitive efficiency of using speech recognition in combination with the mouse and keyboard for a range of word processing tasks. The first experiment examined the potential of this multimodal combination to increase performance by engaging concurrent multiple resources. Speech and mouse responses were compared when using menu and direct (toolbar icon) commands, making for a fairer comparison than in previous research which has been biased against the mouse. Only a limited basis for concurrent resource use was found, with speech leading to poorer task performance with both command types. Task completion times were faster with direct commands for both speech and mouse responses, and direct commands were preferred. In the second experiment, participants were free to choose command type, and nearly always chose to use direct commands with both response modes. Speech performance was again worse than mouse, except for tasks which involved a large amount of hand and eye movement, or where direct speech was used but mouse commands were made via menus. In both experiments recognition errors were low, and although they had some detrimental effect on speech use, problems in combining speech and manual modes were highlighted. Potential verbal interference effects when using speech are discussed. (C) 2000 Academic Press.