An agent-based architecture for multimodal interaction

In this paper, an executable generic process model is proposed for combined verbal and non-verbal communication processes and their interaction. The agent-based architecture can be used to create multimodal interaction. The generic process model has been designed, implemented and used to simulate different types of interaction between verbal and non-verbal communication processes: a non-verbal communication process can add and modify content to a verbal communication process, but can also provide a protocol for the (control of the) verbal communication process. With respect to the communication protocol both stimulus-response behaviour and deliberative behaviour have been modelled and simulated. The semantics of the model has been formalized by three-levelled partial temporal models, covering both the material and mental proceses and their relations. (C) 2001 Academic Press.