Prolog To: Audiovisual Information Fusion in Human-Computer Interfaces and Intelligent Environments: A Survey Introduction

The models and algorithms, used to combine information from multimodal sensors, are restricted to audio-visual modalities, though fusion techniques can be generally applied to other modalities. Fusion strategies have been classified into five categories that include signal enhancement and sensor level fusion, feature level fusion, classifier level fusion, decision level fusion, and semantic level fusion. Feature level fusion or the data to decision fusion scheme, correlates multiple modalities, as found in automatic speech recognition, person tracking, and affect analysis. Classifier level fusion or intermediate fusion are generally found where hidden Markov models (HMM) and dynamic Bayesian networks (DBN) are used to model individual streams. The feature vectors are processed separately, and then the information is fused within the classifier. Decision level fusion or late fusion, combines probability scores or likelihood values from separate unimodal classifiers to arrive at a combined decision.