Analysis of multimodal data fusion from an information theory perspective

Inspired by the McGurk effect, studies on multimodal data fusion start with audio-visual speech recognition tasks. Multimodal data fusion research was not popular for a period of time because the capacity of traditional machine learning is limited. Recently, advances in deep learning techniques have provided new opportunities for multimodal data fusion. Powerful deep learning models have the capacity to process high-dimensional and com-plex multimodal data, and multimodal deep learning has the potential to process multi -modal data at the human level. However, there is still a lack of theoretical analytical methods relating data information with model performance. In this work, we propose basic concepts and principles to gain insight into the process of multimodal data fusion from an information theory perspective. We analyze different multimodal data fusion cases, such as redundant, noisy, consistent, and contradictory data fusion. We define the model accuracy upper bound for multimodal tasks and prove that a multimodal model with an extra modal channel can perform better in theory when extra modal data provide more effective infor-mation for prediction. We explicitly inspect the latent representation space and analyze the information loss of the representation space transformation in deep learning for the first time. From a naive example to a multimodal deep learning example, we demonstrate the theoretical analysis method for evaluating a multimodal data fusion model, and the experimental results validate the definitions and principles.(c) 2022 Elsevier Inc. All rights reserved.