Deep Multimodal Learning and Fusion Based Intelligent Fault Diagnosis Approach

Industrial Internet of Things (IoT) connecting society and industrial systems represents a tremendous and promising paradigm shift. With IoT, multimodal and heterogeneous data from industrial devices can be easily collected, and further analyzed to discover device maintenance and health related potential knowledge behind. IoT data-based fault diagnosis for industrial devices is very helpful to the sustainability and applicability of an IoT ecosystem. But how to efficiently use and fuse this multimodal heterogeneous data to realize intelligent fault diagnosis is still a challenge. In this paper, a novel Deep Multimodal Learning and Fusion (DMLF) based fault diagnosis method is proposed for addressing heterogeneous data from IoT environments where industrial devices coexist. First, a DMLF model is designed by combining a Convolution Neural Network (CNN) and Stacked Denoising Autoencoder (SDAE) together to capture more comprehensive fault knowledge and extract features from different modal data. Second, these multimodal features are seamlessly integrated at a fusion layer and the resulting fused features are further used to train a classifier for recognizing potential faults. Third, a two-stage training algorithm is proposed by combining supervised pre-training and fine-tuning to simplify the training process for deep structure models. A series of experiments are conducted over multimodal heterogeneous data from a gear device to verify our proposed fault diagnosis method. The experimental results show that our method outperforms the benchmarking ones in fault diagnosis accuracy. © 2021 Journal of Beijing Institute of Technology