Fusion-restoration model for industrial multimodal anomaly detection

Industrial anomaly detection based on multimodal data is receiving increasing attention. The application of the feature mapping paradigm represents a prevailing trend. However, the existing feature mapping method is limited by the lack of multimodal fusion, which hinders the comprehensive interaction between RGB and point cloud features. In this paper, we introduce a novel feature reconstruction paradigm called Fusion-Restoration Model (FRM) to ameliorate this problem. A fusion encoder integrates the information of two domains into a fusion embedding. Then, a pair of decoupled decoders independently restore embeddings of the corresponding domains from the fusion embedding. FRM learns nominal feature reconstruction from anomaly-free training samples and detects and localizes anomalies based on the reconstruction residuals in the inference phase. A joint loss that constrains both direction and magnitude is used to enhance the robustness of the reconstruction. Additionally, a semi-frozen training strategy is designed to adapt the batch normalization parameters of the 3D feature extractor to the target industrial dataset. Extensive experiments show that our method achieves effective and efficient multimodal anomaly detection on the MVTec 3D-AD dataset.