Gated weighted normative feature fusion for multispectral object detection

Multispectral image pairs can provide independent and complementary information to more comprehensively describe detection targets, thereby improving the robustness and reliability of object detectors. The performance of an object detector depends on how cross-modality features are extracted and fused. To exploit the different modalities fully, we propose a lightweight yet effective cross-modality feature fusion approach named gated weighted normative feature fusion. In the feature extraction stage, our proposed dual-input backbone network can extract richer and more useful features. In the feature fusion stage, the fusion module can eliminate redundant features, dynamically weigh the importance of two image features, and further normalize fused features. Experiments and ablation studies on several publicly available datasets demonstrate the effectiveness of our method. Our proposed method achieved better performance in terms of mAP50 with 80.3%, mAP with 41.8%, and mAP50 with 98.0%, mAP with 68.0% on the FLIR and LLVIP datasets, respectively. In particular, the inference speed of our method is twice as fast as the current state-of-the-art method.