MSI-RGB Dual-Source Multiscale Fusion-Based Solid Waste Object Detection

This article focuses on the significant challenges faced by single-modal visual sensors in object detection under the fine-sorting process of solid waste. Object detection using a single sensor has inherent limitations due to the lack of comprehensive material and contour information about the object. Therefore, this article proposes a real-time detection network based on the MSI-RGB dual-source multiscale fusion network (DMFNet) and introduces a coarse-to-fine dual-stage band selection (dual-stage BS) method for reducing spectral redundant information. As a case study, we focus on plastic solid waste and build a dual-sensor experimental platform to collect color and hyperspectral images (HSIs) using a face-matrix color camera and a hyperspectral camera. Specifically, the dual-stage BS method is employed to analyze the hyper-spectral data and a small number of spectral bands were selected to generate a multispectral image (MSI) as the first input source for DMFNet. Meanwhile, the RGB image serves as the second input source for DMFNet. First, a spectral image feature extraction backbone (SIFEBackbone) and a multiscale stacking unit (MSU) are designed to extract multiscale features from spectral images fully. Second, a multibranch fusion unit (MBFU) is designed to realize the fusion of multiscale features with the spectral feature maps in the RGB-enhanced feature extraction stage. The experimental results show that dual-stage BS exhibits advantages over other methods, while DMF-YOLOv7, which integrates DMFNet and YOLOv7 detectors, improves the average precision and recall by 7.2% and 5.5%, respectively, compared with the original YOLOv7, and the mAP reaches 95.4% at IOU = 0.5. It effectively reduces the material misrecognition problem and reflects the advantages of the dual-sensor algorithm. The source code is available at: https://github.com/Caicai-D/DMFNet. © 1963-2012 IEEE.