3D Semantic Scene Segmentation with Multi-View RGB-D Images in Indoor Environments

This paper proposes a novel model for 3D semantic scene segmentation in indoor environments. Existing models for 3D semantic scene segmentation use either only 3D geometric features of the scene point cloud or only 2D visual features of RGB color images. We overcome the limitations of existing models and improve the performance of 3D semantic scene segmentation by proposing a multimodal 3D semantic scene segmentation model to use both 3D geometric features of the scene point cloud and rich 2D visual features of multi-view color images. The proposed model overcomes the point sparsity problem by using the dense point cloud obtained from multi-view depth images and uses an adaptive point feature extractor to extract 3D geometric features representing the local structural characteristics of points. Moreover, the model adopts a unique early fusion strategy to fuse the 2D-3D features. Based on experiments conducted using the ScanNet benchmark dataset, we demonstrate the effectiveness and superiority of the proposed model. © ICROS 2023.