Joint Semantic and Instance Segmentation in 3D Point Cloud Based on Transformer

Semantic and instance segmentation in 3D point cloud have made tremendous progress in recent years. In most current methods, semantic and instance segmentation are studied separately, but fall to integrate them. In addition, the irregular domain and lack of ordering make it challenging to design deep neural networks for point cloud learning. In this paper, we propose a joint segmentation method based on Transformer that able to make the two tasks take advantage of each other, leading to a win-win situation. The novel Transformer architecture is used to capture long-short contexts and demonstrates higher segmentation performance. Specifically, for each query point, we use two different key sampling strategies: nearby dense points sampling and distant sparse points sampling. Then, matrix multiplication with query results in two different attention maps, which enables the model to enlarge the effective receptive field. Also, to make the two tasks take advantage of each other, we first combine the semantic perception abstracted from the high-dimensional semantic features with the instance features to enhance instance segmentation performance. Meanwhile, the k-nearest neighbor (KNN) clustering method is used in the instance embedding to find a fixed number of adjacent points for each point in the instance clustering space. The points of the same class are close to each other, the points of different classes are far away from each other, so as to realize the semantic segmentation model of instance embedding. Extensive experiments demonstrate the effectiveness and superiority of our method on S3DIS and ShapeNet datasets.