3D Shape Segmentation Based on Viewpoint Entropy and Projective Fully Convolutional Networks Fusing Multi-view Features

This paper introduces an architecture for segmenting 3D shapes into labeled semantic parts. Our architecture combines viewpoint selection method based on viewpoint entropy, multi-view image-based Fully Convolutional Networks (FCNs) and graph cuts optimization method to yield coherent segmentation of 3D shapes. First, we select iteratively a fixed number of perspectives with the maximum viewpoint entropy from existing viewpoints that can cover the shape's triangles, to maximally and automatically adjust the distance between the viewpoint and the center point of the shape to make sure the shape projected to fill the render window as wide as possible. Second, the image-based FCN is used for efficient view-based reasoning about 3D shape parts. In this process, global features generated by max view pooling are concatenated with every single view's feature in the fully connected layer before upsampling. Then, the multi-view FCN outputs confidence maps per part, which are then input into the projection layer that contains the mapping relationship of every shape's triangles and their projective pixels' positions in the rendered images from selected perspectives. And then, the FCN outputs are projected back onto 3D shape surfaces. and max view pooling is applied to the output of the projection layer so that every triangle of each shape has a unique probability for each label. Finally, graph cuts algorithm is implemented for the final segmentation result.