3D reconstruction and segmentation system for pavement potholes based on improved structure-from-motion (SFM) and deep learning

Traditional pothole detection based on two-dimensional images lacks three-dimensional (3D) quantitative information such as depth and volume, although the high accuracy. In addition, existing 3D collection and reconstruction equipment based on depth cameras and lasers are expensive and difficult to operate. To solve the above problems, a low cost and automatic 3D reconstruction and segmentation system for potholes is proposed in this study. The system consists of a pavement pothole Structure-from-motion (PP-SFM) and a 3D point cloud segmentation network for potholes. Firstly, a point cloud reconstruction method called PP-SFM is proposed to reconstruct the easily obtained multi-view 2D potholes images. In addition, to a certain extent, the point cloud sparse problem is solved by the proposed PP-SFM, and the stereo display of potholes is realized. Next, Trans-3DSeg is developed based on an improved 3D segmentation network modified by the transformer module to ealize effective segmentation of 3D point cloud data of potholes. The accuracy of the improved system is 93.44%, and the F1-score is 92.58%. In addition, the precision-recall (P-R) curve is near the upper right. Comparative experiments show that the proposed system has better segmentation performance. Compared with PointNet++, PointRCNN and PointCNN, the segmentation accuracy are increased by 4.13%, 2.96% and 3.17%, respectively. The F1-score are increased by 2.41%, 2.43% and 2.93%, respectively. The proposed system requires only a few multi-view images taken from ordinary high-definition cameras, which is a high accuracy and low cost method for 3D reconstruction and segmentation of pavement potholes.