Displacement monitoring has always been an important branch of structural health monitoring. Among them, multi-point displacement monitoring can not only reduce the monitoring costs, but also improve the monitoring efficiency. Therefore, this paper proposes a multi-point displacement monitoring technology based on full convolutional neural network (FCN) and smartphone. It combines machine vision and deep learning to extract the object information for displacement monitoring. First, a dataset with 400 images was collected, and this dataset contains only one category: mark. Second, the dataset was trained by FCN to obtain a detection model, which can perform pixel-level segmentation on multiple masks in the image, and the average mean intersection over union (MIoU) is 0.9774. Third, in order to verify the feasibility of the proposed method to monitor multi-point displacement, a validation experiment was completed, and the information of four masks in the image can be extracted simultaneously. Fourth, sensitivity analysis was completed. The test results showed that this method can achieve 0.5 mm monitoring for the mark at a distance of 2.5 m. Finally, error analysis was completed. The test results showed that the monitoring errors of mark at 2.5 m and 5 m are less than 1%. The monitoring errors of mark at 7.5 m and 10 m are 1.696% and 1.997% respectively. Although the monitoring error increases with the increase of distance, it can still meet the needs of engineering practice. In addition, this paper uses smartphone to collect the images of mark, it further reduces the monitoring costs, and improves the convenience of monitoring.