With the increasing service life of highways, pavement diseases, especially asphalt road cracks are becoming more serious. To detect asphalt pavement cracks quickly and accurately, the application of deep learning has long been a question of great interest in this field. However, existing methods have disadvantages such as slow processing of images, low accuracy, and lack of evaluation of pavement conditions. In this study, a pavement damage detection and evaluation system based on deep learning is constructed, which can quickly and accurately complete the detection and segmentation of cracks and evaluate the degree of pavement damage. Based on the pre-prepared training set, a YOLOv5-based asphalt pavement crack object detection model and a U-net-based asphalt pavement crack semantic segmentation model are trained respectively, and their effectiveness is verified on the test set. In the training, the training set is preprocessed using a combination of different image enhancement methods including histogram equalization, image sharpening, and denoising, then compared by recognition results. Based on the identification results of the above two models, the area of crack is estimated and the damage degree of the pavement is evaluated by the pavement condition index from Highway Performance Assessment Standard (JTG 5210-2018). The results showed that the method performed well in crack detection and segmentation, with 80.2% mAP and 49.3% MioU, and provided a quantitative estimate of pavement condition.