Surface damage on roads can significantly affect driving comfort, easily leading to traffic accidents and compromising road safety. Currently, the most common detection methods rely on manual inspections or road inspection vehicles to collect road information. However, these methods are not only inefficient but also costly, hindering widespread adoption. Therefore, there is an urgent need for a fast and low-cost road detection method. Drones offer a wide coverage during image capture without disrupting traffic, thus, this paper selects drone-captured images as the dataset and proposes an improved RT-DETR road defect detection model using Adown and RepNCSPELAN. The Adown module helps capture features at different levels in the input data, reducing spatial dimensions during downsampling while retaining essential features. RepNCSPELAN is a neural network module designed to capture long-distance dependencies, extract features with different receptive fields, and enhance feature diversity and expressive power through effective layer aggregation operations. Finally, experimental results demonstrate that the improved model achieves an mAP50 of 72.3%, significantly enhancing detection accuracy and outperforming all models tested. Moreover, compared to the original RT-DETR model, the parameter and computation quantities of the model have decreased by 54.4% and 53.5%, respectively.