In order to achieve accurate, rapid and automatic detection of high-speed railway defective fasteners, an adaptive vision detection algorithm for high-speed railway fastener was proposed based on image processing technology. Aiming at the particularity of high-speed railway fastener images, the improved LBP (local binary pattern) operator was used to extract the salient features of fastener. Based on the prominent feature maps of fastener, the template matching algorithm was used to obtain the precise position of fastener region in the original image, and then get the sub-map of fastener and use the position information of fastener to verify the localization result; The difference between two adjacent sub-maps was used as the judgment basis, if the difference was greater than the preset threshold, the corresponding fasteners were judged as defective fasteners. The detection algorithm was applied to the real fastener image provided by the track maintenance division. The results show that the adaptive fastener detection algorithm proposed in this paper performs worst on rainy days, with a correct detection rate of 96% and a false detection rate of 0.50%; It performs best on sunny days, with a correct detection rate of 100% and a false detection rate of 0.22%; It achieves a comprehensive correct detection rate of 99% and a comprehensive false detection rate of 0.33% under different weather, lighting and environment. © 2020, Editorial Department of Journal of Southwest Jiaotong University. All right reserved.