Due to the influence of manufacturing process and external factors, there will be some undesired defects on the steel surface, which seriously affects the lifetime of steel, and the traditional surface defect detection efficiency and speed are not satisfactory. Therefore, based on the industrial scenario of low computational force, this study proposed a lightweight and efficient defect detector called LE-YOLOv5. First, we utilize ShuffleNetv2 as the backbone of the model, which greatly reduces the number of parameters. Second, we propose a CBMM module to expand the global receptive field of the model in the initial down sampling stage, which facilitates the model in capturing global information. Third, we also propose a parallelized C2N module for the detection of small defects. Finally, we design a global coordination attention (GCA) to efficiently connect position and spatial information from the feature map. Numerous experimental results demonstrate that LE-YOLOv5 has a highly superior overall performance, reaching 79.1% mean Average Precision (mAP) on the NEU-DET dataset while inferring an image on the CPU in 196.1 ms, which is 5% and 1.5% improved mAP compared to YOLOv5M and YOLOv5L, respectively. At the same time, under the condition that the inference time for an image on a CPU-dependent low computing power force remains the same, the accuracy has improved by 5.3% compared to YOLOv8. It provides excellent potential for defect detection of steel in industrial environment.