The number of broken viscose filaments is an important index to judge the quality of viscose filaments. According to the standard, the number of filaments of 10,000 meters of fuzzes is less than 10 for excellent products, and less than 30 for qualified products. In order to improve the accuracy of the detection equipment, this paper starts with the research on the detection algorithm of broken viscose filaments, and proposes an improved algorithm of broken viscose filaments detection based on YOLOv8, which aims to solve the problems of misdetection and missing detection of broken viscose filaments. Firstly, this paper selects Swin Transformer with window-based self-attention mechanism to optimize the feature fusion network to enhance the capturing ability of the model's effective information. Secondly, considering the difficulty of real-time detection of the large-volume model, this paper introduces MobilenetV3 to lightweight the model. Finally, the advance of the proposed method was verified by comparison experiment and ablation experiment. The experimental results show that the detection speed of the improved model can reach 173.76m/min, and the mAP can reach 95.4%, which not only realizes the real-time detection, but also improves the recognition rate of the broken viscose filaments, and better meets the detection requirements of viscose long silk.