Existing deep learning-based methods for coal gangue identification, when applied in complex environments such as overhead conveyor belts, often suffer from false positives, false negatives, and low accuracy in recognizing small coal gangue objects. To address this problem, an improved deep recognition network model for coal gangue identification is proposed. The aim is to further enhance the performance of the network model while ensuring minimal increase in model parameter volume and complexity, making it more adaptable to real production environments. The model is based on YOLOv5s, which offers a relatively fast speed and high accuracy. In the backbone of YOLOv5s, the CoT module with adaptive force attention mechanism is introduced. This module utilizes contextual information between input keys to guide the learning of dynamic attention matrices, enhancing the visual representation capability and improving the coal gangue detection performance. The Spatial Pyramid Pooling (SPP) module is improved to maintain the receptive field and strengthen the extraction of deep features, resulting in speed improvements. In the prediction part, the decoupled head, which excels in multi-object classification and comprehensive feature map extraction, is employed. Weight coefficients are set to accelerate the learning of highly confident targets, further enhancing the accuracy of coal gangue detection. Experimental results demonstrate that the improved model is easy to train, embed, and achieves high recognition accuracy. It effectively improves the precision, recall, and mean average precision (mAP) of the YOLOv5 model. The model achieves an mAP of 95.5%, a 3.3% improvement compared to the original YOLOv5 model. Its performance surpasses mainstream object detection models such as SSD, Faster R-CNN, YOLOv3, and YOLOv4. In complex environments, the improved YOLOv5 model accurately delineates object boundaries, demonstrating superior detection results compared to other improved YOLOv5 models. Additionally, the model exhibits stronger generalization and robustness, holding significant application value in the intelligent sorting of coal in real-world scenarios.