Traditional methods for coal gangue sorting exhibit low efficiency, significant safety hazards, and limited applicability. Existing machine vision-based coal gangue image recognition methods struggle to balance model recognition speed and accuracy. In response to these challenges, this paper first utilizes the improved MSRCR algorithm to process images, enhancing the dark areas of coal gangue images while ensuring uniform brightness enhancement and image clarity. Furthermore, a novel lightweight coal gangue recognition method is proposed based on YOLOv8n, aiming to reduce data redundancy and improve recognition accuracy. Experimental results demonstrate that the improved lightweight model has a computational load of 7.1 GFLOPs, representing only 86.6% of the original model. The model detection rate is 73 fps, a 17 fps improvement over the original model. The accuracy, recall rate, and average precision reach 98.1%, 97.6%, and 98.9%, respectively, with improvements of 1.1, 0.9, and 0.5% points over the original model. The missing detection phenomenon is avoided effectively, and the accuracy and portability of the model are improved.