Garbage image classification often suffers from three aspect challenges: complex image background, same-shape category, and low-quality image. The existing machine vision methods have excellent learning capabilities. However, they require powerful computational resources. In this work, an efficient garbage image classification network (GScbamKL-Net) is proposed in this work to address the problems mentioned. The proposed network is designed from the following three aspects. First, the new network unit with group convolution and channel shuffle is designed. This unit can significantly reduce the number of parameters of the model and achieve good performance. Second, the CBAM attention mechanism, which can extract key features by weighting the output features in space and channel, is added to the network unit. Furthermore, the LeakyReLu function is introduced as the activation function model. A label smoothing function is constructed as the loss function. It can mitigate the errors and effects of sample imbalance and obtain a good nonlinear transformation effect. The normal garbage images and garbage images using infrared imaging technology were tested respectively. Experimental results show that the proposed GScbam-Net has excellent classification performance while maintaining its lightweight.