Aiming at the limited fault state data in industrial practice and guaranteeing the effect of fault diagnosis using deep learning, a novel intelligent fault diagnosis method using images under small samples of helical gears is proposed. The vibration signals collected by a three-channel sensor are processed by the piecewise aggregation approximation (PAA) to construct a RGB Gramian angular field (GAF) image, and the size of the obtained image is reduced by the bicubic interpolation downsampling. Then, an improved sample expansion method of the Wasserstein generative adversarial network with the gradient penalty (WGAN-GP) based on the deep feature fusion of dual-branch discriminator is proposed, and a large number of high-quality GAF image samples are constructed. Next, an improved fault diagnosis model of ConvNeXt with the inverted triangle channel distribution (ITCD-ConvNeXt) is proposed to avoid the overfitting and enhance the diagnosis effect by refining the number of input channels at each stage. Finally, four experiments under low speed with load, low speed with no load, high speed with load and high speed with no load are designed to prove the effectiveness of all the proposed methods. It can be seen that the RGB-GAF image have the obvious advantages over the single-channel GAF images in terms of the data feature expression. The minimum and average values of Frechet inception distance (FID) of the proposed sample expansion model are smaller than those of the compared methods. The size of the proposed fault diagnosis model is reduced to 2.4% of the original model, and after the sample expansion, all the accuracy, precision, recall and F1-score values under four conditions exceed 90%. © 2024 Elsevier Ltd