For fault detection of gear pair with tooth root crack, it is often used to collect vibration signal of gearbox for analysis and evaluation of health state of gear pair. However, in process of vibration fault detection of fault gear pair, the gear pair doesn't operate in an ideal state due to rotating speed error of gear pair, load torque fluctuation and other unstable working conditions. This more largely affects vibration signal of gear pair to make it difficult to distinguish characteristic frequency of gear pair vibration. Here, the tooth root crack model of gear pair was established according to the potential energy method, the analytical formula for calculating the time-varying meshing stiffness of gear pair after local failure was deduced, the short-time Fourier transform was performed for vibration non-stationary signal of gear pair under variable working conditions to analyze vibration characteristics of gear pair under variable working conditions. The results showed that under stable working conditions, vibration amplitude of gear pair increases with increase in rotating speed and load torque, and its sideband distribution also presents a certain law; under unstable working conditions, after using the short-time Fourier transform, there are obvious shocks in time domain and frequency domain of gearbox vibration signal; the interval between adjacent shock peaks in frequency domain reflects the rotating speed error of gear pair, and the distribution and amplitude of its main frequencies can also reflect fluctuation of rotating speed and torque load during gear pair operating; at the same time, a conclusion obtained under unstable working conditions is consistent with that obtained under stable working conditions, i.e., vibration amplitude of gear pair increases with increase in rotating speed and load torque. © 2022, Editorial Office of Journal of Vibration and Shock. All right reserved.