Compound fault diagnosis is a hard problem in mechanical fault diagnosis. When compound fault occurs in a gearbox, strength imbalances between fault signal components in the measured compound fault vibration signal will appear usually, which is affected by the transfer path, measuring point arrangement and so on. Especially when the gearbox runs with variable rotational speed, the fault feature is of time-varying characteristics. Therefore, aiming at compound fault diagnosis for gearboxes operating with variable rotational speed, a frequency-domain adaptive time-varying filtering method was proposed. In the proposed method, an adaptive time-varying filter was designed in frequency-domain, and the time-varying filtered signal carrying gear fault characteristics was extracted from the compound fault signal of gearbox by using the adaptive time-varying filter, which in turn by the envelope order spectrum analysis, the fault features of the gears were extracted. Meanwhile, by the envelope order spectrum analysin on the residual signal, which is the difference between the gearbox compound fault signal and time-varying filtered signal, the fault features of the bearings were extracted. Simulations and application examples indicate that the proposed adaptive time-varying filtering method can effectively separate the fault features of the of gears and rolling bearings running with variable rotational speed. © 2019, Editorial Office of Journal of Vibration and Shock. All right reserved.