Overcoming Catastrophic Forgetting for Fine-Tuning Pre-trained GANs

The great transferability of DNNs has induced a popular paradigm of "pre-training & fine-tuning", by which a data-scarce task can be performed much more easily. However, compared to the existing efforts made in the context of supervised transfer learning, fewer explorations have been made on effectively fine-tuning pre-trained Generative Adversarial Networks (GANs). As reported in recent empirical studies, fine-tuning GANs faces the similar challenge of catastrophic forgetting as in supervised transfer learning. This causes a severe capacity loss of the pre-trained model when adapting it to downstream datasets. While most existing approaches suggest to directly interfere parameter updating, this paper introduces novel schemes from another perspective, i.e. inputs and features, thus essentially focuses on data aspect. Firstly, we adopt a trust-region method to smooth the adaptation dynamics by progressively adjusting input distributions, aiming to avoid dramatic parameter changes, especially when the pre-trained GAN has no information of target data. Secondly, we aim to avoid the loss of the diversity of the generated results of the fine-tuned GAN. This is achieved by explicitly encouraging generated images to encompass diversified spectral components in their deep features. We theoretically study the rationale of the proposed schemes, and conduct extensive experiments on popular transfer learning benchmarks to demonstrate the superiority of the schemes. The code and corresponding supplemental materials are available at https://github.com/zezeze97/Transfer-Pretrained-Gan.