Text-driven Face Image Generation and Manipulation via Multi-level Residual Mapper

Although generative adversarial networks (GANs) have achieved great success in face image generation and manipulation, discovering meaningful directions in the latent encoding space of GANs to manipulate semantic attributes of faces is a great challenge in computer vision. The solution to this challenge requires a large amount of labeled data and several hours of network fine-tuning. However, many difficulties are confronted in the collection and annotation of similar data, such as great technical barriers and high labor costs. Recent studies have been attempting to overcome the problem of lacking labeled data by pre-trained models. Such efforts are proved capable of accomplishing the above task, but the accuracy of the manipulation and the authenticity of the results cannot meet the needs of real face editing scenarios. To address these problems, this study encodes the image and text descriptions into a shared latent encoding space by leveraging the joint representation capability of contrastive language-image pre-training (CLIP). With carefully designed network structures and loss functions, the proposed framework can accurately recognize relevant face attributes and learn a residual mapping network. The network can predict the latent code residuals according to image and text description codes and perform high-quality image generation and manipulation by the pre-trained model StyleGAN2. Extensive experiments demonstrate the superiority of the proposed approach in terms of manipulation accuracy, visual realism, and irrelevant attribute preservation. © 2023 Chinese Academy of Sciences. All rights reserved.