Face sketch-to-photo transformation with multi-scale self-attention GAN

In this study, we investigate the sketch-to-photo problem, which currently poses a significant challenge in the field of computer vision. A large number of GAN-based encoder-decoder methods have been proposed for image transformation, inspired by the pix2pix model; however, these methods do not produce satisfactory results for photo generation, due to the fact that (1) they miss detailed information of input images because of a single-scale convolution operator in the shallow encoder layers, and (2) they fail to learn long-range dependencies in the deep encoder layers. To better handle these challenges, we present an approach that follows a "divide and conquer" strategy. Our method combines the advantages of a multi-scale convolutional neural network and an attention mechanism and applies these two modules to different encoder layers. Additionally, by optimizing a well-designed loss function, the complex correlations between the sketch and the photo can be calculated. Experimental results show that our method is able to generate high-quality photos from sketch images, and qualitative and quantitative analysis demonstrates its effectiveness and superiority over state-of-the-art models. This work paves a path to replace the traditional encoder structure with the "divide and conquer" strategy to handle image transformation tasks. (C) 2020 Elsevier B.V. All rights reserved.