DualStyle3D: Real-time Exemplar-based Artistic Portrait View Synthesis Based on Radiance Field

To generate novel views of artistic portraits from a source face image and a reference style exemplar in real-time, we propose a novel structure, namely DualStyle3D. In our DualStyle3D, we introduce a novel "encoder-decoder" structure to reduce the generation time cost by directly taking face images and exemplars as inputs while eliminating the time-consuming 3D-GAN inversion process. Moreover, to manage the style of the generated portrait, we propose a StyleAdaption3D module to control the color tune and the structure of the synthesized artistic portrait based on three different granular features. Furthermore, we employ a progressive training strategy to distill knowledge from both pretrained 3D-GAN and 2D artistic GAN to achieve robust convergence. Our experiments demonstrate that the time required to synthesize an artistic portrait from an input portrait using our DualStyle3D is 0.08% of CIPS-3D, 0.16% of 3DAvatarGAN, and 30.56% of DeformToon3D on Nvidia RTX 3090, showcasing its potential for real-time, exemplar-based portrait style transfer.