End-to-End Deep Neural Network for Illumination Consistency and Global Illumination

In this study, we propose a real-time method for realizing illumination consistency and global illumination in augmented reality (AR). The proposed method uses pix2pix, which is a generative adversarial network (GAN) for image-to-image translation. The network takes an image with k channels as the input, and attempts to generate reflections and shadows of a virtual object corresponding to the illumination condition. We also propose an approach for improving the applicability of the method by combining RGB information with geometric information (normal and depth) as the network input. For evaluating the proposed method, we created a synthetic dataset by using Unreal Engine 4, which can render computer graphics (CG) images with global illumination. The results of an experiment indicated that although generated images were not completely the same as the ground truth, the proposed method reproduced natural-looking reflections and shadows of a virtual object.