Image Registration Between Real Image and Virtual Image Based on Self-supervised Keypoint Learning

A digital twin is a next-generation technology that connects virtual and physical environments into a single world. Although the virtual environment of a digital twin models the real world, the technology used to match the real world with the virtual environment has yet to be studied. The existing deep-learning-based image registration methods aim to extract feature points and descriptors and show a good registration performance in real images. However, these methods are difficult to apply in an actual digital twin environment because 3D and real 2D images have a significant difference in terms of the external and physical characteristics of the image itself. In this paper, we propose a deep learning model that self-learns the difference between virtual and real environments using a generative-adversarial network and self-supervised learning. Image registration between virtual environments with real-world images is a new method that has not been previously achieved, and we have demonstrated experimentally that the proposed method is applicable to various virtual environments and real-world image matching.