An Efficient Diminished Reality Approach Using Real-Time Surface Reconstruction

Augmented reality applications become more and more important to enhance our daily life and workflows. The possibility of showing additional virtual content in a camera stream is helpful for many different use cases like room planning where such applications can offer a simple and intuitive visualization. However, augmented reality applications can suffer from the interference by real objects that may disrupt the user experience. In recent years, there has been research regarding the removal of real objects from camera streams by applying diminished reality techniques. Current approaches are generally limited to flat objects, video streams with little camera movement, or can only remove objects in front of simple and mostly planar backgrounds. In our approach, we show a robust and efficient way to remove a selected 3D object from the camera stream visually. The removal is based on a dense 3D reconstruction of the physical environment stored in a voxel grid that can be created and extended on-the-fly. Hereby, an undesired object can be replaced by a background rendered from the reconstruction allowing for more complex environments than previous approaches. Furthermore, remaining holes by the removal of the object are removed applying an inpainting approach. Finally, we apply color correction to get a seamless transition between the virtual content and the camera image.