Arbitrary Viewpoint Visualization for Teleoperated Hydraulic Excavators

In this paper, we propose a visualization system for the teleoperation of excavation works using a hydraulic excavator. An arbitrary viewpoint visualization system is a visualization system that enables teleoperators to observe the environment around a machine by combining multiple camera images. However, when applied to machines with arms (such as hydraulic excavators), a part of the field of view is shielded by the image of the excavator's arm; hence, an occlusion occurs behind the arm. Furthermore, it is difficult for teleoperators to understand the three-dimensional (3D) condition of the excavating point because the current system approximates the surrounding environment with a predetermined shape. To solve these problems, we propose two methods: (1) a method to reduce the occluded region and expand the field of view, and (2) a method to measure and integrate the 3D information of the excavating point to the image. In addition, we conduct experiments using a real hydraulic excavator, and we demonstrate that an image with sufficient accuracy can be presented in real-time.