Cross-Modal 360° Depth Completion and Reconstruction for Large-Scale Indoor Environment

In a large-scale epidemic, reducing direct contact among medical personnel, attendants and patients has become a necessary means of epidemic prevention and control. Intelligent vehicles and mobile robots in the hospital environment, such as disinfection vehicles, logistics vehicles, nursing robots, and guiding robots, play an important role in improving the operational efficiency of the medical system and promoting epidemic prevention and governance. Powerful capabilities of environmental spatial perception and reconstruction are the keys to accurate localization, navigation, and obstacle avoidance for intelligent vehicles and autonomous robots in such operations. Omnidirectional perception is becoming increasingly important and proliferative in autonomous vehicles and robots since its wide field of view significantly enhances the perception ability. However, the lack of dense and accurate 360 degrees depth datasets has brought the challenge to the omnidirectional perception. In this paper, we propose a depth-sensing and reconstruction system to address this challenge in the large-scale indoor environment. First, we design an omnidirectional depth completion convolutional neural network model, in which a spherical normalized convolutional and the unit sphere area-based loss are introduced to extract features from cross-modal omnidirectional input with unequal sparsity and deal with the imbalanced data distribution and distortion in the panoramic input. In addition, we present a 3D reconstruction system by integrating our depth completion into omnidirectional localization and dense mapping. We evaluate our method on 360D large-scale indoor datasets and real-world sequences of a challenging hospital scene. Extensive experiments show that the proposed method outperforms the other state-of-the-art (SoTA) approaches in terms of depth completion and 3D reconstruction.