Self Attention Guided Depth Completion using RGB and Sparse LiDAR Point Clouds

We address the problem of completing per pixel dense depth map using a single RGB image and the sparse point cloud of the scene. Depth prediction from RGB image is a hard problem and while dense point clouds obtained from LiDAR sensors can be used in addition to RGB image, the cost of such sensors is a significant barrier. Having LiDAR sensors which capture sparse point clouds is a reasonable middle ground. We propose a novel architecture which incorporates geometric primitives and self attention mechanisms, to improve the prediction. The motivation of self attention is to capture the correlations between scene and object elements, e.g. between the right and left window of car, early on in the network. While that for using geometric primitives is to have a high level clustering cue to enable the network to exploit similar correlations. In addition, we enforce complimentarity in the predictions made with RGB and sparse LiDAR respectively, this forces the two corresponding branches to focus on hard areas which are not already well predicted by the other branch. With exhaustive experiments on KITTI depth completion benchmark, NYU v2 and Matterport3D we show that the proposed method provides state-of-the-art results.