Upsampling Range Camera Depth Maps Using High-Resolution Vision Camera and Pixel-Level Confidence Classification

We consider the problem of upsampling a low-resolution depth map generated by a range camera, by using information from one or more additional high-resolution vision cameras. The goal is to provide an accurate high resolution depth map from the viewpoint of one of the vision cameras. We propose an algorithm that first converts the low resolution depth map into a depth/disparity map through coordinate mappings into the coordinate frame of one vision camera, then classifies the pixels into regions according to whether the range camera depth map is trustworthy, and finally refine the depth values for the pixels in the untrustworthy regions. For the last refinement step, both a method based on graph cut optimization and that based on bilateral filtering are examined. Experimental results show that the proposed methods using classification are able to upsample the depth map by a factor of 10 x 10 with much improved depth details, with significantly better accuracy comparing to those without the classification. The improvements are visually perceptible on a 3D auto-stereoscopic display.