Monocular Depth from Small Motion Video Accelerated

We propose a novel four-stage pipeline densely reconstructing depth from video sequences with small baselines with the goal of being fast. This work is particularly motivated by the high frame rate (HFR) ability of many modern smartphones which leads to smaller inter-frame motion and less motion blur, but with more image noise as the sensitivity is adjusted to compensate for the increased shutter speed. While small baselines lead to larger uncertainties in depth estimations, they allow for easier point tracking and assumptions of brightness constancy hold more true. In our pipeline we make use of the sub-pixel precision of direct photometric bundle adjustment to reduce the number of tracked points required to estimate an accurate pose. We show that by considering pixel intensities across the entire video, it is more robust to image noise. Instead of using the exhaustive plane sweeping approach of existing small baseline methods, dense depth maps are calculated efficiently using an algorithm inspired by PatchMatch[2, 3]. Instead of a stereo matching error, our algorithm minimizes the variance over multiple frames with a robust mean estimation. We present a detailed, quantitative comparison between our method and the most recent small baseline method from Ha et al. [7] for speed and quality across and spectrum of baselines and sequence sizes using a synthesized, photorealistic dataset. The results suggest that our method has the ability to cope with a wider range of baselines and sequence sizes. We also compare qualitative results on real small motion clips from Ha et al. [7] in addition to our own, and show that our method outputs dense depth maps of similar or better quality and at least 10 times faster.