Semantic and Optical Flow Guided Self-supervised Monocular Depth and Ego-Motion Estimation

The self-supervised depth and camera pose estimation methods are proposed to address the difficulty of acquiring the densely labeled ground-truth data and have achieved a great advance. As the stereo vision could constrain the predicted depth to a real-world scale, in this paper, we study the use of both left-right pairs and adjacent frames of stereo sequences for self-supervised semantic and optical flow guided monocular depth and camera pose estimation without real pose information. In particular, we explore (i) to construct a cascaded structure of the depth-pose and optical flow for well-initializing the optical flow, (ii) a cycle learning strategy to further constrain the depth-pose learning by the cross-task consistency, and (iii) a weighted semantic guided smoothness loss to match the real nature of a depth map. Our method produces favorable results against the state-of-the-art methods on several benchmarks. And we also demonstrate the generalization ability of our method on the cross dataset.