6DoF Vehicle Pose Estimation Using Segmentation-Based Part Correspondences

Over the last years, pixel-wise analysis of semantic segmentation was established as a powerful method in scene understanding for autonomous driving, providing classification and 2D shape estimation even with monocular camera systems. Despite this positive resonance, a way to take advantage of this representation for the extraction of 3D information solely from a single-shot RGB image has never been presented. In this paper we present a full-fledged six degree-of-freedom vehicle pose estimation algorithm, demonstrating that a segmentation representation can be utilized to extract precise 3D information for non-ego vehicles. We train a neural network to predict a multi-class mask from segmentation, defining classes based on mechanical parts and relative part positions, treating different entities of a part as separate classes. The multi-class mask is transformed to a variable set of key points, serving as a set of 2D-3D correspondences for a Point-n-Perspective-solver. Our paper shows not only promising results for 3D vehicle pose estimation on a publicly available dataset but also exemplifies the high potential of the representation for vehicle state analysis. We present detailed insight on network configuration as well as correspondence calculation and their effect on the quality of the estimated vehicle pose.