Fusion-based holistic road scene understanding

This study addresses the problem of holistic road scene understanding based on the integration of visual and range data. To achieve the grand goal, the authors propose an approach that jointly tackles object-level image segmentation and semantic region labelling within a conditional random field (CRF) framework. Specifically, the authors first generate semantic object hypotheses by clustering 3D points, learning their prior appearance models, and using a deep learning method for reasoning their semantic categories. The learned priors, together with spatial and geometric contexts, are incorporated in CRF. With this formulation, visual and range data are fused thoroughly, and moreover, the coupled segmentation and semantic labelling problem can be inferred via graph cuts. The authors' approach is validated on the challenging KITTI dataset that contains diverse complicated road scenarios. Both quantitative and qualitative evaluations demonstrate its effectiveness.