Adapting Depth Distribution for 3D Object Detection with a Two-Stage Training Paradigm

Lift-Splat-Shoot based 3D object detection systems aim to predict the targets' bounding boxes from images, by leveraging an explicit depth distribution that facilitates coherence between the depth and detection modules. Contrary to conventional end-to-end models that prioritize minimizing the disparity between estimated and ground-truth depth maps, our study underscores the intrinsic value of the depth distribution itself. To exploit this perspective, we introduce a novel two-stage training paradigm designed to optimize the depth and detection module separately, adopting a targeted approach to refine the depth distribution for 3D object detection. Specifically, the first stage involves training the depth module for precise depth estimation, which is supplemented by an auxiliary detection module that provides additional supervisory feedback for detection accuracy. This auxiliary component is designed to be discarded once it has served its purpose in improving the depth distribution. For the second stage, with the depth module's parameters now fixed, we train a fresh detection module from scratch under direct detection supervision. Additionally, a trainable and lightweight depth adapter is incorporated post the depth module to further adapt and polish the depth distribution, aligning it more closely with the detection objectives. Our experiments on the nuScenes dataset reveal that our approach significantly surpasses baseline models, achieving a notable 1.13% improvement on the NDS metric.