Development and Evaluation of a Performance Metric for Image-Based Driver Assistance Systems

This report describes the formulation and application of a performance metric designed to assist the design and development of automotive active safety systems. The impetus for developing the metric was to calibrate the relative performance of the pedestrian detection algorithms used by infrared (IR) night-vision systems. The two types of system, far-and near-infrared (FIR and NIR), have complementary strengths and weaknesses that produce different sets of misses and false alarms and reveal limitations in traditional methods for comparing their performance. To overcome these limitations, our metric quantifies the similarity between system output and it ground truth - the actual locations of pedestrians in the IR image. The metric reaches its maximum value of 1.0 when the system highlights all and only the elements in the ground truth. The evaluation of the metric compared a benchmark FIR system and a prototype system that fuses inputs from the FIR system and an NIR system. The prototype was found to outperform the benchmark by 10 to 26%. This finding supports the contentions that the metric provides an effective means for assessing systems with complementary strengths. It can be used in the development of image-based driver assistance systems to assess changes in system performance due to system modifications and to evaluate large databases by highlighting situations or events worthy of developers' scrutiny.