Metrics for Evaluating Feature-Based Mapping Performance

In robotic mapping and simultaneous localization and mapping, the ability to assess the quality of estimated maps is crucial. While concepts exist for quantifying the error in the estimated trajectory of a robot, or a subset of the estimated feature locations, the difference between all current estimated and ground-truth features is rarely considered jointly. In contrast to many current methods, this paper analyzes metrics, which automatically evaluate maps based on their joint detection and description uncertainty. In the tracking literature, the optimal subpattern assignment (OSPA) metric provided a solution to the problem of assessing target tracking algorithms and has recently been applied to the assessment of robotic maps. Despite its advantages over other metrics, the OSPA metric can saturate to a limiting value irrespective of the cardinality errors and it penalizes missed detections and false alarms in an unequal manner. This paper therefore introduces the cardinalized optimal linear assignment (COLA) metric, as a complement to the OSPA metric, for feature map evaluation. Their combination is shown to provide a robust solution for the evaluation of map estimation errors in an intuitive manner.