Performance Evaluation Method for Mobile Computer Vision Systems using Augmented Reality

This paper describes a framework which uses augmented reality for evaluating the performance of mobile computer vision systems. Computer vision systems use primarily image data to interpret the surrounding world, e. g to detect, classify and track objects. The performance of mobile computer vision systems acting in unknown environments is inherently difficult to evaluate since, often, obtaining ground truth data is problematic. The proposed novel framework exploits the possibility to add virtual agents into a real data sequence collected in an unknown environment, thus making it possible to efficiently create augmented data sequences, including ground truth, to be used for performance evaluation. Varying the content in the data sequence by adding different virtual agents is straightforward, making the proposed framework very flexible. The method has been implemented and tested on a pedestrian detection system used for automotive collision avoidance. Preliminary results show that the method has potential to replace and complement physical testing, for instance by creating collision scenarios, which are difficult to test in reality.