Using artificial queries to evaluate image retrieval

This paper addresses the evaluation and comparison of algorithms for generalized image retrieval. The forms of evaluation currently in vogue are not calibrated with each other and thus do not allow the comparison of results reported by different research groups. We address the problem by proposing a class of tests that are algorithmically defined and relatively independent of the image test set. The proposed tests can be tailored to investigate retrieval performance under specific sets of adverse conditions, allowing additional insight into the strengths and weaknesses of different retrieval mechanisms.