Creating a web-based image database for benchmarking image retrieval systems

There is, at present, a critical need within image retrieval research for an image testbed which would enable the objective evaluation of different content-based search engines, indexing and metadata schemes, and search heuristics, as well as research and evaluation in image-based knowledge structures and system architectures, user's needs in image retrieval and the cognitive processes involved in image searching. This paper discusses a Pilot Project specifying and establishing a prototype testbed for the evaluation of image retrieval techniques. A feasibility study is underway focusing on the development of a Large set of standardized test images accessible through a web interface, and researchers in the field are being surveyed for input. Areas being addressed in the feasibility study include technical specifications (e.g., storage/server capacity, image resolution, compression and display, existing sources of images, database structure) as well as content issues such as: which specific image domains to include; the useful proportion of images belonging to specific domains to images belonging to a general "world" domain; types of image attributes and baseline and "advanced" levels of image description needed, and research needs to be accommodated, as well as development of a standardized set of test queries and the establishment of methods for "truthing" the database and test queries.