Unifying Content and Context Similarities of the Textual and Visual Information in an Image Clustering Framework

Content-based image retrieval (CBIR) has been a challenging problem and its performance relies on the efficiency in modeling the underlying content and the similarity measure between the query and the retrieved images. Most existing metrics evaluate pairwise image similarity based only on image content, which is denoted as content similarity. However, other schemes utilize the annotations and the surrounding text to improve the retrieval results. In this study we refer to content as the visual and the textual information belonging to an image. We propose a representation of an image surrounding text in terms of concepts by utilizing an online knowledge source, e.g., Wikipedia, and propose a similarity metric that takes into account the new conceptual representation of the text. Moreover, we combine the content information with the contexts of an image to improve the retrieval percentage. The context of an image is built by constructing a vector with each dimension representing the content (visual and textual/conceptual) similarity between the image and any image in the collection. The context similarity between two images is obtained by computing the similarity between the corresponding context vectors using the vector similarity functions. Then, we fuse the similarity measures into a unified measure to evaluate the overall image similarity. Experimental results demonstrate that the new text representation and the use of the context similarity can significantly improve the retrieval performance.