A Study of Visual and Semantic Similarity for Social Image Search Recommendation

Partially due to the short and ambiguous keyword queries, many image search engines group search results into conceptual image clusters to minimize the chance of completely missing user search intent. Very often, a small subset of image clusters in search is relevant to user's search intent. However, existing search engines do not support further exploration once a user has located the image cluster(s) that interest her. Similar to the problem of finding similar images of a given image, in this paper, we study the problem of "finding similar image clusters of a given image cluster". We study this problem in the context of socially annotated images (e.g., images annotated with tags in Flickr). Each image cluster is then represented in two feature spaces: the visual feature space to describe the visual characteristics of the images in the image clusters; and the semantic feature space to describe an image cluster based on the tags of its member images. Two measures named relatedness and diversity are proposed to evaluate the effectiveness of the visual and semantic similarities in image cluster recommendation. Our experimental results show that both visual and semantic similarities should be considered in image cluster recommendation to support search result exploration. We also note that using visual similarity leads to more diversified recommendations while the semantic similarity recommends conceptually more related image clusters.