Graph-based multimodal semi-supervised image classification

We investigate an image classification task where training images come along with tags, but only a subset being labeled, and the goal is to predict the class label of test images without tags. This task is important for image search engine on photo sharing websites. In previous studies, it is handled by first training a multiple kernel learning classifier using both image content and tags to score unlabeled training images and then establishing a least-squares regression (LSR) model on visual features to predict the label of test images. Nevertheless, there remain three important issues in the task: (1) image tags on photo sharing websites tend to be imperfect, and thus it is beneficial to refine them for final image classification; (2) since supervised learning with a subset of labeled samples may be unreliable in practice, we adopt a graph-based label propagation approach by extra consideration of unlabeled data, and also an approach to combining multiple graphs is proposed; (3) kernel method is a powerful tool in the literature, but LSR simply treats the visual kernel matrix as an image feature matrix and does not consider the powerful kernel method. By considering these three issues holistically, we propose a graph-based multimodal semi-supervised image classification (GraMSIC) framework to handle the aforementioned task. Extensive experiments conducted on three publicly available datasets show the superior performance of the proposed framework. (C) 2014 Elsevier B.V. All rights reserved.