Multi-modal microblog classification via multi-task learning

Recent years have witnessed the flourishing of social media platforms (SMPs), such as Twitter, Facebook, and Sina Weibo. The rapid development of these SMPs has resulted in increasingly large scale multimedia data, which has been proved with remarkable marketing values. It is in an urgent need to classify these social media data into a specified list of concerned entities, such as brands, products, and events, to analyze their sales, popularity or influences. But this is a rather challenging task due to the shortness, conversationality, the incompatibility between images and text, and the data diversity of microblogs. In this paper, we present a multi-modal microblog classification method in a multi-task learning framework. Firstly features of different modalities are extracted for each microblog. Specifically, we extract TF-IDF features for each microblog text and low-level visual features and high-level semantic features for each microblog image. Then multiple related classification tasks are learned simultaneously for each feature to increase the sample size for each task and improve the prediction performance. Finally the outputs of each feature are integrated by a Support Vector Machine that learns how to optimally combine and weight each feature. We evaluate the proposed method on Brand-Social-Net to classify the contained 100 brands. Experimental results demonstrate the superiority of the proposed method, as compared to the state-of-the-art approaches.