Learning Discriminative Sentiment Representation from Strongly- and Weakly Supervised CNNs

Visual sentiment analysis is attracting increasing attention with the rapidly growing amount of images uploaded to social networks. Learning rich visual representations often requires training deep convolutional neural networks (CNNs) on massive manually labeled data, which is expensive or scarce especially for a subjective task like visual sentiment analysis. Meanwhile, a large quantity of social images is quite available yet noisy by querying social networks using the sentiment categories as keywords, where various types of images related to the specific sentiment can be easily collected. In this article, we propose a multiple kernel network for visual sentiment recognition, which learns representation from strongly- and weakly supervised CNNs. Specifically, the weakly supervised deep model is trained using the large-scale data from social images, whereas the strongly supervised deep model is fine tuned on the affecitve datasets with manual annotation. We employ the multiple kernel scheme on the multiple layers of CNNs, which can automatically select the discriminative representation by learning a linear combination from a set of pre-defined kernels. In addition, we introduce a large-scale dataset collected from popular comics of various countries, such as America, Japan, China, and France, which consists of 11,821 images with various artistic styles. Experimental results show that the multiple kernel network achieves consistent improvements over the state-of-the-art methods on the public affective datasets, as well as the newly established Comics dataset. The Comics dataset can be found at http://cv.nankai.edu.cn/projects/Comic.