Image sentiment classification via multi-level sentiment region correlation analysis

Human's understanding of image content is a multi-level and multi-stage process. For visual sentiment analysis, this process can be specified as the gradual perception from semantic to emotion of regions in an image. The mining of emotion-related regions is valuable for sentiment recognition, and it is even more important to further investigate the semantic associations formed between these regions. In this paper, we propose a novel multi-level sentiment region correlation analysis model, which exploits the regions in an image that are most potentially affected by emotions from multiple perspectives and motivates the interaction between sentiment regions. It makes the visual content of multi-level sentiment regions and the implicit correlations within them robust cues for image sentiment recognition. We innovatively propose a module of correlation analysis of multi-level sentiment regions to exploit the effects of higher-order and rich interactions on emotions with encoders of the Transformer. Experiments on a variety of public visual sentiment analysis datasets at different scales show that the proposed MSRCA model achieve excellent performance in image sentiment classification and outperforms other existing methods. (c) 2021 Elsevier B.V. All rights reserved.