Memeplate: A Chinese Multimodal Dataset for Humor Understanding in Meme Templates

Humor plays an important role in human communication. Besides language, multimodal information is also of great significance in humor expression and understanding, which promotes the development of multimodal humor research. However, in existing datasets, images and text often have a one-to-one relationship, making it difficult to control image modality variables. It causes the low correlation and low enhancement between the two modalities in humor recognition tasks. Moreover, with the development of Vision Transformers (ViTs), the generalization ability of visual models has been greatly enhanced. Using ViTs alone can achieve impressive performance, but is difficult to explain. In this paper, we introduce Memeplate (Our dataset is available at https:// github.com/chineselzf/memeplate.), a novel multimodal humor dataset containing 203 templates, 5,184 memes and manually annotated humor levels. The template transfers images and text into a one-to-many relationship, which can make it easier for researchers to cut through the linguistic lens to multimodal humor. And it provides examples closer to human behavior for generation research. In addition, we provide multiple baseline results on the humor recognition task, which demonstrate the effectiveness of our control over image modality and the importance of introducing multimodal cues.