Multimodal knowledge graph construction of Chinese traditional operas and sentiment and genre recognition

The advancement of digital technologies promotes the documentation of traditional operas, leaving a large amount of data but in a state of fragmentation. Constructing a knowledge graph (KG) is an ef-fective way to realize the knowledge integration and reduce fragmentation, which can help the public understand traditional operas. However, constructed KGs in cultural heritage are mainly unimodal, lack-ing the ability to give the public a comprehensive perception, especially when they do not have related in-depth knowledge. In this paper, we take Chinese nation-level traditional operas as an example and construct a traditional opera ontology OpeOnto including classes with deep semantics (topic, sentiment). Then, we adopt an OpeOnto-driven way to construct multimodal knowledge graph OpeMKG including im-ages and music links from several data resources. Aimed at analysing sentiments in OpeMKG and the automatic genre recognition of works for the preparation of automatic updating of OpeMKG, we develop a novel unified sentiment and genre recognition model (SGRM) for traditional operas with multimodal fusion and multi-task learning (MTL). The proposed model is examined on the built dataset of traditional operas and experimental results demonstrate its superiority compared with several state-of-the-art base-lines.(c) 2023 Consiglio Nazionale delle Ricerche (CNR). Published by Elsevier Masson SAS. All rights reserved.