Multi-label movie genre classification based on multimodal fusion

Determining the genre of a movie based on its relevant information is a challenging multi-label classification task. Previous studies tended to classify movies based on only one or two modalities, ignoring some valuable modalities. Considering this, we propose a multimodal movie genre classification framework which comprehensively considers the data from different modalities including the audio, poster, plot and frame sequences from video. To be specific, it processes the data from various modalities with the help of deep learning technologies, and fuses them in the way of decision-level fusion and intermediate fusion including concatenation and element-wise sum, which can improve the classification performance due to making full use of the information complementarity between multiple modalities. We train and evaluate the proposed framework on the LMTD-9 dataset. The results show that our best multimodal model outperforms state-of-the-art methods by 8.6% improvement in AU(PRC) and 5.3% improvement in AU(PRC)w. It can be seen that the performance of movie genre classification can be effectively improved by means of multimodal fusion.