Partial Modal Conditioned GANs for Multi-modal Multi-label Learning with Arbitrary Modal-Missing

Multi-modal multi-label (MMML) learning serves an important framework to learn from objects with multiple representations and annotations. Previous MMML approaches assume that all instances are with complete modalities, which usually does not hold for real-world MMML data. Meanwhile, most existing works focus on data generation using GAN, while few of them explore the downstream tasks, such as multi-modal multi-label learning. The major challenge is how to jointly model complex modal correlation and label correlation in a mutually beneficial way, especially under the arbitrary modal-missing pattern. Aim at addressing the aforementioned research challenges, we propose a novel framework named Partial Modal Conditioned Generative Adversarial Networks (PMC-GANs) for MMML learning with arbitrary modal-missing. The proposed model contains a modal completion part and a multi-modal multi-label learning part. Firstly, in order to strike a balance between consistency and complementary across different modalities, PMC-GANs incorporates all available modalities during training and generates high-quality missing modality in an efficient way. After that, PMC-GANs exploits label correlation by leveraging shared information from all modalities and specific information of each individual modality. Empirical studies on 3 MMML datasets clearly show the superior performance of PMC-GANs against other state-of-the-art approaches.