Collaboration based multi-modal multi-label learning

Complex objects can be represented as multiple modal features and associated with multiple labels. The major challenge of complex object classification is how to jointly utilize heterogeneous modals in a mutually beneficial way. Besides, how to effectively utilize label correlations is also a challenging issue. Previous methods model the label correlations by requiring that any two label-specific classifiers behave similarly on the same modal if the associated labels are similar. To address the above challenges, we propose a novel modal-oriented deep learning framework named Collaboration based Multi-modal Multi-label Learning (CoM3L). With the help of memory structure in LSTM, CoM3L handles modalities sequentially, which predicts next modal to be extracted and learns label correlations simultaneously. On the one hand, CoM3L can extract the most useful modal sequence, which extracts different modal sequences for different instances. On the other hand, for each label, CoM3L combines the collaboration between its own prediction and the prediction of other labels. Extensive experiments on 5 multi-modal multi-label datasets validate the effectiveness of the proposed CoM3L approach.