Audio-Visual Event Localization based on Cross-Modal Interacting Guidance

This paper studies the audio-visual event localization task, which requires the machine to locate the start and end time of the visual and audio events in the unconstrained video at the same time and identify the event category. To address this task, we propose a cross-modal interacting guidance network. Unlike previous works, it can model the complex relationship within the modality through the audio and video interacting guidance mechanism. Specifically, our cross-modal interacting guidance network is mainly composed of the cross-modal relation-aware network used as the baseline and the audio-visual interacting guidance module we joined. The cross-modal interacting guidance module (CMIG) can dynamically adjust the intra-modal attention of the target modality based on the attention flow of another modality, which is very important for modeling the complex relationships within the modality. Experiments show that our framework achieves the state-of-the-art performance in both full supervised and weakly supervised settings on the Audio-Visual Event Location (AVE) dataset.