Neuromorphic event-based recognition boosted by motion-aware learning

Neuromorphic visual sensors (event cameras) that produce asynchronous responses to changes in pixel brightness offer significant advantages such as high dynamic range, low power consumption, and reduced data redundancy. These features enable breakthroughs in object recognition and detection under extreme conditions such as fluctuating brightness or rapid movements. Therefore, event cameras have attracted great attention in many mainstream applications such as autonomous driving, UAV, remote sensing, etc. However, their unique output format poses challenges to existing vision models. Typically, event signals are converted into event-based frames for processing by 2D convolutional neural networks (CNNs). While 2D CNNs are effective at capturing spatial semantics, they are less adept at encoding motion information. To overcome this, we introduce a motion-aware branch (MAB) as an auxiliary network for 2D CNNs. The MAB leverages self- and cross-attention mechanisms to integrate learned motion messages from event frame patches into the recognition backbone. We unify features from scattered 3D patches and dense 2D features into joint representations for multi-branch fusion through spatial and temporal feature guidance and alignment. The fused features are then input into the classifier for prediction. Extensive experiments demonstrate the quantitative and qualitative superiority of our approach compared to previous methods.