EvRepSL: Event-Stream Representation via Self-Supervised Learning for Event-Based Vision

被引：0
|
作者：
Qu, Qiang [1 ]
Chen, Xiaoming [2 ]
Ying Chung, Yuk [1 ]
Shen, Yiran [3 ]
机构：
[1] The University of Sydney, School of Computer Science, Sydney,NSW,2050, Australia
[2] Beijing Technology and Business University, School of Computer and Artificial Intelligence, Beijing,102401, China
[3] Shandong University, School of Software, Jinan,250100, China
来源：
IEEE Transactions on Image Processing | 2024年 / 33卷
关键词：
Event-stream representation is the first step for many computer vision tasks using event cameras. It converts the asynchronous event-streams into a formatted structure so that conventional machine learning models can be applied easily. However; most of the state-of-the-art event-stream representations are manually designed and the quality of these representations cannot be guaranteed due to the noisy nature of event-streams. In this paper; we introduce a data-driven approach aiming at enhancing the quality of event-stream representations. Our approach commences with the introduction of a new event-stream representation based on spatial-temporal statistics; denoted as EvRep. Subsequently; we theoretically derive the intrinsic relationship between asynchronous event-streams and synchronous video frames. Building upon this theoretical relationship; we train a representation generator; RepGen; in a self-supervised learning manner accepting EvRep as input. Finally; the event-streams are converted to high-quality representations; termed as EvRepSL; by going through the learned RepGen (without the need of fine-tuning or retraining). Our methodology is rigorously validated through extensive evaluations on a variety of mainstream event-based classification and optical flow datasets (captured with various types of event cameras). The experimental results highlight not only our approach's superior performance over existing event-stream representations but also its versatility; being agnostic to different event cameras and tasks. © 1992-2012 IEEE;
D O I：
10.1109/TIP.2024.3497795
中图分类号：
学科分类号：
摘要：
引用
收藏
页码：6579 / 6591
相关论文
共 50 条
  • [31] Learning Adaptive Parameter Representation for Event-Based Video Reconstruction
    Gu, Daxin
    Li, Jia
    Zhu, Lin
    IEEE SIGNAL PROCESSING LETTERS, 2024, 31 : 1950 - 1954
  • [32] Event-based Features for Robotic Vision
    Lagorce, Xavier
    Ieng, Sio-Hoi
    Benosman, Ryad
    2013 IEEE/RSJ INTERNATIONAL CONFERENCE ON INTELLIGENT ROBOTS AND SYSTEMS (IROS), 2013, : 4214 - 4219
  • [33] Self-supervised Video Representation Learning with Cross-Stream Prototypical Contrasting
    Toering, Martine
    Gatopoulos, Ioannis
    Stol, Maarten
    Hu, Vincent Tao
    2022 IEEE WINTER CONFERENCE ON APPLICATIONS OF COMPUTER VISION (WACV 2022), 2022, : 846 - 856
  • [34] SELF-SUPERVISED VISION TRANSFORMERS FOR JOINT SAR-OPTICAL REPRESENTATION LEARNING
    Wang, Yi
    Albrecht, Conrad M.
    Zhu, Xiao Xiang
    2022 IEEE INTERNATIONAL GEOSCIENCE AND REMOTE SENSING SYMPOSIUM (IGARSS 2022), 2022, : 139 - 142
  • [35] Towards Pointsets Representation Learning via Self-Supervised Learning and Set Augmentation
    Arsomngern, Pattaramanee
    Long, Cheng
    Suwajanakorn, Supasorn
    Nutanong, Sarana
    IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, 2023, 45 (01) : 1201 - 1216
  • [36] Self-Supervised Graph Learning With Hyperbolic Embedding for Temporal Health Event Prediction
    Lu, Chang
    Reddy, Chandan K.
    Ning, Yue
    IEEE TRANSACTIONS ON CYBERNETICS, 2023, 53 (04) : 2124 - 2136
  • [37] Event-based Vision meets Deep Learning on Steering Prediction for Self-driving Cars
    Maqueda, Ana I.
    Loquercio, Antonio
    Gallego, Guillermo
    Garcia, Narciso
    Scaramuzza, Davide
    2018 IEEE/CVF CONFERENCE ON COMPUTER VISION AND PATTERN RECOGNITION (CVPR), 2018, : 5419 - 5427
  • [38] Meta-Learning and Self-Supervised Pretraining for Storm Event Imagery Translation
    Rugina, Ileana
    Dangovski, Rumen
    Simek, Olga
    Veillette, Mark
    Khorrami, Pooya
    Soljacic, Marin
    Cheung, Brian
    2023 IEEE HIGH PERFORMANCE EXTREME COMPUTING CONFERENCE, HPEC, 2023,
  • [39] Stereo Depth Estimation via Self-supervised Contrastive Representation Learning
    Tukra, Samyakh
    Giannarou, Stamatia
    MEDICAL IMAGE COMPUTING AND COMPUTER ASSISTED INTERVENTION, MICCAI 2022, PT VII, 2022, 13437 : 604 - 614
  • [40] Self-Supervised Facial Motion Representation Learning via Contrastive Subclips
    Sun, Zheng
    Torrie, Shad A.
    Sumsion, Andrew W.
    Lee, Dah-Jye
    ELECTRONICS, 2023, 12 (06)
12345