EvRepSL: Event-Stream Representation via Self-Supervised Learning for Event-Based Vision

被引：0
|
作者：
Qu, Qiang [1 ]
Chen, Xiaoming [2 ]
Ying Chung, Yuk [1 ]
Shen, Yiran [3 ]
机构：
[1] The University of Sydney, School of Computer Science, Sydney,NSW,2050, Australia
[2] Beijing Technology and Business University, School of Computer and Artificial Intelligence, Beijing,102401, China
[3] Shandong University, School of Software, Jinan,250100, China
来源：
IEEE Transactions on Image Processing | 2024年 / 33卷
关键词：
Event-stream representation is the first step for many computer vision tasks using event cameras. It converts the asynchronous event-streams into a formatted structure so that conventional machine learning models can be applied easily. However; most of the state-of-the-art event-stream representations are manually designed and the quality of these representations cannot be guaranteed due to the noisy nature of event-streams. In this paper; we introduce a data-driven approach aiming at enhancing the quality of event-stream representations. Our approach commences with the introduction of a new event-stream representation based on spatial-temporal statistics; denoted as EvRep. Subsequently; we theoretically derive the intrinsic relationship between asynchronous event-streams and synchronous video frames. Building upon this theoretical relationship; we train a representation generator; RepGen; in a self-supervised learning manner accepting EvRep as input. Finally; the event-streams are converted to high-quality representations; termed as EvRepSL; by going through the learned RepGen (without the need of fine-tuning or retraining). Our methodology is rigorously validated through extensive evaluations on a variety of mainstream event-based classification and optical flow datasets (captured with various types of event cameras). The experimental results highlight not only our approach's superior performance over existing event-stream representations but also its versatility; being agnostic to different event cameras and tasks. © 1992-2012 IEEE;
D O I：
10.1109/TIP.2024.3497795
中图分类号：
学科分类号：
摘要：
引用
收藏
页码：6579 / 6591
相关论文
共 50 条
  • [1] Self-Supervised Prototype Representation Learning for Event-Based Corporate Profiling
    Yuan, Zixuan
    Liu, Hao
    Hu, Renjun
    Zhang, Denghui
    Xiong, Hui
    THIRTY-FIFTH AAAI CONFERENCE ON ARTIFICIAL INTELLIGENCE, THIRTY-THIRD CONFERENCE ON INNOVATIVE APPLICATIONS OF ARTIFICIAL INTELLIGENCE AND THE ELEVENTH SYMPOSIUM ON EDUCATIONAL ADVANCES IN ARTIFICIAL INTELLIGENCE, 2021, 35 : 4644 - 4652
  • [2] SHERLock: Self-Supervised Hierarchical Event Representation Learning
    Roychowdhury, S.
    Sontakke, S. A.
    Itti, L.
    Sarkar, M.
    Aggarwal, M.
    Badjatiya, P.
    Puri, N.
    Krishnamurthy, B.
    2022 26TH INTERNATIONAL CONFERENCE ON PATTERN RECOGNITION (ICPR), 2022, : 2672 - 2678
  • [3] Self-Supervised Learning of Event-Based Optical Flow with Spiking Neural Networks
    Hagenaars, Jesse J.
    Paredes-Valles, Federico
    de Croon, Guido C. H. E.
    ADVANCES IN NEURAL INFORMATION PROCESSING SYSTEMS 34 (NEURIPS 2021), 2021, 34
  • [4] Self-supervised learning representation for abnormal acoustic event detection based on attentional contrastive learning
    Wei, Juan
    Zhang, Qian
    Ning, Weichen
    DIGITAL SIGNAL PROCESSING, 2023, 142
  • [5] Improving Event Causality Identification via Self-Supervised Representation Learning on External Causal Statement
    Zuo, Xinyu
    Cao, Pengfei
    Chen, Yubo
    Liu, Kang
    Zhao, Jun
    Peng, Weihua
    Chen, Yuguang
    FINDINGS OF THE ASSOCIATION FOR COMPUTATIONAL LINGUISTICS, ACL-IJCNLP 2021, 2021, : 2162 - 2172
  • [6] EventPoint: Self-Supervised Interest Point Detection and Description for Event-based Camera
    Huang, Ze
    Sun, Li
    Zhao, Cheng
    Li, Song
    Su, Songzhi
    2023 IEEE/CVF WINTER CONFERENCE ON APPLICATIONS OF COMPUTER VISION (WACV), 2023, : 5385 - 5394
  • [7] Back to Event Basics: Self-Supervised Learning of mage Reconstruction for Event Cameras via Photometric Constancy
    Paredes-Valles, Federico
    de Croon, Guido C. H. E.
    2021 IEEE/CVF CONFERENCE ON COMPUTER VISION AND PATTERN RECOGNITION, CVPR 2021, 2021, : 3445 - 3454
  • [8] FEDERATED SELF-SUPERVISED LEARNING FOR ACOUSTIC EVENT CLASSIFICATION
    Feng, Meng
    Kao, Chieh-Chi
    Tang, Qingming
    Sun, Ming
    Rozgic, Viktor
    Matsoukas, Spyros
    Wang, Chao
    2022 IEEE INTERNATIONAL CONFERENCE ON ACOUSTICS, SPEECH AND SIGNAL PROCESSING (ICASSP), 2022, : 481 - 485
  • [9] EV-FlowNet: Self-Supervised Optical Flow Estimation for Event-based Cameras
    Zhu, Alex Zihao
    Yuan, Liangzhe
    Chaney, Kenneth
    Daniilidis, Kostas
    ROBOTICS: SCIENCE AND SYSTEMS XIV, 2018,
  • [10] Representation learning using event-based STDP
    Tavanaei, Amirhossein
    Masquelier, Timothee
    Maida, Anthony
    NEURAL NETWORKS, 2018, 105 : 294 - 303
12345