Toward robust and scalable deep spiking reinforcement learning

被引:5
|
作者
Akl, Mahmoud [1 ]
Ergene, Deniz [1 ]
Walter, Florian [1 ]
Knoll, Alois [1 ]
机构
[1] Tech Univ Munich, Chair Robot, TUM Sch Computat Informat & Technol, Artificial Intelligence & Embedded Syst, Munich, Germany
关键词
spiking neural network (SNN); reinforcement learning; deep reinforcement learning (Deep RL); continuous control; hyperparameter tuning; NETWORKS;
D O I
10.3389/fnbot.2022.1075647
中图分类号
TP18 [人工智能理论];
学科分类号
081104 ; 0812 ; 0835 ; 1405 ;
摘要
Deep reinforcement learning (DRL) combines reinforcement learning algorithms with deep neural networks (DNNs). Spiking neural networks (SNNs) have been shown to be a biologically plausible and energy efficient alternative to DNNs. Since the introduction of surrogate gradient approaches that allowed to overcome the discontinuity in the spike function, SNNs can now be trained with the backpropagation through time (BPTT) algorithm. While largely explored on supervised learning problems, little work has been done on investigating the use of SNNs as function approximators in DRL. Here we show how SNNs can be applied to different DRL algorithms like Deep Q-Network (DQN) and Twin-Delayed Deep Deteministic Policy Gradient (TD3) for discrete and continuous action space environments, respectively. We found that SNNs are sensitive to the additional hyperparameters introduced by spiking neuron models like current and voltage decay factors, firing thresholds, and that extensive hyperparameter tuning is inevitable. However, we show that increasing the simulation time of SNNs, as well as applying a two-neuron encoding to the input observations helps reduce the sensitivity to the membrane parameters. Furthermore, we show that randomizing the membrane parameters, instead of selecting uniform values for all neurons, has stabilizing effects on the training. We conclude that SNNs can be utilized for learning complex continuous control problems with state-of-the-art DRL algorithms. While the training complexity increases, the resulting SNNs can be directly executed on neuromorphic processors and potentially benefit from their high energy efficiency.
引用
收藏
页数:11
相关论文
共 50 条
  • [31] RSAC: A Robust Deep Reinforcement Learning Strategy for Dimensionality Perturbation
    Gupta, Surbhi
    Singal, Gaurav
    Garg, Deepak
    Das, Swagatam
    [J]. IEEE TRANSACTIONS ON EMERGING TOPICS IN COMPUTATIONAL INTELLIGENCE, 2022, 6 (05): : 1157 - 1166
  • [32] Deep reinforcement learning-based robust missile guidance
    Ahn, Jeongsu
    Shin, Jongho
    Kim, Hyeong-Geun
    [J]. 2022 22ND INTERNATIONAL CONFERENCE ON CONTROL, AUTOMATION AND SYSTEMS (ICCAS 2022), 2022, : 927 - 930
  • [33] Robust Deep Reinforcement Learning with Adversarial Attacks Extended Abstract
    Pattanaik, Anay
    Tang, Zhenyi
    Liu, Shuijing
    Bommannan, Gautham
    Chowdhary, Girish
    [J]. PROCEEDINGS OF THE 17TH INTERNATIONAL CONFERENCE ON AUTONOMOUS AGENTS AND MULTIAGENT SYSTEMS (AAMAS' 18), 2018, : 2040 - 2042
  • [34] Robust Deep Reinforcement Learning for Interference Avoidance in Wideband Spectrum
    Aref, Mohamed A.
    Jayaweera, Sudharman K.
    [J]. 2019 IEEE COGNITIVE COMMUNICATIONS FOR AEROSPACE APPLICATIONS WORKSHOP (CCAAW), 2019,
  • [35] Deep Reinforcement Learning Based Robust Communication for Internet of Vehicles
    [J]. Automatic Control and Computer Sciences, 2023, 57 : 364 - 370
  • [36] ROBUST DEEP REINFORCEMENT LEARNING FOR UNDERWATER NAVIGATION WITH UNKNOWN DISTURBANCES
    Parras, Juan
    Zazo, Santiago
    [J]. 2021 IEEE INTERNATIONAL CONFERENCE ON ACOUSTICS, SPEECH AND SIGNAL PROCESSING (ICASSP 2021), 2021, : 3440 - 3444
  • [37] Deep Reinforcement Learning Based Robust Communication for Internet of Vehicles
    Gasmi, Rim
    [J]. AUTOMATIC CONTROL AND COMPUTER SCIENCES, 2023, 57 (04) : 364 - 370
  • [38] Leveraging Domain Knowledge for Robust Deep Reinforcement Learning in Networking
    Zheng, Ying
    Chen, Haoyu
    Duan, Qingyang
    Lin, Lixiang
    Shao, Yiyang
    Wang, Wei
    Wang, Xin
    Xu, Yuedong
    [J]. IEEE CONFERENCE ON COMPUTER COMMUNICATIONS (IEEE INFOCOM 2021), 2021,
  • [39] An efficient and robust gradient reinforcement learning: Deep comparative policy
    Wang, Jiaguo
    Li, Wenheng
    Lei, Chao
    Yang, Meng
    Pei, Yang
    [J]. JOURNAL OF INTELLIGENT & FUZZY SYSTEMS, 2024, 46 (02) : 3773 - 3788
  • [40] Robust Deep Reinforcement Learning through Bootstrapped Opportunistic Curriculum
    Wu, Junlin
    Vorobeychik, Yevgeniy
    [J]. INTERNATIONAL CONFERENCE ON MACHINE LEARNING, VOL 162, 2022,