Toward robust and scalable deep spiking reinforcement learning

被引:5
|
作者
Akl, Mahmoud [1 ]
Ergene, Deniz [1 ]
Walter, Florian [1 ]
Knoll, Alois [1 ]
机构
[1] Tech Univ Munich, Chair Robot, TUM Sch Computat Informat & Technol, Artificial Intelligence & Embedded Syst, Munich, Germany
关键词
spiking neural network (SNN); reinforcement learning; deep reinforcement learning (Deep RL); continuous control; hyperparameter tuning; NETWORKS;
D O I
10.3389/fnbot.2022.1075647
中图分类号
TP18 [人工智能理论];
学科分类号
081104 ; 0812 ; 0835 ; 1405 ;
摘要
Deep reinforcement learning (DRL) combines reinforcement learning algorithms with deep neural networks (DNNs). Spiking neural networks (SNNs) have been shown to be a biologically plausible and energy efficient alternative to DNNs. Since the introduction of surrogate gradient approaches that allowed to overcome the discontinuity in the spike function, SNNs can now be trained with the backpropagation through time (BPTT) algorithm. While largely explored on supervised learning problems, little work has been done on investigating the use of SNNs as function approximators in DRL. Here we show how SNNs can be applied to different DRL algorithms like Deep Q-Network (DQN) and Twin-Delayed Deep Deteministic Policy Gradient (TD3) for discrete and continuous action space environments, respectively. We found that SNNs are sensitive to the additional hyperparameters introduced by spiking neuron models like current and voltage decay factors, firing thresholds, and that extensive hyperparameter tuning is inevitable. However, we show that increasing the simulation time of SNNs, as well as applying a two-neuron encoding to the input observations helps reduce the sensitivity to the membrane parameters. Furthermore, we show that randomizing the membrane parameters, instead of selecting uniform values for all neurons, has stabilizing effects on the training. We conclude that SNNs can be utilized for learning complex continuous control problems with state-of-the-art DRL algorithms. While the training complexity increases, the resulting SNNs can be directly executed on neuromorphic processors and potentially benefit from their high energy efficiency.
引用
收藏
页数:11
相关论文
共 50 条
  • [1] Robust and energy-efficient RPL optimization algorithm with scalable deep reinforcement learning for IIoT
    Wang, Ying
    Li, Yuanyuan
    Lei, Jianjun
    Shang, Fengjun
    [J]. Computer Networks, 2024, 255
  • [2] Toward Scalable and Efficient Hierarchical Deep Reinforcement Learning for 5G RAN Slicing
    Huang, Renlang
    Guo, Miao
    Gu, Chaojie
    He, Shibo
    Chen, Jiming
    Sun, Mingyang
    [J]. IEEE TRANSACTIONS ON GREEN COMMUNICATIONS AND NETWORKING, 2023, 7 (04): : 2153 - 2162
  • [3] Toward a Spintronic Deep Learning Spiking Neural Processor
    Sengupta, Abhronil
    Han, Bing
    Roy, Kaushik
    [J]. PROCEEDINGS OF 2016 IEEE BIOMEDICAL CIRCUITS AND SYSTEMS CONFERENCE (BIOCAS), 2016, : 544 - 547
  • [4] Scalable Deep Reinforcement Learning for Ride-Hailing
    Feng, Jiekun
    Gluzman, Mark
    Dai, J. G.
    [J]. IEEE CONTROL SYSTEMS LETTERS, 2021, 5 (06): : 2060 - 2065
  • [5] Scalable Deep Reinforcement Learning for Ride-Hailing
    Feng, Jiekun
    Gluzman, Mark
    Dai, J. G.
    [J]. 2021 AMERICAN CONTROL CONFERENCE (ACC), 2021, : 3743 - 3748
  • [6] Adversary Agnostic Robust Deep Reinforcement Learning
    Qu, Xinghua
    Gupta, Abhishek
    Ong, Yew-Soon
    Sun, Zhu
    [J]. IEEE TRANSACTIONS ON NEURAL NETWORKS AND LEARNING SYSTEMS, 2023, 34 (09) : 6146 - 6157
  • [7] Robust Deep Reinforcement Learning for Quadcopter Control
    Deshpande, Aditya M.
    Minai, Ali A.
    Kumar, Manish
    [J]. IFAC PAPERSONLINE, 2021, 54 (20): : 90 - 95
  • [8] Advancing Spiking Neural Networks Toward Deep Residual Learning
    Hu, Yifan
    Deng, Lei
    Wu, Yujie
    Yao, Man
    Li, Guoqi
    [J]. IEEE TRANSACTIONS ON NEURAL NETWORKS AND LEARNING SYSTEMS, 2024, : 1 - 15
  • [9] Deep Reinforcement Learning toward Robust Multi-echelon Supply Chain Inventory Optimization
    El Shar, Ibrahim
    Sun, Wenhuan
    Wang, Haiyan
    Gupta, Chetan
    [J]. 2022 IEEE 18TH INTERNATIONAL CONFERENCE ON AUTOMATION SCIENCE AND ENGINEERING (CASE), 2022, : 1385 - 1391
  • [10] BrainQN: Enhancing the Robustness of Deep Reinforcement Learning with Spiking Neural Networks
    Feng, Shuo
    Cao, Jian
    Ou, Zehong
    Chen, Guang
    Zhong, Yi
    Wang, Zilin
    Yan, Juntong
    Chen, Jue
    Wang, Bingsen
    Zou, Chenglong
    Feng, Zebang
    Wang, Yuan
    [J]. ADVANCED INTELLIGENT SYSTEMS, 2024, 6 (09)