Synaptic metaplasticity in binarized neural networks

被引:33
|
作者
Laborieux, Axel [1 ]
Ernoult, Maxence [1 ,2 ]
Hirtzlin, Tifenn [1 ]
Querlioz, Damien [1 ]
机构
[1] Univ Paris Saclay, CNRS, Ctr Nanosci & Nanotechnol, Palaiseau, France
[2] Univ Paris Saclay, Unite Mixte Phys, CNRS, Thales, Palaiseau, France
基金
欧洲研究理事会;
关键词
MODELS;
D O I
10.1038/s41467-021-22768-y
中图分类号
O [数理科学和化学]; P [天文学、地球科学]; Q [生物科学]; N [自然科学总论];
学科分类号
07 ; 0710 ; 09 ;
摘要
While deep neural networks have surpassed human performance in multiple situations, they are prone to catastrophic forgetting: upon training a new task, they rapidly forget previously learned ones. Neuroscience studies, based on idealized tasks, suggest that in the brain, synapses overcome this issue by adjusting their plasticity depending on their past history. However, such "metaplastic" behaviors do not transfer directly to mitigate catastrophic forgetting in deep neural networks. In this work, we interpret the hidden weights used by binarized neural networks, a low-precision version of deep neural networks, as metaplastic variables, and modify their training technique to alleviate forgetting. Building on this idea, we propose and demonstrate experimentally, in situations of multitask and stream learning, a training technique that reduces catastrophic forgetting without needing previously presented data, nor formal boundaries between datasets and with performance approaching more mainstream techniques with task boundaries. We support our approach with a theoretical analysis on a tractable task. This work bridges computational neuroscience and deep learning, and presents significant assets for future embedded and neuromorphic systems, especially when using novel nanodevices featuring physics analogous to metaplasticity. Deep neural networks usually rapidly forget the previously learned tasks while training new ones. Laborieux et al. propose a method for training binarized neural networks inspired by neuronal metaplasticity that allows to avoid catastrophic forgetting and is relevant for neuromorphic applications.
引用
收藏
页数:12
相关论文
共 50 条
  • [1] Synaptic metaplasticity in binarized neural networks
    Axel Laborieux
    Maxence Ernoult
    Tifenn Hirtzlin
    Damien Querlioz
    [J]. Nature Communications, 12
  • [2] Synaptic metaplasticity for image processing enhancement in convolutional neural networks
    Vives-Boix, Victor
    Ruiz-Fernandez, Daniel
    [J]. NEUROCOMPUTING, 2021, 462 : 534 - 543
  • [3] Diabetic retinopathy detection through convolutional neural networks with synaptic metaplasticity
    Vives-Boix, Victor
    Ruiz-Fernandez, Daniel
    [J]. COMPUTER METHODS AND PROGRAMS IN BIOMEDICINE, 2021, 206
  • [4] Binarized Neural Networks
    Hubara, Itay
    Courbariaux, Matthieu
    Soudry, Daniel
    El-Yaniv, Ran
    Bengio, Yoshua
    [J]. ADVANCES IN NEURAL INFORMATION PROCESSING SYSTEMS 29 (NIPS 2016), 2016, 29
  • [5] Metaplasticity in Multistate Memristor Synaptic Networks
    Zohora, Fatima Tuz
    Zyarah, Abdullah M.
    Soures, Nicholas
    Kudithipudi, Dhireesha
    [J]. 2020 IEEE INTERNATIONAL SYMPOSIUM ON CIRCUITS AND SYSTEMS (ISCAS), 2020,
  • [6] Memristor Binarized Neural Networks
    Khoa Van Pham
    Tien Van Nguyen
    Son Bao Tran
    Nam, Hyunkyung
    Lee, Mi Jung
    Choi, Byung Joon
    Son Ngoc Truong
    Min, Kyeong-Sik
    [J]. JOURNAL OF SEMICONDUCTOR TECHNOLOGY AND SCIENCE, 2018, 18 (05) : 568 - 577
  • [7] A Review of Binarized Neural Networks
    Simons, Taylor
    Lee, Dah-Jye
    [J]. ELECTRONICS, 2019, 8 (06)
  • [8] From Synaptic Metaplasticity to Behavioral Metaplasticity
    Abraham, Wickliffe C.
    Richter-Levin, Gal
    [J]. NEUROBIOLOGY OF LEARNING AND MEMORY, 2018, 154 : 1 - 4
  • [9] Neural Spike Sorting Using Binarized Neural Networks
    Valencia, Daniel
    Alimohammad, Amir
    [J]. IEEE TRANSACTIONS ON NEURAL SYSTEMS AND REHABILITATION ENGINEERING, 2021, 29 : 206 - 214
  • [10] Evaluating FPGA Acceleration on Binarized Neural Networks and Quantized Neural Networks
    Surapally, Sarala K.
    Yang, Xiaokun
    Harman, Thomas L.
    Shih, Liwen
    [J]. 2022 INTERNATIONAL SYMPOSIUM ON MEASUREMENT AND CONTROL IN ROBOTICS (ISMCR), 2022, : 159 - 163