Mode-assisted unsupervised learning of restricted Boltzmann machines

被引:0
|
作者
Haik Manukian
Yan Ru Pei
Sean R. B. Bearden
Massimiliano Di Ventra
机构
[1] University of California,Department of Physics
来源
关键词
D O I
暂无
中图分类号
学科分类号
摘要
Restricted Boltzmann machines (RBMs) are a powerful class of generative models, but their training requires computing a gradient that, unlike supervised backpropagation on typical loss functions, is notoriously difficult even to approximate. Here, we show that properly combining standard gradient updates with an off-gradient direction, constructed from samples of the RBM ground state (mode), improves training dramatically over traditional gradient methods. This approach, which we call ‘mode-assisted training’, promotes faster training and stability, in addition to lower converged relative entropy (KL divergence). We demonstrate its efficacy on synthetic datasets where we can compute KL divergences exactly, as well as on a larger machine learning standard (MNIST). The proposed mode-assisted training can be applied in conjunction with any given gradient method, and is easily extended to more general energy-based neural network structures such as deep, convolutional and unrestricted Boltzmann machines.
引用
收藏
相关论文
共 50 条
  • [1] Mode-assisted unsupervised learning of restricted Boltzmann machines
    Manukian, Haik
    Pei, Yan Ru
    Bearden, Sean R. B.
    Di Ventra, Massimiliano
    [J]. COMMUNICATIONS PHYSICS, 2020, 3 (01)
  • [2] Mode-assisted joint training of deep Boltzmann machines
    Manukian, Haik
    Di Ventra, Massimiliano
    [J]. SCIENTIFIC REPORTS, 2021, 11 (01)
  • [3] Mode-assisted joint training of deep Boltzmann machines
    Haik Manukian
    Massimiliano Di Ventra
    [J]. Scientific Reports, 11
  • [4] Deterministic and Generalized Framework for Unsupervised Learning with Restricted Boltzmann Machines
    Tramel, Eric W.
    Gabrie, Marylou
    Manoel, Andre
    Caltagirone, Francesco
    Krzakala, Florent
    [J]. PHYSICAL REVIEW X, 2018, 8 (04):
  • [5] Unsupervised hierarchical clustering using the learning dynamics of restricted Boltzmann machines
    Decelle, Aurelien
    Seoane, Beatriz
    Rosset, Lorenzo
    [J]. PHYSICAL REVIEW E, 2023, 108 (01)
  • [6] UNSUPERVISED LEARNING FOR BOLTZMANN MACHINES
    DECO, G
    PARRA, L
    [J]. NETWORK-COMPUTATION IN NEURAL SYSTEMS, 1995, 6 (03) : 437 - 448
  • [7] Unsupervised Rotation Factorization in Restricted Boltzmann Machines
    Giuffrida, Mario Valerio
    Tsaftaris, Sotirios A.
    [J]. IEEE TRANSACTIONS ON IMAGE PROCESSING, 2020, 29 (01) : 2166 - 2175
  • [8] Unsupervised Synaptic Pruning Strategies for Restricted Boltzmann Machines
    Kalyan, Surabhi
    Joshi, Siddharth
    Sheik, Sadique
    Pedroni, Bruno U.
    Cauwenberghs, Gert
    [J]. 2018 IEEE BIOMEDICAL CIRCUITS AND SYSTEMS CONFERENCE (BIOCAS): ADVANCED SYSTEMS FOR ENHANCING HUMAN HEALTH, 2018, : 447 - 450
  • [9] Unsupervised and Supervised Visual Codes with Restricted Boltzmann Machines
    Goh, Hanlin
    Thome, Nicolas
    Cord, Matthieu
    Lim, Joo-Hwee
    [J]. COMPUTER VISION - ECCV 2012, PT V, 2012, 7576 : 298 - 311
  • [10] Unsupervised Audio Segmentation based on Restricted Boltzmann Machines
    Pikrakis, Aggelos
    [J]. 5TH INTERNATIONAL CONFERENCE ON INFORMATION, INTELLIGENCE, SYSTEMS AND APPLICATIONS, IISA 2014, 2014, : 311 - 314