Natural gradient descent for on-line learning

被引：71
|
作者：
Rattray, M
Saad, D
Amari, S
机构：
[1] Univ Manchester, Dept Comp Sci, Manchester M13 9PL, Lancs, England
[2] Aston Univ, Neural Comp Res Grp, Birmingham B4 7ET, W Midlands, England
[3] RIKEN, Brain Sci Inst, Lab Informat Synth, Urawa, Saitama, Japan
来源：
PHYSICAL REVIEW LETTERS | 1998年 / 81卷 / 24期
关键词：
D O I：
10.1103/PhysRevLett.81.5461
中图分类号：
O4 [物理学];
学科分类号：
0702 ;
摘要：
Natural gradient descent is an on-line variable-metric optimization algorithm which utilizes an underlying Riemannian parameter space. We analyze the dynamics of natural gradient descent beyond the asymptotic regime by employing an exact statistical mechanics description of learning in two-layer feed-forward neural networks. For a realizable learning scenario we find significant improvements over standard gradient descent for both the transient and asymptotic stages of learning, with a slower power law increase in learning time as task complexity grows. [S0031-9007(98)07950-2].
引用
收藏
页码：5461 / 5464
页数：4
相关论文
共 50 条
  • [31] Distributed Gradient Descent for Functional Learning
    Yu, Zhan
    Fan, Jun
    Shi, Zhongjie
    Zhou, Ding-Xuan
    [J]. IEEE TRANSACTIONS ON INFORMATION THEORY, 2024, 70 (09) : 6547 - 6571
  • [32] Gradient descent learning in and out of equilibrium
    Caticha, N
    de Oliveira, EA
    [J]. PHYSICAL REVIEW E, 2001, 63 (06):
  • [33] Efficient learning with robust gradient descent
    Matthew J. Holland
    Kazushi Ikeda
    [J]. Machine Learning, 2019, 108 : 1523 - 1560
  • [34] Gradient descent for general reinforcement learning
    Baird, L
    Moore, A
    [J]. ADVANCES IN NEURAL INFORMATION PROCESSING SYSTEMS 11, 1999, 11 : 968 - 974
  • [35] Learning ReLUs via Gradient Descent
    Soltanolkotabi, Mandi
    [J]. ADVANCES IN NEURAL INFORMATION PROCESSING SYSTEMS 30 (NIPS 2017), 2017, 30
  • [36] Learning to learn using gradient descent
    Hochreiter, S
    Younger, AS
    Conwell, PR
    [J]. ARTIFICIAL NEURAL NETWORKS-ICANN 2001, PROCEEDINGS, 2001, 2130 : 87 - 94
  • [37] Efficient Dictionary Learning with Gradient Descent
    Gilboa, Dar
    Buchanan, Sam
    Wright, John
    [J]. INTERNATIONAL CONFERENCE ON MACHINE LEARNING, VOL 97, 2019, 97
  • [38] Annealed gradient descent for deep learning
    Pan, Hengyue
    Niu, Xin
    Li, RongChun
    Dou, Yong
    Jiang, Hui
    [J]. NEUROCOMPUTING, 2020, 380 : 201 - 211
  • [39] Efficient learning with robust gradient descent
    Holland, Matthew J.
    Ikeda, Kazushi
    [J]. MACHINE LEARNING, 2019, 108 (8-9) : 1523 - 1560
  • [40] On early stopping in gradient descent learning
    Yao, Yuan
    Rosasco, Lorenzo
    Caponnetto, Andrea
    [J]. CONSTRUCTIVE APPROXIMATION, 2007, 26 (02) : 289 - 315
12345