Matrix momentum for practical natural gradient learning

被引：4

作者：

Scarpetta, S ^{[1
]}

Rattray, M

Saad, D

机构：

[1] Univ Salerno, Dept Phys ER Caianiello, Baronissi, SA, Italy

[2] INFM, Sezione Salerno, Salerno, Italy

[3] Univ Manchester, Dept Comp Sci, Manchester M13 9PL, Lancs, England

[4] Aston Univ, Neural Comp Res Grp, Birmingham B4 7ET, W Midlands, England

来源：

JOURNAL OF PHYSICS A-MATHEMATICAL AND GENERAL | 1999年 / 32卷 / 22期

关键词：

D O I：

10.1088/0305-4470/32/22/305

中图分类号：

O4 [物理学];

学科分类号：

0702 ;

摘要：

An on-line learning rule, based on the introduction of a matrix momentum term, is presented, aimed at alleviating the computational costs of standard natural gradient learning. The new rule, natural gradient matrix momentum, is analysed in the case of two-layer feed-forward neural network learning via methods of statistical physics. It appears to provide a practical algorithm that performs as well as standard natural gradient descent in both the transient and asymptotic regimes but with a hugely reduced complexity.

引用

页码：4047 / 4059

页数：13

共 50 条

[1] Natural gradient matrix momentum
Scarpetta, S
Rattray, M
Saad, D
[J]. NINTH INTERNATIONAL CONFERENCE ON ARTIFICIAL NEURAL NETWORKS (ICANN99), VOLS 1 AND 2, 1999, (470): : 43 - 48
[2] Blind equalization using matrix momentum and natural gradient adaptation
Morison, G
Durrani, T
[J]. 2003 IEEE XIII WORKSHOP ON NEURAL NETWORKS FOR SIGNAL PROCESSING - NNSP'03, 2003, : 439 - 448
[3] Scalable and Practical Natural Gradient for Large-Scale Deep Learning
Osawa, Kazuki
Tsuji, Yohei
Ueno, Yuichiro
Naruse, Akira
Foo, Chuan-Sheng
Yokota, Rio
[J]. IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, 2022, 44 (01) : 404 - 415
[4] On the momentum term in gradient descent learning algorithms
Qian, N
[J]. NEURAL NETWORKS, 1999, 12 (01) : 145 - 151
[5] Learning by Natural Gradient on Noncompact Matrix-Type Pseudo-Riemannian Manifolds
Fiori, Simone
[J]. IEEE TRANSACTIONS ON NEURAL NETWORKS, 2010, 21 (05): : 841 - 852
[6] Accelerating Federated Learning via Momentum Gradient Descent
Liu, Wei
Chen, Li
Chen, Yunfei
Zhang, Wenyi
[J]. IEEE TRANSACTIONS ON PARALLEL AND DISTRIBUTED SYSTEMS, 2020, 31 (08) : 1754 - 1766
[7] Natural gradient learning algorithms for decorrelation
Choi, S
Amari, S
Cichocki, A
[J]. PROGRESS IN CONNECTIONIST-BASED INFORMATION SYSTEMS, VOLS 1 AND 2, 1998, : 645 - 648
[8] Natural gradient works efficiently in learning
Amari, S
[J]. NEURAL COMPUTATION, 1998, 10 (02) : 251 - 276
[9] Natural gradient works efficiently in learning
Amari, S
[J]. KNOWLEDGE-BASED INTELLIGENT INFORMATION ENGINEERING SYSTEMS & ALLIED TECHNOLOGIES, PTS 1 AND 2, 2001, 69 : 11 - 14
[10] Practical pulse engineering: Gradient ascent without matrix exponentiation
Gaurav Bhole
Jonathan A. Jones
[J]. Frontiers of Physics, 2018, 13

← 1 2 3 4 5 →