Steepest descent with momentum for quadratic functions is a version of the conjugate gradient method

被引：64

作者：

Bhaya, A ^{[1
]}

Kaszkurewicz, E ^{[1
]}

机构：

[1] Univ Fed Rio de Janeiro, COPPE, PEE, Dept Elect Engn, BR-21945970 Rio de Janeiro, RJ, Brazil

来源：

NEURAL NETWORKS | 2004年 / 17卷 / 01期

关键词：

backpropagation; steepest descent; momentum; conjugate gradient algorithm; convergence; continuous optimization; bilinear system; control Liapunov function;

D O I：

10.1016/S0893-6080(03)00170-9

中图分类号：

TP18 [人工智能理论];

学科分类号：

081104 ; 0812 ; 0835 ; 1405 ;

摘要：

It is pointed out that the so called momentum method, much used in the neural network literature as an acceleration of the backpropagation method, is a stationary version of the conjugate gradient method. Connections with the continuous optimization method known as heavy ball with friction are also made. In both cases, adaptive (dynamic) choices of the so called learning rate and momentum parameters are obtained using a control Liapunov function analysis of the system. (C) 2003 Elsevier Ltd. All rights reserved.

引用

页码：65 / 71

页数：7

共 50 条

[1] Stability of steepest descent with momentum for quadratic functions
Torii, M
Hagan, MT
[J]. IEEE TRANSACTIONS ON NEURAL NETWORKS, 2002, 13 (03): : 752 - 756
[2] Semistability of Steepest Descent with Momentum for Quadratic Functions
Zhang, Naimin
[J]. NEURAL COMPUTATION, 2013, 25 (05) : 1277 - 1301
[3] CONJUGATE GRADIENT VERSUS STEEPEST DESCENT
ALLWRIGHT, JC
[J]. JOURNAL OF OPTIMIZATION THEORY AND APPLICATIONS, 1976, 20 (01) : 129 - 134
[4] On the steepest descent algorithm for quadratic functions
Clóvis C. Gonzaga
Ruana M. Schneider
[J]. Computational Optimization and Applications, 2016, 63 : 523 - 542
[5] On the steepest descent algorithm for quadratic functions
Gonzaga, Clovis C.
Schneider, Ruana M.
[J]. COMPUTATIONAL OPTIMIZATION AND APPLICATIONS, 2016, 63 (02) : 523 - 542
[6] Global convergence of steepest descent for quadratic functions
Zeng, ZG
Huang, DS
Wang, ZF
[J]. INTELLIGENT DAA ENGINEERING AND AUTOMATED LEARNING IDEAL 2004, PROCEEDINGS, 2004, 3177 : 672 - 677
[7] Nonsymmetric preconditioning for conjugate gradient and steepest descent methods
Bouwmeester, Henricus
Dougherty, Andrew
Knyazev, Andrew V.
[J]. INTERNATIONAL CONFERENCE ON COMPUTATIONAL SCIENCE, ICCS 2015 COMPUTATIONAL SCIENCE AT THE GATES OF NATURE, 2015, 51 : 276 - 285
[8] Steepest descent and conjugate gradient methods with variable preconditioning
Knyazev, Andrew V.
Lashuk, Ilya
[J]. SIAM JOURNAL ON MATRIX ANALYSIS AND APPLICATIONS, 2007, 29 (04) : 1267 - 1280
[9] Near optimal step size and momentum in gradient descent for quadratic functions
Tas, Engin
Memmedli, Memmedaga
[J]. TURKISH JOURNAL OF MATHEMATICS, 2017, 41 (01) : 110 - 121
[10] On the behavior of the gradient norm in the steepest descent method
Nocedal, J
Sartenaer, A
Zhu, CY
[J]. COMPUTATIONAL OPTIMIZATION AND APPLICATIONS, 2002, 22 (01) : 5 - 35

← 1 2 3 4 5 →