Convergence rates for the stochastic gradient descent method for non-convex objective functions

被引：0
|
作者：
Fehrman, Benjamin [1 ]
Gess, Benjamin [2 ,3 ]
Jentzen, Arnulf [4 ]
机构：
[1] Mathematical Institute, University of Oxford, Oxford,OX2 6GG, United Kingdom
[2] Max Planck Institute for Mathematics in the Sciences, Leipzig,04103, Germany
[3] Fakultat fur Mathematik, Universitat Bielefeld, Bielefeld,33615, Germany
[4] Seminar for Applied Mathematics, Department of Mathematics, ETH Zurich, Zurich,8092, Switzerland
来源：
Journal of Machine Learning Research | 2020年 / 21卷
基金：
美国国家科学基金会;
关键词：
Gradient methods - Stochastic systems;
D O I：
暂无
中图分类号：
学科分类号：
摘要：
We prove the convergence to minima and estimates on the rate of convergence for the stochastic gradient descent method in the case of not necessarily locally convex nor contracting objective functions. In particular, the analysis relies on a quantitative use of mini-batches to control the loss of iterates to non-attracted regions. The applicability of the results to simple objective functions arising in machine learning is shown. © 2020 Fehrman, Gess, Jentzen.
引用
收藏
相关论文
共 50 条
  • [1] Convergence Rates for the Stochastic Gradient Descent Method for Non-Convex Objective Functions
    Fehrman, Benjamin
    Gess, Benjamin
    Jentzen, Arnulf
    [J]. JOURNAL OF MACHINE LEARNING RESEARCH, 2020, 21
  • [2] On the Convergence of (Stochastic) Gradient Descent with Extrapolation for Non-Convex Minimization
    Xu, Yi
    Yuan, Zhuoning
    Yang, Sen
    Jin, Rong
    Yang, Tianbao
    [J]. PROCEEDINGS OF THE TWENTY-EIGHTH INTERNATIONAL JOINT CONFERENCE ON ARTIFICIAL INTELLIGENCE, 2019, : 4003 - 4009
  • [3] Convergence of Constant Step Stochastic Gradient Descent for Non-Smooth Non-Convex Functions
    Pascal Bianchi
    Walid Hachem
    Sholom Schechtman
    [J]. Set-Valued and Variational Analysis, 2022, 30 : 1117 - 1147
  • [4] Convergence of Constant Step Stochastic Gradient Descent for Non-Smooth Non-Convex Functions
    Bianchi, Pascal
    Hachem, Walid
    Schechtman, Sholom
    [J]. SET-VALUED AND VARIATIONAL ANALYSIS, 2022, 30 (03) : 1117 - 1147
  • [5] Adaptive Stochastic Gradient Descent Method for Convex and Non-Convex Optimization
    Chen, Ruijuan
    Tang, Xiaoquan
    Li, Xiuting
    [J]. FRACTAL AND FRACTIONAL, 2022, 6 (12)
  • [6] On the Almost Sure Convergence of Stochastic Gradient Descent in Non-Convex Problems
    Mertikopoulos, Panayotis
    Hallak, Nadav
    Kavis, Ali
    Cevher, Volkan
    [J]. ADVANCES IN NEURAL INFORMATION PROCESSING SYSTEMS 33, NEURIPS 2020, 2020, 33
  • [7] Convergence Rates of Non-Convex Stochastic Gradient Descent Under a Generic Lojasiewicz Condition and Local Smoothness
    Scaman, Kevin
    Malherbe, Cedric
    Dos Santos, Ludovic
    [J]. INTERNATIONAL CONFERENCE ON MACHINE LEARNING, VOL 162, 2022, : 19310 - 19327
  • [8] Global Convergence of Stochastic Gradient Descent for Some Non-convex Matrix Problems
    De Sa, Christopher
    Olukotun, Kunle
    Re, Christopher
    [J]. INTERNATIONAL CONFERENCE ON MACHINE LEARNING, VOL 37, 2015, 37 : 2332 - 2341
  • [9] Taming Convergence for Asynchronous Stochastic Gradient Descent with Unbounded Delay in Non-Convex Learning
    Zhang, Xin
    Liu, Jia
    Zhu, Zhengyuan
    [J]. 2020 59TH IEEE CONFERENCE ON DECISION AND CONTROL (CDC), 2020, : 3580 - 3585
  • [10] Scaling up stochastic gradient descent for non-convex optimisation
    Mohamad, Saad
    Alamri, Hamad
    Bouchachia, Abdelhamid
    [J]. MACHINE LEARNING, 2022, 111 (11) : 4039 - 4079
12345