An accelerated stochastic variance-reduced method for machine learning problems

被引:6
|
作者
Yang, Zhuang [1 ]
Chen, Zengping [1 ]
Wang, Cheng [2 ]
机构
[1] Sun Yat Sen Univ, Sch Elect & Commun Engn, Guangzhou 510275, Peoples R China
[2] Xiamen Univ, Sch Informat Sci & Engn, Fujian Key Lab Sensing & Comp Smart Cities, Xiamen 361005, Fujian, Peoples R China
基金
中国博士后科学基金;
关键词
Stochastic optimization; Variance reduction; Hypergradient; Recursive gradient; MINI-BATCH ALGORITHMS; GRADIENT DESCENT; NONCONVEX OPTIMIZATION;
D O I
10.1016/j.knosys.2020.105941
中图分类号
TP18 [人工智能理论];
学科分类号
081104 ; 0812 ; 0835 ; 1405 ;
摘要
Variance reduction techniques provide simple and fast algorithms for solving machine learning problems. In this paper, we present a novel stochastic variance-reduced method. The proposed method relies on the mini-batch version of stochastic recursive gradient algorithm (MB-SARAH), which updates stochastic gradient estimates by using a simple recursive scheme. However, facing the challenge of the step size sequence selection in MB-SARAH, we introduce an online step size sequence based on the hypergradient descent (HD) method, which only requires little additional computation. For the proposed method, referred to as MB-SARAH-HD, we provide a general convergence analysis and prove linear convergence for strongly convex problems in expectation. Specifically, we prove that the proposed method has sublinear convergence rate in a single outer loop. We also prove that the iteration complexity outperforms several variants of the state-of-the-art stochastic gradient descent (SGD) method under suitable conditions. Numerical experiments on standard datasets are provided to demonstrate the efficacy and superiority of our MB-SARAH-HD method over existing approaches in the literature. (C) 2020 Elsevier B.V. All rights reserved.
引用
收藏
页数:12
相关论文
共 50 条
  • [1] Variance-Reduced Methods for Machine Learning
    Gower, Robert M.
    Schmidt, Mark
    Bach, Francis
    Richtarik, Peter
    [J]. PROCEEDINGS OF THE IEEE, 2020, 108 (11) : 1968 - 1983
  • [2] Variance-Reduced Decentralized Stochastic Optimization With Accelerated Convergence
    Xin, Ran
    Khan, Usman A.
    Kar, Soummya
    [J]. IEEE TRANSACTIONS ON SIGNAL PROCESSING, 2020, 68 : 6255 - 6271
  • [3] Accelerated variance-reduced methods for saddle-point problems
    Borodich, Ekaterina
    Tominin, Vladislav
    Tominin, Yaroslav
    Kovalev, Dmitry
    Gasnikov, Alexander
    Dvurechensky, Pavel
    [J]. EURO JOURNAL ON COMPUTATIONAL OPTIMIZATION, 2022, 10
  • [4] Variance-Reduced Stochastic Learning Under Random Reshuffling
    Ying, Bicheng
    Yuan, Kun
    Sayed, Ali H.
    [J]. IEEE TRANSACTIONS ON SIGNAL PROCESSING, 2020, 68 : 1390 - 1408
  • [5] Accelerated variance-reduced methods for saddle-point problems
    Borodich, Ekaterina
    Tominin, Vladislav
    Tominin, Yaroslav
    Kovalev, Dmitry
    Gasnikov, Alexander
    Dvurechensky, Pavel
    [J]. EURO Journal on Computational Optimization, 2022, 10
  • [6] A stochastic variance-reduced accelerated primal-dual method for finite-sum saddle-point problems
    Hamedani, Erfan Yazdandoost
    Jalilzadeh, Afrooz
    [J]. COMPUTATIONAL OPTIMIZATION AND APPLICATIONS, 2023, 85 (02) : 653 - 679
  • [7] A stochastic variance-reduced accelerated primal-dual method for finite-sum saddle-point problems
    Erfan Yazdandoost Hamedani
    Afrooz Jalilzadeh
    [J]. Computational Optimization and Applications, 2023, 85 : 653 - 679
  • [8] A unified variance-reduced accelerated gradient method for convex optimization
    Lan, Guanghui
    Li, Zhize
    Zhou, Yi
    [J]. ADVANCES IN NEURAL INFORMATION PROCESSING SYSTEMS 32 (NIPS 2019), 2019, 32
  • [9] Stochastic Variance-Reduced Policy Gradient
    Papini, Matteo
    Binaghi, Damiano
    Canonaco, Giuseppe
    Pirotta, Matteo
    Restelli, Marcello
    [J]. INTERNATIONAL CONFERENCE ON MACHINE LEARNING, VOL 80, 2018, 80
  • [10] AN INEXACT VARIANCE-REDUCED METHOD FOR STOCHASTIC QUASI-VARIATIONAL INEQUALITY PROBLEMS WITH AN APPLICATION IN HEALTHCARE
    Alizadeh, Zeinab
    Otero, Brianna M.
    Jalilzadeh, Afrooz
    [J]. 2022 WINTER SIMULATION CONFERENCE (WSC), 2022, : 3099 - 3109