A stochastic variance reduced gradient method with adaptive step for stochastic optimization

In this paper, we propose a stochastic variance reduction gradient method with adaptive step size, referred to as the SVRG-New BB method, to solve the convex stochastic optimization problem. The method could be roughly viewed as a hybrid of the SVRG algorithm and a new BB step mechanism. Under the condition that the objective function is strongly convex, we provide the linear convergence proof of this algorithm. Numerical experiment results show that the performance of the SVRG-New BB algorithm can surpass other existing algorithms if parameters in the algorithm are properly chosen. This paper introduces the SVRG-New BB method, enhancing SVRG by incorporating the New BB step mechanism for adaptive step size calculation in stochastic variance reduction gradient methods. It eliminates the need for manual step size tuning, enhancing efficiency and applicability to complex problems in machine learning. SVRG-New BB has been shown to achieve linear convergence under strong convexity, and numerical experiments demonstrate its superior performance over existing stochastic gradient methods in solving regularized logistic regression problems. image