Difference-enhanced adaptive momentum methods for non-convex stochastic optimization in image classification

Stochastic gradient descent with momentum (SGDM) is a classic optimization method that determines the update direction using a moving average of the gradient over historical steps. However, SGDM suffers from slow convergence. In 2022, Yuan et al. [6] proposed stochastic gradient descent with momentum and difference (SGDMD), which incorporates the concept of differences to adjust the convergence direction and accelerate the optimization process. Despite its improvements, SGDMD requires careful parameter tuning and is prone to oscillations due to the difference mechanism. In this work, we introduce a new momentum method: stochastic gradient descent with adaptive momentum and difference (SGDAMD). Compared to SGDMD, SGDAMD demonstrates superior performance in experiments, achieving greater stability in terms of both loss values and accuracy in deep learning image classification tasks. Additionally, SGDAMD attains a sublinear convergence rate in non-convex settings while requiring less restrictive assumptions than standard smoothness conditions. These features underscore the algorithm's efficiency and effectiveness in addressing complex optimization challenges.