A momentum accelerated stochastic method and its application on policy search problems

With the dramatic increase in model complexity and problem scales in the machine learning area, researches on the first-order stochastic methods and its accelerated variants for non-convex problems have attracted wide research interest. However, most works on convergence analysis of accelerated methods focus on general convex or strongly convex objective functions. In this paper, we consider an accelerated scheme coming from dynamic systems and ordinary differential equations, which has a simpler and more direct form than the traditional scheme. We construct auxiliary sequences of iteration points as analysis tools, which can be interpreted as extension of Nesterov’s estimate sequence in non-convex case. We analyze the convergence property under different cases when momentum parameters are fixed or varying over iterations. For non-smooth and general convex objective functions, we give a relaxed step-size requirement to ensure convergence. For the non-convex policy search problem in classical reinforcement learning, we propose an accelerated stochastic policy gradient method with restart technique and construct numerical experiments to verify its effectiveness.