Escape saddle points by a simple gradient-descent based algorithm

Escaping saddle points is a central research topic in nonconvex optimization. In this paper, we propose a simple gradient-based algorithm such that for a smooth function f : R-n -> R, it outputs an epsilon-approximate second-order stationary point in O (log n/epsilon(1.75)) iterations. Compared to the previous state-of-the-art algorithms by Jin et al. with (O) over tilde (log(4) n/epsilon(2)) or (O) over tilde (log(6) n/epsilon(1.75)) iterations, our algorithm is polynomially better in terms of log n and matches their complexities in terms of 1/c. For the stochastic setting, our algorithm outputs an epsilon-approximate second-order stationary point in (O) over tilde (log(2) n/epsilon(4)) iterations. Technically, our main contribution is an idea of implementing a robust Hessian power method using only gradients, which can find negative curvature near saddle points and achieve the polynomial speedup in log n compared to the perturbed gradient descent methods. Finally, we also perform numerical experiments that support our results.