Accelerating Gradient Descent with Projective Response Surface Methodology

We present a new modification of gradient descent algorithm based on surrogate optimization with projection into low-dimensional space. It consequently builds an approximation of the target function in low-dimensional space and takes the approximation optimum point mapped back to original parameter space as the next parameter estimate. An additional projection step is used to fight the curse of dimensionality. Major advantage of the proposed modification is that it does not change gradient descent iterations, thus it may be used with almost any zero- or first-order iterative method. We give a theoretical motivation for the proposed algorithm and experimentally illustrate its properties on modelled data.