Projective Approximation Based Gradient Descent Modification

We present a new modification of the gradient descent algorithm based on the surrogate optimization with projection into low-dimensional space. It iteratively approximates the target function in low-dimensional space and takes the approximation optimum point mapped back to original parameter space as next parameter estimate. Main contribution of the proposed method is in application of projection idea in approximation process. Major advantage of the proposed modification is that it does not change the gradient descent iterations, thus it can be used with some other variants of the gradient descent. We give a theoretical motivation for the proposed algorithm and a theoretical lower bound for its accuracy. Finally, we experimentally study its properties on modelled data. (C) 2017, IFAC (International Federation of Automatic Control) Hosting by Elsevier Ltd. All rights reserved.