Thinking Outside the Ball: Optimal Learning with Gradient Descent for Generalized Linear Stochastic Convex Optimization

We consider linear prediction with a convex Lipschitz loss, or more generally, stochastic convex optimization problems of generalized linear form, i.e. where each instantaneous loss is a scalar convex function of a linear function. We show that in this setting, early stopped Gradient Descent (GD), without any explicit regularization or projection, ensures excess error at most epsilon (compared to the best possible with unit Euclidean norm) with an optimal, up to logarithmic factors, sample complexity of (O) over tilde (1/epsilon(2)) and only (O) over tilde (1/epsilon(2)) iterations. This contrasts with general stochastic convex optimization, where (O) over tilde (1/epsilon(4)) iterations are needed Amir et al. [2]. The lower iteration complexity is ensured by leveraging uniform convergence rather than stability. But instead of uniform convergence in a norm ball, which we show can guarantee suboptimal learning using Theta(1/epsilon(4)) samples, we rely on uniform convergence in a distribution-dependent ball.