On robust regression with high-dimensional predictors

We study regression M-estimates in the setting where p, the number of covariates, and n, the number of observations, are both large, but p <= n. We find an exact stochastic representation for the distribution of (beta) over cap = argmin(beta is an element of Rp) Sigma(n)(i=1)rho(Y-i - X-i'beta) at fixed p and n under various assumptions on the objective function rho and our statistical model. A scalar random variable whose deterministic limit r(rho)(kappa) can be studied when p/n -> kappa > 0 plays a central role in this representation. We discover a nonlinear system of two deterministic equations that characterizes r(rho)(kappa). Interestingly, the system shows that r(rho)(kappa) depends on rho through proximal mappings of rho as well as various aspects of the statistical model underlying our study. Several surprising results emerge. In particular, we show that, when p/n is large enough, least squares becomes preferable to least absolute deviations for double-exponential errors.