Precise High-dimensional Error Analysis of Regularized M-Estimators

A general approach for estimating an unknown signal x(0) is an element of R-n from noisy, linear measurements y = Ax(0) + z is an element of R-m is via solving a so called regularized M-estimator: (x) over cap : = arg min(x) L (y - Ax) + lambda f (x). Here, L is a convex loss function, f is a convex (typically, non-smooth) regularizer, and, lambda > 0 a regularizer parameter. We analyze the squared error performance parallel to(x) over cap - x(0)parallel to(2)(2) of such estimators in the high-dimensional proportional regime where m, n -> infinity and m/n -> delta. We let the design matrix A have entries iid Gaussian, and, impose minimal and rather mild regularity conditions on the loss function, on the regularizer, and, on the distributions of the noise and of the unknown signal. Under such a generic setting, we show that the squared error converges in probability to a nontrivial limit that is computed by solving four nonlinear equations on four scalar unknowns. We identify a new summary parameter, termed the expected Moreau envelope, which determines how the choice of the loss function and of the regularizer affects the error performance. The result opens the way for answering optimality questions regarding the choice of the loss function, the regularizer, the penalty parameter, etc.