Tight Query Complexity Lower Bounds for PCA via Finite Sample Deformed Wigner Law

We prove a query complexity lower bound for approximating the top r dimensional eigenspace of a matrix. We consider an oracle model where, given a symmetric matrix M is an element of R-dxd, an algorithm Alg is allowed to make T exact queries of the form w((i)) = Mv((i)) for i in {1, ... ,T}, where v((i)) is drawn from a distribution which depends arbitrarily on the past queries and measurements {v((j)), w((i))}1 <= j <= i-1. We show that for every gap is an element of (0, 1/2], there exists a distribution over matrices M for which 1) gap(r) (M) = Omega(gap) (where gap(r) (M) is the normalized gap between the r and r + 1-st largest-magnitude eigenvector of M), and 2) any algorithm Alg which takes fewer than const x r log d/root gap queries fails (with overwhelming probability) to identity a matrix (V) over cap is an element of R-dxr with orthonormal columns for which ((V) over cap, M (V) over cap) >= (1 - const x gap) Sigma(r)(i-1) lambda(i) (M). Our bound requires only that d is a small polynomial in 1/gap and r, and matches the upper bounds of Musco and Musco '15. Moreover, it establishes a strict separation between convex optimization and randomized, "strict-saddle" non-convex optimization of which PCA is a canonical example: in the former, first-order methods can have dimension-free iteration complexity, whereas in PCA, the iteration complexity of gradient-based methods must necessarily grow with the dimension. Our argument proceeds via a reduction to estimating a rank-r spike in a deformed Wigner model M = W + lambda UU inverted perpendicular, where W is from the Gaussian Orthogonal Ensemble, U is uniform on the d x r-Stieffel manifold and lambda > 1 governs the size of the perturbation. Surprisingly, this ubiquitous random matrix model witnesses the worst-case rate for eigenspace approximation, and the 'accelerated' inverse square-root dependence on the gap in the rate follows as a consequence of the correspendence between the asymptotic eigengap and the size of the perturbation lambda, when lambda is near the "phase transition" lambda = 1. To verify that d need only be polynomial in gap(-1) and r, we prove a finite sample convergence theorem for top eigenvalues of a deformed Wigner matrix, which may be of independent interest. We then lower bound the above estimation problem with a novel technique based on Fano-style data-processing inequalities with truncated likelihoods; the technique generalizes the Bayes-risk lower bound of Chen et al. '16, and we believe it is particularly suited to lower bounds in adaptive settings like the one considered in this paper.