Affine Invariant Covariance Estimation for Heavy-Tailed Distributions

In this work we provide an estimator for the covariance matrix of a heavy-tailed multivariate distribution. We prove that the proposed estimator (S) over cap admits an affine-invariant bound of the form (1 - epsilon)S <= (S) over cap <= (1 + epsilon)S in high probability, where S is the unknown covariance matrix, and <= is the positive semidefinite order on symmetric matrices. The result only requires the existence of fourth-order moments, and allows for epsilon = O(root kappa(4)d log(d/delta)/n) where kappa(4) is a measure of kurtosis of the distribution, d is the dimensionality of the space, n is the sample size, and 1 - delta is the desired confidence level. More generally, we can allow for regularization with level lambda, then d gets replaced with the degrees of freedom number. Denoting cond(S) the condition number of S, the computational cost of the novel estimator is O(d(2)n + d(3) log(cond(S))), which is comparable to the cost of the sample covariance estimator in the statistically interesing regime n >= d. We consider applications of our estimator to eigenvalue estimation with relative error, and to ridge regression with heavy-tailed random design.