Approximating the Covariance Matrix of GMMs with Low-Rank Perturbations

Covariance matrices capture correlations that are invaluable in modeling real-life datasets. Using all d(2) elements of the covariance (in d dimensions) is costly and could result in over-fitting; and the simple diagonal approximation can be over-restrictive. We present an algorithm that improves upon the diagonal matrix by allowing a low rank perturbation. The efficiency is comparable to the diagonal approximation, yet one can capture correlations among the dimensions. We show that this method outperforms the diagonal when training GMMs on both synthetic and real-world data.