Online Sparse and Orthogonal Subspace Estimation from Partial Information

We consider an online version of the sparse PCA problem with missing data in which we seek a set of sparse orthogonal basis vectors. We are motivated by big data applications where we must sequentially process possibly incomplete vector observations to find an approximating subspace, and we desire the subspace representation to be sparse and have orthogonal columns for reasons of interpretability. We propose two different algorithms for solving this problem inspired by the work of [15], where the main idea is to find a rotation matrix such that the subspace basis is sparse after rotation. Our first algorithm is a batch algorithm with updates for the rotation matrix estimate using gradient steps on the Stiefel manifold. The second algorithm is online, and for each observation it performs two updates, one of the rotation matrix estimate and one of the subspace estimate, the latter of which is updated using gradient steps on the Grassmannian. The batch algorithm is competitive with state-of-the-art on full data. The online algorithm allows for a trade-off between subspace fit and sparsity of the subspace, and its performance degrades gracefully with missing data. We evaluate the performance of these two algorithms on both synthetic and real data.