A Nonlinear Matrix Decomposition for Mining the Zeros of Sparse Data

We describe a simple iterative solution to a widely recurring problem in multivariate data analysis: given a sparse nonnegative matrix X, how to estimate a low-rank matrix 8 such that X approximate to f(8), where f is an elementwise nonlinearity? We develop a latent variable model for this problem and consider those sparsifying nonlinearities, popular in neural networks, that map all negative values to zero. The model seeks to explain the variability of sparse high-dimensional data in terms of a smaller number of degrees of freedom. We show that exact inference in this model is tractable and derive an expectation-maximization (EM) algorithm to estimate the low-rank matrix 8. Notably, we do not parameterize 8 as a product of smaller matrices to be alternately optimized; instead, we estimate 8 directly via the singular value decomposition of matrices that are repeatedly inferred (at each iteration of the EM algorithm) from the model's posterior distribution. We use the model to analyze large sparse matrices that arise from data sets of binary, grayscale, and color images. In all of these cases, we find that the model discovers much lower-rank decompositions than purely linear approaches.