Sparse-Coding Variational Autoencoders

The sparse coding model posits that the visual system has evolved to ef-ficiently code natural stimuli using a sparse set of features from an over-complete dictionary. The original sparse coding model suffered from twokey limitations; however: (1) computing the neural response to an imagepatch required minimizing a nonlinear objective function via recurrentdynamics and (2) fitting relied on approximate inference methods thatignored uncertainty. Although subsequent work has developed several methods to overcome these obstacles, we propose a novel solution in-spired by the variational autoencoder (VAE) framework. We introduce the sparse coding variational autoencoder (SVAE), which augments the sparse coding model with a probabilistic recognition model parameterized by a deep neural network. This recognition model provides a neu-rally plausible feedforward implementation for the mapping from image patches to neural activities and enables a principled method for fitting the sparse coding model to data via maximization of the evidence lower bound (ELBO). The SVAE differs from standard VAEs in three key respects: the latent representation is overcomplete (there are more latent dimensions than image pixels), the prior is sparse or heavy-tailed instead of gaussian, and the decoder network is a linear projection instead of a deepnetwork. We fit the SVAE to natural image data under different assumedprior distributions and show that it obtains higher test performance thanprevious fitting methods. Finally, we examine the response properties of the recognition network and show that it captures important nonlinear properties of neurons in the early visual pathway