Uniform Manifold Sampling (UMS): Sampling the Maximum Entropy PDF

Maximum entropy PDF projection (MEPP) is a way to construct generative models from feature transformations. Corresponding to each dimension-reducing feature mapping, such as a feed-forward neural network or an algorithm to calculate linear-prediction coefficients from time series, and given a prior distribution for the features, there exists a unique generative model for the input data, which subject to mild requirements, is maximum entropy (MaxEnt) among all probability density functions (PDFs) that are consistent with the given feature prior. In this paper, we consider the problem of sampling from these MaxEnt projected PDFs. The sampling process consists of drawing a sample from the given feature prior distribution, then drawing samples uniformly distributed on the inversion set (set of input samples consistent with the drawn feature value, usually a manifold). The process is called uniform manifold sampling (UMS). We describe UMS for simple nonlinear and iterative feature transformations, then focus on linear transformations with input data constraints (x(i) > 0 or 0 <= x(i) <= 1), which require MCMC-based sampling. We discover that the manifold centroid (sample mean for a fixed feature value) is useful as a deterministic MaxEnt feature inversion solution. We show how to predict the centroid efficiently without sampling and demonstrate its usefulness in speeding up MCMC by an order of magnitude, and in spectral estimation and image reconstruction. Finally, we provide an example of UMS in a classification experiment in which we use Monte Carlo integration to create true generative models from arbitrary classifiers.