Information theoretic approach to Bayesian inference

A problem of central importance in formulating theoretical explanations of some observed process is the construction of a quantitative comparison of observations and examples within the context of a specific theory. From a Bayesian perspective, the ultimate goal is to compute the posterior probability for a theory given the observations. A direct approach to Bayesian inference can be prohibited when the likelihood itself is not known (i.e. or computationally intractable to compute, as in the case of non-linear processes with stochastic initial conditions). An information theoretic approach to this problem is developed in which a sequence of exponential family Gibbs densities are used as approximations to the likelihood. The sequence of Gibbs densities are constructed by successively matching expectation value constraints, giving a positive Gibbs Information Gain, or rate of convergence to the limiting density in the Kullback-Leibler distance sense. Evaluating the sequence of Gibbs densities for the observed data gives a sequence of Bayesian posterior densities which are successively more concentrated. It is shown that the successive confidence intervals form a decreasing sequence of subsets which include, and converge to, the true Bayesian confidence intervals. This provides a justifiable approach for extending Bayesian inference to problems where the likelihood is unknown, as "error bars" are simply larger. Furthermore, the Bayes Information Gain, defined as the rate at which the confidence intervals contract to the limiting interval, is shown to be maximized by a "greedy" approach to inference when constructing the Gibbs likelihoods.