Approximating posterior distributions for mixture-model parameters

We consider a fully-Bayesian approach to using mixture models in pattern classification problems. From a Bayesian point of view, classification should be done by integrating over the posterior distribution for the model parameters (including the number of components in the mixture) given previous observations. Although it is relatively easy to compute the value of the joint probability density of the observed data and a particular choice of model parameters, it is usually difficult to integrate over the whole distribution because of the high dimensionality of the parameter space. Most current methods using mixture models settle for finding a mode of the posterior distribution using the Expectation Maximization (EM) algorithm. But, much more can be learned about the posterior distribution. We explore the multi-state MCMC methods introduced by Skilling and show how these methods can be applied to Gaussian mixture models. In addition, we examine genetic algorithms, which are most often used as optimization algorithms. We show how these algorithms can be adapted to act as multi-state MCMC algorithms, as suggested by MacKay.