Bayesian neuroevolution using distributed swarm optimization and tempered MCMC[Formula presented]

The major challenge of Bayesian neural networks has been in developing effective sampling methods that address deep neural networks and big data-related problems. As an alternative to gradient-based training methods, neuro-evolution features evolutionary algorithms that provide a black-box approach to learning in neural networks. Neuroevolution employs evolutionary and swarm optimization methods to provide an alternative where the training algorithm is not constrained to the architecture of the network and has the potential to reduce local-minima and vanishing gradient problems. Bayesian neural networks use variational inference and Markov chain Monte Carlo (MCMC) sampling methods. Tempered MCMC is a powerful MCMC method that can take advantage of a parallel computing environment and efficient proposal distributions. In this paper, we present a synergy of neuroevolution and Bayesian neural networks where operators in particle swarm optimization (PSO) are used for forming efficient proposals in tempered MCMC sampling. The results show that the proposed method provides better prediction accuracy when compared to random-walk proposal distribution in MCMC for both time-series and pattern classification problems. The results also show substantial computational time reduction compared to gradient-based proposals while generating comparable accuracy performance. The Bayesian neuroevolution framework can be further introduced to models that do not have gradient information. © 2022 Elsevier B.V.