Replica theory for learning curves for Gaussian processes on random graphs

We use a statistical physics approach to derive accurate predictions for the challenging problem of predicting the performance of Gaussian process regression. Performance is quantified by the learning curve, defined as the average error versus number of training examples. We assume the Gaussian process prior is defined by a random walk kernel, inputs are vertices on a random graph and the outputs are noisy function values. We show that replica techniques can be used to obtain exact performance predictions in the limit of large graphs, after first rewriting the average error in terms of a graphical model. Conventionally, the Gaussian process kernel is only globally normalized, so that the prior variance can differ between vertices. As a more principled alternative we also consider local normalization, where the prior variance is uniform. The normalization constants for the prior then have to be defined as thermal averages in an unnormalized model and this requires the introduction of a second, auxiliary set of replicas. Our results for both types of kernel normalization apply generically to all random graph ensembles constrained by a fixed but arbitrary degree distribution. We compare with numerically simulated learning curves and find excellent agreement, a significant improvement over existing approximations.