The modern geomagnetic held is usually expressed as a spherical harmonic expansion. Although the palaeomagnetic record is very incomplete in both space and time, sufficient data are available from a span of ages to generate time-averaged spherical harmonic field models with many degrees of freedom. Here three data sets are considered: directional measurements from lavas, inclination measurements from ocean sediments, and intensity measurements from lavas. Individual data are analysed, as well as site-averages, using the same methods that have been developed for the modern held, to give models for the past 5 Myr. The normal-polarity held model has an axial-dipole intensity similar to that of the modern-day held, whilst the equatorial-dipole component is very much smaller. The held is not axisymmetric, but shows flux concentrations at the core's surface under Canada and Siberia similar to those observed in the held over historical timescales. Tests on synthetic data show that it is unlikely that these similarities result from the overprinting of the palaeomagnetic held due to inadequate cleaning of the samples. The reverse-polarity held model does not show such obvious features, but this may be due to the sparsity of the data. The patterns observed in the normal-polarity held, with persistent features in the northern hemisphere and a smooth southern hemisphere, could be explained if the present pattern of secular variation is typical of the past several million years. This would reveal itself as large variations over time in the direction of the magnetic vector in regions of high secular variation, with relatively little change over quieter regions. However, we have been unable to find any evidence for a geographical pattern of secular variation in the data.