In five spacetime dimensions, instantons are finite energy, solitonic particles. We describe the dynamics of these objects in the presence of a Chern-Simons interaction. For U(N) instantons, we show that the 5d Chern-Simons term induces a corresponding Chern-Simons term in the ADHM quantum mechanics. For SU(N) instantons, we provide a description in terms of geodesic motion on the instanton moduli space, modified by the presence of a magnetic field. We show that this magnetic field is equal to the first Chern character of an index bundle. All of these results are derived by a simple method which follows the fate of zero modes as fermions are introduced, made heavy, and subsequently integrated out.