Intermediate mass black holes (IMBHs; 10(1.3) to 10(5) M-circle dot) are thought to form as relics of Population III stars or from the runaway collapse of stars in young clusters; their number and very existence are uncertain. We ran N-body simulations of Galactic IMBHs, modelling them as a halo population distributed according to a Navarro, Frenk & White (NFW) or a more concentrated Diemand, Madau & Moore (DMM) density profile. As IMBHs pass through Galactic molecular/atomic hydrogen regions, they accrete gas, thus becoming X-ray sources. We constrain the density of Galactic IMBHs, Omega(center dot), by comparing the distribution of simulated X-ray sources with the observed one. From the null detections of Milky Way ultra-luminous X-ray sources, and from a comparison of simulations with unidentified sources in the Imager on Board the International Gamma-Ray Astrophysics Laboratory (INTEGRAL) Satellite/INTEGRAL Soft Gamma-Ray Imager (IBIS/ISGRI) catalogue, we find a strong upper limit Omega(center dot)<= 10(-2)Omega(b)(<= 10(-1)Omega(b)) for a DMM (NFW) profile, if IMBHs accrete via advection-dominated accretion flow (ADAF) discs. Slightly stronger constraints (Omega(center dot)<= 10(-3)Omega(b) for a DMM profile; Omega(center dot)<= 10(-2)Omega(b) for a NFW profile) can be derived if IMBHs accrete with higher efficiency, such as by forming thin accretion discs. Although not very tight, such constraints are the most stringent ones derived so far in the literature.