In this paper we study the action for N D0-branes in a curved background. In particular, we focus on the meaning of space-time diffeomorphism invariance. For a single D-brane, diffeomorphism invariance acts in a naive way on the world-volume fields., but for multiple D-branes, the meaning of diffeomorphism invariance is much more obscure. The problem goes beyond the determination of an ordering of the U(N)-valued fields, because one can show that there is no lift of ordinary diffeomorphisms to matrix- valued diffeomorphisms. On the other hand, the action can presumably be constructed from perturbative string theory calculations. Based on the general characteristics of such calculations we determine a set of constraints on the action for N D0-branes, that ensure space-time covariance. These constraints can be solved order by order, but they are insufficient to determine the action completely. All solutions to the constraints obey the axioms of D-geometry. Moreover the action must contain new terms. This exhibits clearly that the answer is more than a suitable ordering of the action of a single D0 brane.