On a Riemannian manifold of dimension n we extend the known analytic results on Yang-Mills connections to the class of connections called Omega-Yang-Mills connections, where Omega is a smooth, not necessarily closed, (n - 4)-form on M. Special cases include Omega-anti-self-dual connections and Hermitian-Yang-Mills connections over general complex manifolds. By a key observation, a weak compactness result is obtained for moduli space of smooth Omega-Yang-Mills connections with uniformly L-2 bounded curvature, and it can be improved in the case of Hermitian-Yang-Mills connections over general complex manifolds. A removable singularity theorem for singular Omega-Yang-Mills connections on a trivial bundle with small energy concentration is also proven. As an application, it is shown how to compactify the moduli space of smooth Hermitian-Yang-Mills connections on unitary bundles over a class of balanced manifolds of Hodge-Riemann type. This class includes the metrics coming from multipolarizations, and in particular, the Kahler metrics. In the case of multipolarizations on a projective algebraic manifold, the compactification of smooth irreducible Hermitian-Yang- Mills connections with fixed determinant modulo gauge transformations inherits a complex structure from algebro-geometric considerations.