This paper presents algebraic methods for the study of polynomial relative invariants, when the group Gamma formed by the symmetries and relative symmetries is a compact Lie group. We deal with the case when the subgroup H of symmetries is normal in Gamma with index m, m >= 2. For this, we develop the invariant theory of compact Lie groups acting on complex vector spaces. This is the starting point for the study of relative invariants and the computation of their generators. We first obtain the space of the invariants under the subgroup H of Gamma as a direct sum of m submodules over the ring of invariants under the whole group. Then, based on this decomposition, we construct a Hilbert basis of the ring of Gamma-invariants from a Hilbert basis of the ring of H-invariants. In both results the knowledge of the relative Reynolds operators defined on H-invariants is shown to be an essential tool to obtain the invariants under the whole group. The theory is illustrated with some examples. (C) 2013 Elsevier B.V. All rights reserved.