We report far-infrared reflectance measurements of Zn- and Si-doped CuGeO3 single crystals as a function of applied magnetic field at low temperature. Overall, the low-energy far-infrared spectra are extraordinarily sensitive to the various phase boundaries in the H-T diagram, with the features bring especially rich in thr low-temperature dimerized state. Zn impurity substitution rapidly collapses the 44 cm(-1) zone-boundary spin Peierls gap, although broadened magnetic excitations are observed at the lightest doping level (0.2%) and a remnant is still observable at 0.7% substitution. In a 0.7% Si-doped sample, there is no evidence of the spin gap. Impurity substitution effects on the intensity of the 98 cm(-1) zone-folding mode are striking as well. The lightly doped Zn crystals display an enhanced response, and even at intermediate doping levels, the mode intensity is larger than that in the pristine material. The Si-doped sample also displays an increased intensity of the 98 cm(-1) mode in the spin Peierls phase relative to the pure material. The observed trends are discussed in terms of the effect of disorder on the spin gap and 98 cm(-1) mode, local oscillator strength sum rules, and broken selection rules.