The single crystals of diamond grown by chemical vapor deposition (CVD) are getting wider acceptance in the market for gem purposes and various other applications. Most popular applications are gems, particle detectors, high pressure anvils, and laser windows. It is widely accepted that, unless specifically made, the CVD diamond single crystals are type IIa, which implies that they have a nitrogen concentration of less than 5 ppm in the diamond. The concentration of nitrogen in CVD grown single crystal diamond is a serious deterrent for certain applications such as particle detectors, high pressure anvils, and laser windows. The threshold of nitrogen for these specific applications is subparts per billion (less than 1 ppb). On the other hand, the CVD single crystal diamonds for gems could hold nitrogen in subparts per million (less than 1 ppm) range and would still qualify as excellent gems. In this paper, an attempt is made to understand the nature of nitrogen in two types of CVD single crystal diamonds: one that shows very week fluorescence in the PL spectrum (light color or white color) with low nitrogen concentration and the other with high nitrogen concentration that shows strong fluorescence in the PL spectrum (brown to dark brown color). The correlation of nitrogen in different forms of vacancy centers in these two types of single crystals of CVD diamonds is studied. The results indicate that a strong relationship exists in concentrations of the (NVH)(0) and N-S(+) centers in brown and dark brown color CVD grown single crystal diamonds.