The initial mass function (IMF) of the first (Population III) stars and Population II (Pop II) stars is poorly known due to a lack of observations of the period between recombination and reionization. In simulations of the formation of the first stars, it has been shown that, due to the limited ability of metal-free primordial gas to cool, the IMF of the first stars is a few orders of magnitude more massive than the current IMF. The transition from a high-mass IMF of the first stars to a lower-mass current IMF is thus important to understand. To study the underlying physics of this transition, we performed several simulations using the cosmological hydrodynamic adaptive mesh refinement code Enzo for metallicities of 10(-4), 10(-3), 10(-2), and 10(-1) Z(circle dot). In our simulations, we include a star formation prescription that is derived from a metallicity-dependent multi-phase interstellar medium (ISM) structure, an external UV radiation field, and a mechanical feedback algorithm. We also implement cosmic ray heating, photoelectric heating, and gas-dust heating/cooling, and follow the metal enrichment of the ISM. It is found that the interplay between metallicity and UV radiation leads to the coexistence of Pop III and Pop II star formation in non-zero-metallicity (Z/Z(circle dot) >= 10(-2)) gas. A cold (T < 100 K) and dense (rho > 10(-22) g cm(-3)) gas phase is fragile to ambient UV radiation. In a metal-poor (Z/Z(circle dot) <= 10(-3)) gas, the cold and dense gas phase does not form in the presence of a radiation field of F-0 similar to 10(-5)-10(-4) erg cm(-2) s(-1). Therefore, metallicity by itself is not a good indicator of the Pop III-Pop II transition. Metal-rich (Z/Z(circle dot) >= 10(-2)) gas dynamically evolves two to three orders of magnitude faster than metal-poor gas (Z/Z(circle dot) <= 10(-3)). The simulations including supernova explosions show that pre-enrichment of the halo does not affect the mixing of metals.