Rice (Oryza sativa L.) is one of the most important crops in the world, and its production is limited by soil phosphorus (P) deficiencies in many parts of the world. Impacts of long-term fertilization regimes on rice productivity and soil P availability is largely unknown. A long-term (26-year) field experiment in a paddy soil of southeastern China was carried out to study the response of rice grain yield and soil P pools to different fertilization regimes including control without fertilization (CK), nitrogen, P, and potassium (NPK) fertilizer (NPK), NPK fertilizer plus cattle manure (NPKM), and NPK fertilizer plus rice straw (NPKS). Application of fertilizers (NPK, NPKM, and NPKS) increased rice grain yield compared with the CK treatment (on average, by 75%, 97%, and 92%, respectively). Soil P was predominately present in the organic form (51–75% of total P) across different treatments. Most soil inorganic P fractions decreased with time due to continuous depletion by rice plants in the nonfertilized treatment (CK), while they generally increased with time in the fertilizer treatments (NPK, NPKM, and NPKS) due to continued supply of P to soil. On the other hand, soil organic P fractions increased continuously with time regardless of treatment, probably due to the retention of stubble and biological immobilization of inorganic P. Positive relationships between the rice grain yield and most inorganic P fractions in the CK treatment indicated the P limitation for rice production due to no P inputs and long-term conversion of inorganic P into organic P.