In agro-ecosystems, the relationship between soil fertility and crop yield is mediated by manure application. In this study, an 8-year field experiment was performed with four fertilizer treatments (NPK, NPKM1, NPKM2, and NPKM3), where NPK refers to chemical fertilizer and M1, M2, and M3 refer to manure application rates of 15, 30, and 45 Mg ha−1 year−1, respectively. The results showed that the NPKM (NPKM1, NPKM2, and NPKM3) treatments produced greater and more stable yields (4.95–5.45 Mg ha−1 and 0.59–0.75) than the NPK treatment (4.01 Mg ha−1 and 0.50). Crop yields under the NPKM treatments showed two trends, with a rate of decrease of 0.48–0.83 Mg ha−1 year−1 during the first 4 years and a rate of increase of 0.10–0.25 Mg ha−1 year−1 during the last 4 years. The soil organic carbon (SOC) significantly increased under all treatments. The estimated annual SOC decomposition rate was 0.35 Mg ha−1 year−1 and the equilibrium SOC level was 6.22 Mg ha−1. Soil total nitrogen (N), available N, total phosphorus (P) and available P under the NPKM treatments increased by 0.15–0.26, 15–33, 0.17–0.66 and 45–159 g kg−1, respectively, compared with the NPK treatment. Manure application mainly influenced crop yield by affecting the soil TN, available N, and available P, which accounted for up to 64% of the crop yield variation. Taken together, applying manure can determine or at least improve the effects of soil fertility on crop yield in acidic soils in South China.