Annual rotations of paddy rice with upland crops has become more popular in recent years under the increasing pressure to ensure food security. Five representative paddy-upland rotations (PURs), i.e., garlic-rice (GR), wheat-rice (WR), broad bean-rice (BR), rapeseed-rice (RR), and potato-rice (PR) under two nitrogen (N) treatments were assessed from 2014 to 2017 to investigate the effects of PURs on paddy productivity and the regulatory effects of N fertilizer. Results showed that different PURs influenced the growth of tillers and leaves in the vegetative growth stage, leading to differences in effective tiller number, heading-stage canopy structure, and dry matter accumulation, resulting in significant differences in rice yield. With N application (NA), rice yields of GR and PR with straw mulching were higher than those of RR and WR rotations. Without N application (CK), rice yield in PR was only lower than that in the BR rotation. Compared with the other four PURs, rice yields under WR were the lowest, with decreases of 3.7% and 12.5% under NA and CK conditions, respectively. In conclusion, preceding crop and N application significantly affect paddy productivity by influencing rice vegetative growth. The WR rotation does not seem conducive to sustainable paddy productivity under traditional cultivation methods, whereas, the PR cropping system shows potential for sustainable crop production. Particular attention should be paid to upland crop selection and crop management practice, especially regarding C and N inputs, to ensure ssustainable production of paddy rice.