Crop production depends on the interaction of climatic factors and agricultural management, such as nitrogen (N) fertilizer input in agricultural ecosystems. Understanding crop yield responses to climatic factors and N fertilizer supply can help select appropriate cultivars and develop appropriate measures to achieve the high and stable crop yield. In this study, a long-term N fertilizer experiment was conducted consisting of two crops (winter wheat and summer maize) from 1992 to 2014, to investigate the crop yield response to climatic factors and N treatments in Hotan, Xinjiang Uygur autonomous region. A long-term N supplied and without N supplied treatments significantly affect the soil properties and crop yields. It was found that low N stress significantly reduced the grain yield ranged from 33.1 to 76.4% and 26.8 to 69.3% for winter wheat and summer maize respectively. The grain N concentration was significantly reduced with the increase in N stress. There was positive correlation between crop yield and accumulated temperature (R2 = 0.58, P < 0.001) under high N conditions, and negative correlation (R2 = 0.50, P < 0.001) under low N conditions. Our result shows that the accumulated temperature is gradually increasing from 4051 to 5218 °C year−1 in the past 23 years (1992–2014) in Xinjiang Uygur autonomous region. Under sufficient N supplied conditions, the increase of accumulated temperature positively promote crop yields. However, under N deficient conditions, increasing accumulated temperature will further exacerbate the N stress effect and reduce crop yields. Hence, we recommended sufficient N fertilizer input in Hotan, Xinjiang region, which can maintain higher crop yield.