Accurate prediction of crop yields is essential to ensure food security. In this study, a new deep networks framework is developed to predict crop yields in Australia, considering the impact of climate fertilizer use, and crop area. It is implemented for oats, corn, rice, and wheat crops, and its forecasting performance is benchmarked against five statistical and machine learning methods. All the software for the implementation of the proposed framework are freely available. The proposed framework shows highest forecasting performance for all the considered crop types. It provides 23%, 38%, 39%, and 40% average mean absolute error than the benchmark methods for oat, corn, rice, and wheat crops, respectively. The reductions in average root mean squared error are 19%, 25%, 37%, and 29% over the benchmark methods. Then, it is used to predict yields of the considered crops in Australia towards 2025 under six different change scenarios. It is observed that although climate change has some boosting impact on crop yield, not sustainable to meet the demand. However, it is possible to keep crop yields rising while mitigating change.