Optimizing LSTM and Bi-LSTM models for crop yield prediction and comparison of their performance with traditional machine learning techniques

Advance prediction of crop yield is very critical in the context of ensuring food security as the region specific challenges in social and environmental conditions often infringe plan of policy makers. This study presents a generic methodology to configure and fine tune the state-of-the-art Long Short-Term Memory (LSTM) based Deep Learning (DL) model through hyperparameter optimization for prediction of yield (annual crop production) in Wheat, Groundnut and Barely over India based on multiple independent input variables identified using multicollinearity test. The Monte Carlo cross-validation method is used to validate the optimized LSTM models. Results from the LSTM model tuning showed that among the 4 optimizers tested, Adam was found to perform better irrespective of the crop and Bi-LSTM outperformed sLSTM in terms of prediction accuracy. The percentage reduction in error with Bi-LSTM compared to sLSTM in predicting wheat and groundnut crop yield was 39% and 13% respectively while in case of barley crop, error reduction was marginal (0.34%). The performance of optimized Bi-LSTM model is compared with the performance of traditional machine learning (ML) models such as support vector regression (SVR) and SVR polynomial {2nd and 3rd order}, Auto Regressive Integrated Moving Average (ARIMA) and ARIMAX (ARIMA with exogenous variables) and Vector Auto-regression (VAR). The Bi-LSTM model is found to be superior to ML models; the percentage reduction in mean absolute scaled error with the Bi-LSTM compared to the best performing ML model was 94%, 72%, and 71% in predicting wheat, groundnut and barley yield respectively. This study showed that by choosing proper explanatory (independent) variable and hyperparameter optimization, a simple (single layer) structure of deep neural network (LSTM) outperformed traditional ML models in terms of accuracy for crop yield prediction application.