Analysis of missing data and comparing the accuracy of imputation methods using wheat crop data

In a realistic scenario, the dataset has missing values encountered during the data collection. To effectively build the prediction model, the missingness of the attributes that impact crop growth needs to be appropriately handled in the crop dataset. The study aims to impute missing data in the Wheat crop yield Dataset, consisting of climatic parameters and historical data of 370 districts of Major Wheat Producer states of India. This study plays a vital role in crop estimation or forecasting of production at regular intervals. The imputation techniques that replace missing data have been categorized into Statistical and Machine Learning based Methods. We explored the performance of popular Techniques such as Arithmetic Average Replacement, Median Imputation, Linear Interpolation, Average Imputation by Nearby Districts, K-Nearest Neighbour, Miss Forest, Regression, and MICE. We have also evaluated these methods on the UCI machine learning repository's Bias and Steel energy consumption datasets. These imputed results were fed to the multiple regression prediction models to evaluate the efficiency of the imputation approaches qualitatively. The results conclude that the Arithmetic Average Replacement method provides good results among the statistical methods (R2 = 0.83; RMSE = 0.47; MAE = 0.372; MSE = 0.229), whereas in Machine Learning based methods, Miss Forest Random Forest-based method, and MICE performed well (R2 = 0.80; MAE = 0.3825; MSE = 0.249; RMSE = 0.499) to impute the missing data. We hope our results help the researchers to select the appropriate pre-processing strategies and improve the data quality.