Automatic detection of quality soil spectra in an online vis-NIR soil sensor

The quality of online visible and near infrared (vis-NIR) soil spectra can be deteriorated by interferences of ambient light, and debris e.g., stones, roots or plant residues among others, which considerably reduces the accuracy of the predictions. Filtering of very noisy and non-soil spectra from good-quality soil spectra needs to be performed prior the modelling. Nevertheless, manual filtering of the large amount of vis-NIR online measurement is a laborious and time-consuming task. This study was conducted to develop an automatic filtering system of very noisy and non-soil spectra. Soil and non-soil spectra obtained during online vis-NIR measurements in four commercial fields in Flanders, Belgium were used to build two databases. The main difference in the databases is that flat spectra, mostly found in wet soil conditions, were considered as non-soil spectra in the first group and as soil spectra in the second group. Similarity algorithms [i.e., Pearson correlation, Spearman correlation, Euclidian distance, cosine distance and principal component analysis (PCA)] and machine learning algorithms (i.e., linear discriminant analysis, support vector machine and K-nearest neighbors) for spectra filtering using the two databases were evaluated and compared. Results suggest that the similarity algorithms were not successful to classify spectra into soil and non-soil classes for both groups, since the best prediction accuracy in cross validation achieved by the cosine distance algorithm was 76.11%. However, the machine learning algorithms provided high classification accuracies for both databases. For the former database, the best classification result of 98.5% in cross-validation and 98.6% in independent validation was obtained by using the k-nearest neighbor algorithm. While for the latter database, the best result was achieved by the support vector machine algorithm with a gaussian kernel obtaining 81.4% in cross-validation and 82.03% in independent validation. The best performing model was used to build a cleaning function to automatically pre-process and classify spectra into soil or non-soil classes. This automatic spectrum filtering system enables time saving and ensures only high-quality spectra are used for accurate online prediction of soil properties, necessary for sensor-based variable rate applications.