Navigation algorithm based on semantic segmentation in wheat fields using an RGB-D camera

Determining the navigation line is critical for the automatic navigation of agricultural robots in the farmland. In this research, considering a wheat field as the typical scenario, a novel navigation line extraction algorithm based on semantic segmentation is proposed. The data containing horizontal parallax, height, and grayscale information (HHG) is con-structed by combining re-encoded depth data and red-green-blue (RGB) data. The HHG, RGB, and depth data are used to achieve scene recognition and navigation line extraction for a wheat field. The method includes two main steps. First, the semantic segmentation of the wheat, ground, and background are performed using a fully convolutional network (FCN). Second, the navigation line is fitted in the camera coordinate system on the basis of the semantic segmentation result and the principle of camera pinhole imaging. Our segmentation model is trained using 508 randomly selected images from a data set, and the model is tested on 199 images. When labelled data are used as the reference benchmark, the mean intersection over union (mIoU) of the HHG data is greater than 95%, which is the highest among the three types of data. The semantic segmentation methods based on the RGB and HHG data show higher navigation line extraction accuracy rates (with the absolute value of the angle deviation less than 5 degrees) than the compared methods. The mean and standard deviation of the angle deviation of the two methods are within 0.1 degrees and 2.0 degrees, while the mean and standard deviation of the distance deviation are less than 30 mm and 60 mm, respectively. These values meet the basic requirements of agricultural machinery field navigation. The novelty of this work is the proposal of a navigation line extraction algorithm based on semantic segmentation in wheat fields. This method is high in accuracy and robustness to interference from crop occlusion.(c) 2022 China Agricultural University. Production and hosting by Elsevier B.V. on behalf of KeAi. This is an open access article under the CC BY-NC-ND license (http://creativecommons. org/licenses/by-nc-nd/4.0/).