Surface defect detection of Hami melon using deep learning and image processing

An improved Convolutional Neural Network (CNN) was proposed to solve the time-consuming and inefficient detection for the surface defect on the Hami melon in recent years. The Hami melons were purchased from 103 Regiment, 6th Agricultural Division, the Xinjiang Production and Construction Corps, China. A total of 200 images of normal Hami melons were taken by a camera in a black box. 100 images of Hami melons were collected with the various surface defects, such as mildew, sunburn and crack. Since it is difficult to collect samples with three defect types, the data enhancement technique was used to expand the dataset. A total of 10 000 sample images were obtained, and then divided into a training and test dataset, according to the proportion of 4: 1. A VGG-like model was improved by adding a convolutional layer and a pooling layer at the beginning. As such, the improved VGG-like model included three convolutional layers, three max-pooling layers, a flatten layer, and two fully-connected layers. The softmax classifier was used in the last fully-connected layer. The Rectified Linear Unit (ReLU) function was chosen as the activation function. The Stochastic Gradient Descent (SGD) was chosen as the optimizer. The improved VGG-like model was used to identify four-class defect samples. The optimal hyperparameters in the CNN models were determined via the performance under the different learning rates and epochs. In all established CNN models, the test data showed that the AlexNet model outperformed other VGG-16 models, with the learning rate of 0.001 and the epochs of 500. Moreover, the AlexNet model can achieve the best performance with the accuracy of 99.69% and 96.62% in the training and test dataset, respectively. Three image processing techniques were compared to evaluate the preprocessing impact, including the Principal Components Analysis (PCA), Singular Value Decomposition (SVD), and binarization. The results indicated that the preprocessing provided a better detection performance on the various surface features of Hami melon in image preprocessing. The improved VGG-like model was the optimal to detect four-class defect on the Hami melon surface, indicating the learning rate of 0.001 and the epochs of 500. The prediction accuracy of improved VGG-like model in test set reached 97.14%. A visualization technique was used to analyze the features of convolutional layers, particularly on feature extraction in a CNN model. The visualization results showed that the defect features became more and more obvious with the increase of the convolutional layers. The defect features were the clearest in the captured images by the last convolutional layer. In addition, the convolutional features with the input as the preprocessing images were clearer than before. Finally, the improved VGG-like model was verified by the developed software on the plateform of PyQt5. The developed software functions included Open Camera, Read Image, Image Processing (Gray, PCA, SVD and Binarization), and Image Identification. The detection time of a single image was less than 0.7 s. In each type, 50 images were captured under the same environment. A total of 200 test images were collected. The test results showed that none of normal samples was predicted as defect samples. Only 8 crack Hami melons was incorrectly identified, due mainly to the unobvious feature. The average prediction accuracy of 200 samples was 93.5%. The improved VGG-like model with the preprocessing can be expected to apply for the detection of defects on the Hami melon surface, and other on-line nondestructive detection in the future. © 2021, Editorial Department of the Transactions of the Chinese Society of Agricultural Engineering. All right reserved.