(1) With the development of artificial intelligence, people expect to use modern information technology to solve the critical problems encountered in agriculture. How to identify sunflower diseases as early and quickly as possible and take corresponding measures has become a key issue for increasing crop production and farmers' income. Sunflowers, as an important oil crop, are vulnerable to infections by various diseases, such as downy mildew, leaf scar, gray mold, etc. (2) In order to select a better lightweight model that can be embedded into mobile devices or embedded devices for sunflower disease detection, we compared five lightweight deep learning models in this study, including SqueezeNet, ShuffleNetV2, MnasNet-A1, MobileNetV3-Small, and EfficientNetV2-Small. The dataset used to train and test the models included 1892 images. These images were divided into four categories, namely, downy mildew, gray mold, leaf scar, and fresh leaves. (3) By evaluating the accuracy, precision, recall, and F1 score of each model, we found that EfficeintNetV2-Small exhibited the highest performance with an accuracy of 90.19%. Whereas the other models, SqueezeNet, ShuffleNetV2, MnasNet-A1, and MobileNetV3-Small, achieved accuracies of 84.08%, 79.31%, 88.59%, and 84.08%, respectively. To address the problem of poor generalization ability of models caused by small datasets, we adopted the transfer learning technique. After doing that, the recognition accuracies of the five models, SqueezeNet, ShuffleNetV2, MnasNet-A1, MobileNetV3-Small, and EfficeintNetV2-Small, reached 96.02%, 95.23%, 94.96%, 96.92%, and 99.20%, respectively. The accuracies of these five models were improved by 14.2%, 20%, 7.2%, 15.2%, and 10%. Based on the comparative results, we found EfficeintNetV2-Small was an optimal choice for sunflower disease identification due to its high detection accuracy.