The UAV aerial images often contain densely distributed and diverse small target objects, posing challenges for small object detection. The difficulty lies in the interference from background environments, low pixel occupancy, low resolution of the target objects, and susceptibility to occlusion. To address these issues, this paper proposes a small object detection algorithm based on improved YOLOv8n. Firstly, to enhance the focus on small objects, a detection layer targeting large-scale objects was removed from the original YOLOv8n model, and another detection layer for smaller-scale objects was added. Secondly, introducing the recursive gated convolutions to enhance the network's capability to extract feature information. Finally, multiple improved CBAM modules are inserted into the backbone network to reduce attention on non-target features, thereby strengthening the network's focus on small objects in aerial images. In the experimental results on the VisDrone2019 dataset, our model outperformed the baseline YOLOv8n model by achieving a 3.4% increase in mAP@0.50, while also achieving a significant 36.8% reduction in parameters. In summary, the improved YOLOv8n algorithm shows notable enhancement in small object detection in aerial images.