To address the issue of dense targets and mutual occlusion in small target detection for aerial images, this paper proposes a small target detection algorithm based on YOLOv8n for aerial images. The algorithm incorporates several key enhancements. Firstly, at the end of the backbone network, the Bottleneck is replaced in C2f with improved FasterNet, maintaining the number of channels while improving convergence speed. Secondly, the CBS activation function SiLU is replaced in SPPF with ReLU, setting the input negative value to zero, and then the SE attention mechanism is introduced to retain more small target features. Thirdly, the efficient multi-scale attention mechanism EMA is embeded in front of the detection head, obtaining more detailed information and enhancing small target attention. Finally, the baseline network loss function CIoU is replaced with Wise IoU, providing a gain allocation strategy that prioritizes common quality anchor frames and improving network generalization. Ablation and comparison experiments are conducted using the VisDrone2021 and RSOD datasets. Results show an increase in mAP@0.5 by 5.1 and 7.2 percentage points compared to baseline algorithms for each dataset. Additionally, mAP@0.5:0.95 improved by 4.4 and 2.1 percentage points, respectively. These findings demonstrate a notable enhancement in the accuracy of detection metrics. Generalization experiments on the publicly available dataset VOC2007+2012 show an improvement of 3.8 percentage points for mAP@0.5, demonstrating good robustness. © 2024 Journal of Computer Engineering and Applications Beijing Co., Ltd.; Science Press. All rights reserved.