While current tracking methods excel in following large objects with predictable movement, they face limitations in complex backgrounds, extensive object movement ranges, and scenarios involving rapid camera motion. Moreover, many existing tracking models heavily rely on scale-space transformation techniques for feature extraction, often leading to the loss of vital spatial information. To tackle these challenges, we introduce a novel model named multi-kernel layered aggregation and enhancement based-yolo, which stands out as a single-stage object tracking model. This model incorporates a multi-kernel context enhancement module to widen the receptive field and enhance the capture of global contextual information, thereby elevating tracking accuracy. Additionally, we have introduced a multi-link downsampling module to mitigate potential spatial information loss resulting from scale transformation. Furthermore, our approach employs a dual association process integrating Kalman filters and the Hungarian algorithm for both low and high-score detection boxes, effectively mitigating target loss caused by temporary detection failures. Experimental results on the SportsMOT dataset demon-strate that our model exhibits superior performance in tracking irregularly moving objects, especially in detecting and tracking small objects. It outperforms most existing object tracking models with a DetA score of 84.6 and a HOTA score of 68.5.