Cutting vibration has become a major problem to the limited high-efficiency and high-quality machining. The non-equal tooth effect of the variable pitch end mill can better adjust the phase mechanism of the system and suppress chatter. Therefore, an in-depth study is required to make full use of the vibration reduction property of the milling cutter. Considering the regenerative chatter mechanism, a nonlinear milling force model for the variable pitch end mill is analyzed first, then the time-varying coefficient matrix of the cutting force is developed, and finally, a dynamic model with multiple delays is proposed. The developed model is evaluated from the perspective of chatter stability, and the limiting cutting depth is determined by using the frequency domain and semi-discretization methods. Combining with the system dynamic detection tests, the dynamic model parameters and cutting coefficients are determined for predicting stability. The analytical solutions of stability are benchmarked against the results of the time domain digital simulation, and both predictions are validated through cutting tests. The relationship between the limiting cutting depth and pitch parameters is proposed by assessing the effect of multiple delays. A method for optimal pitch parameters is developed to maximize the stability limit. It is shown that the proposed method can improve the chatter stability of milling cutters and alleviate cutting vibration, which can play an important role in improving the machining efficiency and surface quality of workpieces, and enhancing the technological level of main components.