In the present paper, we have discussed the stability of dust-ion acoustic solitary waves obtained from the Korteweg-de Vries-Zakharov-Kuznetsov (KdV-ZK) equation and different modified KdV-ZK equations derived in the recent paper of Halder et al. (Z Naturforsch A 77:659, 2022) in a collisionless magnetized five components electron-positron-ion-dust plasma system consisting of warm adiabatic ions, Cairns distributed nonthermal positrons, Maxwellian distributed cold isothermal electrons, Cairns distributed nonthermal electrons and negatively charged static dust grains. We have used the small-k perturbation expansion method of Rowlands and Infeld (J Plasma Phys 3:567, 1969, J Plasma Phys 8:105, 1972, J Plasma Phys 10:293, 1973, J Plasma Phys 33:171, 1985) to analyze the stability of the steady state solitary wave solution of the KdV-ZK equation and different modified KdV-ZK equations. In this method, we want to find a nonlinear dispersion relation of the nonlinear evolution equation connecting the lowest order of & omega; and k, where & omega; is the wave frequency and k is the wave number. This nonlinear dispersion relation helps to analyze the stability of solitary structures of the KdV-ZK equation and different modified KdV-ZK equations. We have found the instability condition and the growth rate of instability up to the lowest order of wave number (k). We have graphically analyzed the growth rate of instability of different evolution equations with respect to different parameters of the present plasma system.