Aiming at the limitations of traditional fractal box-counting dimension algorithm in the application of subtle feature extraction of radiation source signals, an improved generalized fractal box-counting dimension algorithm is proposed in the paper. Firstly, the signal is preprocessed, and the real and imaginary data of the signal after Hilbert transform is extracted to obtain the instantaneous amplitude of the signal; Then, the improved fractal box-counting dimension of signal instantaneous amplitude is extracted as the first eigenvector; At the same time, the improved fractal box-counting dimension of the signal without Hilbert transform is extracted as the second eigenvector; Finally, the dual improved fractal box-counting dimension eigenvectors form the multidimensional eigenvectors to form a new fractal box-counting dimension eigenvector as signal subtle features, for radiation source signal recognition. By establishing a dual improved fractal box-counting dimension feature space, 11 different practical radiation source signals are classified, compared with the traditional box-counting dimension algorithm, and the recognition rate is calculated. The experimental results show that compared with the traditional fractal box-counting dimension algorithm and the single improved fractal box-counting dimension algorithm, the proposed dual improved fractal box-counting dimension algorithm, can better reflect the signal subtle distribution characteristics under the different reconstruction phase space, and has a better recognition effect with good real-time performance.