Baseline drift is a common phenomenon in the collection process of spectral data, and baseline correction is an important means to combat baseline drift interference. The baseline correction method based on sparse representation can achieve good spectral preprocessing goals. However, when applied to high-dimensional spectral baseline correction, the computational complexity is extremely high and the effectiveness is poor. Moreover, the utility of pure spectral sparse structure is insufficient, and the performance needs to be further improved. This paper proposes a spectral baseline correction method based on down-sampling to utilize sparse structures and fully reduce computational complexity. Constructing a sparse recovery model with multiple snapshots and additional correlation matrices through a down-sampling strategy ensures that each down-sampling snapshot has common sparsity and spatial correlation while reducing the dimensionality of spectral data. Subsequently, in the variational Bayesian inference (VBI) framework, the independent vector decomposition mode is introduced, and the mathematical transformation technique of vector product is used to adaptively decouple the spatial correlation between multiple snapshots, thereby inferring the Bayesian optimal sparse solutions corresponding to each snapshot. In addition, using grid refinement technology to handle off-grid gaps further improves baseline correction performance. The experimental results on simulated and real datasets have verified the superiority of the proposed method.