Bootstrapping Interactive Image-Text Alignment for Remote Sensing Image Captioning

Recently, remote sensing image captioning (RSIC) has gained significant attention in the remote sensing community. Due to the significant differences in spatial resolution of remote sensing images, existing methods in this field have predominantly concentrated on the fine-grained extraction of remote sensing image features, but they cannot effectively handle the semantic consistency between visual features and textual features. To efficiently align the image-text, we propose a novel two-stage vision-language pre-training (VLP)-based approach to bootstrap interactive image-text alignment for RSIC, called BITA, which relies on the design of a lightweight interactive Fourier transformer (IFT) to better align remote sensing image-text features. The Fourier layer in the IFT is capable of extracting multiscale features of remote sensing images in the frequency domain, thereby reducing the redundancy of remote sensing visual features. Specifically, the first stage involves preliminary alignment through image-text contrastive learning (ITC), which aligns the learned multiscale remote sensing features from the IFT with textual features. In the second stage, the IFT connects the frozen image encoder with a large language model (LLM). Then, prefix causal language modeling (PCLM) is utilized to guide the text generation process using visual features. Ultimately, across the UCM-caption, RSICD, and NWPU-caption datasets, the experimental results clearly demonstrate that BITA outperforms other advanced comparative approaches.