DF4LCZ: A SAM-Empowered Data Fusion Framework for Scene-Level Local Climate Zone Classification

Recent advances in remote sensing technologies have highlighted their capability for accurate classification of local climate zones (LCZs). However, traditional methods using convolutional neural networks (CNNs) often fall short of effectively incorporating prior knowledge of ground objects. In addition, data sources such as Sentinel-2 struggle with capturing detailed information on ground objects. To address these issues, we introduce a novel data fusion approach that combines high-resolution Google imagery, which provides ground object priors, with Sentinel-2 multispectral imagery. Our method, the Dual-stream Fusion framework for LCZ classification (DF4LCZ), merges instance-based location features from Google imagery and spatial-spectral features from Sentinel-2. This framework is enhanced by a graph convolutional network (GCN) module, powered by the segment anything model (SAM), to improve feature extraction from Google imagery. Concurrently, a 3D-CNN architecture is utilized to process the spectral-spatial features of Sentinel-2 imagery. The effectiveness of DF4LCZ is demonstrated through experiments conducted on a specialized multisource remote sensing image dataset for LCZ classification. The related code and dataset are available at https://github.com/ctrlovefly/DF4LCZ.