S2GFormer: A Transformer and Graph Convolution Combining Framework for Hyperspectral Image Classification

Transformer-based methods have a great ability to model nonlocal interactions between spectral and spatial information, while the local features are easily ignored. Graph convolutional neural networks (GCNs) tend to do well in exploiting neighborhood vertex interactions based on their unique aggregation mechanism, while the ability to extract global information is limited. In this article, we study to comprehensively utilize the advantages of transformer and graph convolution by combining the two structures into a unified Transformer (Graphormer) to construct both local and global interactions for hyperspectral image (HSI) classification, and spatial-spectral features enhanced Graphormer framework (S(2)GFormer) is proposed. Specifically, a follow patch mechanism is first proposed to transform the pixel in HSI to patches while preserving the local spatial features and reducing the computational cost. Moreover, a patchwise spectral embedding block is designed to extract the spectral features of the patch, in which a neighborhood convolution is inserted for comprehensive spectral information extraction. Finally, a multilayer Graphormer Encoder module is proposed to extract the representative spatial-spectral features from the patch for HSI classification. In our network, we jointly integrate the three aforementioned parts into a unified network, and each component benefits the other. The experimental results demonstrate its suitability for HSI classification when compared with other state-of-the-art (SOTA) classifiers, particularly in scenarios with very limited labeled samples. The code of S(2)GFormer will be made publicly available at: https://github.com/DY-HYX.