Deep Covariance Alignment for Domain Adaptive Remote Sensing Image Segmentation

Unsupervised domain adaptive (UDA) image segmentation has recently gained increasing attention, aiming to improve the generalization capability for transferring knowledge from the source domain to the target domain. However, in high spatial resolution remote sensing image (RSI), the same category from different domains (e.g., urban and rural) can appear to be totally different with extremely inconsistent distributions, which heavily limits the UDA accuracy. To address this problem, in this article, we propose a novel deep covariance alignment (DCA) model for UDA RSI segmentation. The DCA can explicitly align category features to learn shared domain-invariant discriminative feature representations, which enhance the ability of model generalization. Specifically, a category feature pooling (CFP) module is first used to extract category features by combining coarse outputs and deep features. Then, we leverage a novel covariance regularization (CR) to enforce the intracategory features to be closer and the intercategory features to be further separate. Compared with the existing category alignment methods, our CR aims to regularize the correlation between different dimensions of the features, and thus performs more robustly when dealing with divergent category features of imbalanced and inconsistent distributions. Finally, we propose a stagewise procedure to train the DCA to alleviate error accumulation. Experiments on both rural-to-urban and urban-to-rural scenarios of the LoveDA dataset demonstrate the superiority of our proposed DCA over other state-of-the-art UDA segmentation methods. Code is available at https://github.com/Luffy03/DCA.