Interactive Change-Aware Transformer Network for Remote Sensing Image Change Captioning

被引:2
|
作者
Cai, Chen [1 ]
Wang, Yi [2 ]
Yap, Kim-Hui [1 ]
机构
[1] Nanyang Technol Univ, Sch Elect & Elect Engn, Singapore 639798, Singapore
[2] Hong Kong Polytech Univ, Dept Elect & Elect Engn, Hong Kong, Peoples R China
关键词
image change captioning; remote sensing; multi-layer change awareness; transformer;
D O I
10.3390/rs15235611
中图分类号
X [环境科学、安全科学];
学科分类号
08 ; 0830 ;
摘要
Remote sensing image change captioning (RSICC) aims to automatically generate sentences describing the difference in content in remote sensing bitemporal images. Recent works extract the changes between bitemporal features and employ a hierarchical approach to fuse multiple changes of interest, yielding change captions. However, these methods directly aggregate all features, potentially incorporating non-change-focused information from each encoder layer into the change caption decoder, adversely affecting the performance of change captioning. To address this problem, we proposed an Interactive Change-Aware Transformer Network (ICT-Net). ICT-Net is able to extract and incorporate the most critical changes of interest in each encoder layer to improve change description generation. It initially extracts bitemporal visual features from the CNN backbone and employs an Interactive Change-Aware Encoder (ICE) to capture the crucial difference between these features. Specifically, the ICE captures the most change-aware discriminative information between the paired bitemporal features interactively through difference and content attention encoding. A Multi-Layer Adaptive Fusion (MAF) module is proposed to adaptively aggregate the relevant change-aware features in the ICE layers while minimizing the impact of irrelevant visual features. Moreover, we extend the ICE to extract multi-scale changes and introduce a novel Cross Gated-Attention (CGA) module into the change caption decoder to select essential discriminative multi-scale features to improve the change captioning performance. We evaluate our method on two RSICC datasets (e.g., LEVIR-CC and LEVIRCCD), and the experimental results demonstrate that our method achieves a state-of-the-art performance.
引用
收藏
页数:21
相关论文
共 50 条
  • [21] Change Captioning: A New Paradigm for Multitemporal Remote Sensing Image Analysis
    Hoxha, Genc
    Chouaf, Seloua
    Melgani, Farid
    Smara, Youcef
    IEEE TRANSACTIONS ON GEOSCIENCE AND REMOTE SENSING, 2022, 60
  • [22] Cooperative Connection Transformer for Remote Sensing Image Captioning
    Zhao, Kai
    Xiong, Wei
    IEEE TRANSACTIONS ON GEOSCIENCE AND REMOTE SENSING, 2024, 62 : 1 - 14
  • [23] An attention-based multiscale transformer network for remote sensing image change detection
    Liu, Wei
    Lin, Yiyuan
    Liu, Weijia
    Yu, Yongtao
    Li, Jonathan
    ISPRS JOURNAL OF PHOTOGRAMMETRY AND REMOTE SENSING, 2023, 202 : 599 - 609
  • [24] Change-aware preemption prioritization
    Department of Computer Science, University of Illinois at Urbana-Champaign, Urbana, IL 61801, United States
    Int. Symp. Softw. Test. Anal., ISSTA - Proc., 2011, (133-143):
  • [25] AANet: An Ambiguity-Aware Network for Remote-Sensing Image Change Detection
    Hang, Renlong
    Xu, Siqi
    Yuan, Panli
    Liu, Qingshan
    IEEE TRANSACTIONS ON GEOSCIENCE AND REMOTE SENSING, 2024, 62 : 1 - 11
  • [26] Domain Adaptive and Interactive Differential Attention Network for Remote Sensing Image Change Detection
    Ji, Yuliang
    Sun, Weiwei
    Wang, Yumiao
    Lv, Zhiyong
    Yang, Gang
    Zhan, Yuanzeng
    Li, Chong
    IEEE TRANSACTIONS ON GEOSCIENCE AND REMOTE SENSING, 2024, 62 : 1 - 16
  • [27] A Mask-Guided Transformer Network with Topic Token for Remote Sensing Image Captioning
    Ren, Zihao
    Gou, Shuiping
    Guo, Zhang
    Mao, Shasha
    Li, Ruimin
    REMOTE SENSING, 2022, 14 (12)
  • [28] Change Capsule Network for Optical Remote Sensing Image Change Detection
    Xu, Quanfu
    Chen, Keming
    Zhou, Guangyao
    Sun, Xian
    REMOTE SENSING, 2021, 13 (14)
  • [29] Region-guided transformer for remote sensing image captioning
    Zhao, Kai
    Xiong, Wei
    INTERNATIONAL JOURNAL OF DIGITAL EARTH, 2024, 17 (01)
  • [30] Exploring Transformer and Multilabel Classification for Remote Sensing Image Captioning
    Kandala, Hitesh
    Saha, Sudipan
    Banerjee, Biplab
    Zhu, Xiao Xiang
    IEEE GEOSCIENCE AND REMOTE SENSING LETTERS, 2022, 19