Interactive Change-Aware Transformer Network for Remote Sensing Image Change Captioning

被引:2
|
作者
Cai, Chen [1 ]
Wang, Yi [2 ]
Yap, Kim-Hui [1 ]
机构
[1] Nanyang Technol Univ, Sch Elect & Elect Engn, Singapore 639798, Singapore
[2] Hong Kong Polytech Univ, Dept Elect & Elect Engn, Hong Kong, Peoples R China
关键词
image change captioning; remote sensing; multi-layer change awareness; transformer;
D O I
10.3390/rs15235611
中图分类号
X [环境科学、安全科学];
学科分类号
08 ; 0830 ;
摘要
Remote sensing image change captioning (RSICC) aims to automatically generate sentences describing the difference in content in remote sensing bitemporal images. Recent works extract the changes between bitemporal features and employ a hierarchical approach to fuse multiple changes of interest, yielding change captions. However, these methods directly aggregate all features, potentially incorporating non-change-focused information from each encoder layer into the change caption decoder, adversely affecting the performance of change captioning. To address this problem, we proposed an Interactive Change-Aware Transformer Network (ICT-Net). ICT-Net is able to extract and incorporate the most critical changes of interest in each encoder layer to improve change description generation. It initially extracts bitemporal visual features from the CNN backbone and employs an Interactive Change-Aware Encoder (ICE) to capture the crucial difference between these features. Specifically, the ICE captures the most change-aware discriminative information between the paired bitemporal features interactively through difference and content attention encoding. A Multi-Layer Adaptive Fusion (MAF) module is proposed to adaptively aggregate the relevant change-aware features in the ICE layers while minimizing the impact of irrelevant visual features. Moreover, we extend the ICE to extract multi-scale changes and introduce a novel Cross Gated-Attention (CGA) module into the change caption decoder to select essential discriminative multi-scale features to improve the change captioning performance. We evaluate our method on two RSICC datasets (e.g., LEVIR-CC and LEVIRCCD), and the experimental results demonstrate that our method achieves a state-of-the-art performance.
引用
收藏
页数:21
相关论文
共 50 条
  • [1] Change-Aware Cascaded Dual-Decoder Network for Remote Sensing Image Change Detection
    Yang, Feng
    Yuan, Yifeng
    Qin, Anyong
    Zhao, Yue
    Song, Tiecheng
    Gao, Chenqiang
    [J]. IEEE TRANSACTIONS ON GEOSCIENCE AND REMOTE SENSING, 2024, 62
  • [2] PROGRESSIVE SCALE-AWARE NETWORK FOR REMOTE SENSING IMAGE CHANGE CAPTIONING
    Liu, Chenyang
    Yang, Jiajun
    Qi, Zipeng
    Zou, Zhengxia
    Shi, Zhenwei
    [J]. IGARSS 2023 - 2023 IEEE INTERNATIONAL GEOSCIENCE AND REMOTE SENSING SYMPOSIUM, 2023, : 6668 - 6671
  • [3] A Lightweight Sparse Focus Transformer for Remote Sensing Image Change Captioning
    Sun, Dongwei
    Bao, Yajie
    Liu, Junmin
    Cao, Xiangyong
    [J]. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 2024, 17 : 18727 - 18738
  • [4] Lightweight Structure-Aware Transformer Network for Remote Sensing Image Change Detection
    Lei, Tao
    Xu, Yetong
    Ning, Hailong
    Lv, Zhiyong
    Min, Chongdan
    Jin, Yaochu
    Nandi, Asoke K.
    [J]. IEEE GEOSCIENCE AND REMOTE SENSING LETTERS, 2024, 21
  • [5] SwinSUNet: Pure Transformer Network for Remote Sensing Image Change Detection
    Zhang, Cui
    Wang, Liejun
    Cheng, Shuli
    Li, Yongming
    [J]. IEEE TRANSACTIONS ON GEOSCIENCE AND REMOTE SENSING, 2022, 60
  • [6] Multi-scale Attentive Fusion Network for Remote Sensing Image Change Captioning
    Chen, Cai
    Wang, Yi
    Yap, Kim-Hui
    [J]. 2024 IEEE INTERNATIONAL SYMPOSIUM ON CIRCUITS AND SYSTEMS, ISCAS 2024, 2024,
  • [7] Changes to Captions: An Attentive Network for Remote Sensing Change Captioning
    Chang, Shizhen
    Ghamisi, Pedram
    [J]. IEEE TRANSACTIONS ON IMAGE PROCESSING, 2023, 32 : 6047 - 6060
  • [8] Changes to Captions: An Attentive Network for Remote Sensing Change Captioning
    Chang, Shizhen
    Ghamisi, Pedram
    [J]. IEEE TRANSACTIONS ON IMAGE PROCESSING, 2023, 32 : 6047 - 6060
  • [9] GCFormer: Global Context-Aware Transformer for Remote Sensing Image Change Detection
    Yu, Wanting
    Zhuo, Li
    Li, Jiafeng
    [J]. IEEE TRANSACTIONS ON GEOSCIENCE AND REMOTE SENSING, 2024, 62 : 1 - 12
  • [10] VcT: Visual Change Transformer for Remote Sensing Image Change Detection
    Jiang, Bo
    Wang, Zitian
    Wang, Xixi
    Zhang, Ziyan
    Chen, Lan
    Wang, Xiao
    Luo, Bin
    [J]. IEEE TRANSACTIONS ON GEOSCIENCE AND REMOTE SENSING, 2023, 61