TemPanSharpening: A multi-temporal Pansharpening solution based on deep learning and edge extraction

The tradeoff among spatial, temporal, and spectral resolution of remote sensing (RS) images due to sensor properties limits the development of RS applications. Most image enhancement studies tend to focus on either spatio-temporal fusion or spatio-spectral fusion. As a more comprehensive solution, spatial-temporal-spectral fusion (STSF) is complicated but its potential is worth to be further explored. In this study, we propose a novel STSF method from the perspective of multi-temporal Pansharpening. Canny edge extraction is applied to Panchromatic (PAN) images to identify edges while avoiding the disruption of multi-temporal land cover changes. We then build a TemPanSharpening net (TPSnet) which only uses one high-spatio-low-spectra-temporal PAN and one low-spatio-high-spectra-temporal multispectral image as input. TPSnet follows a super-resolution structure and embeds two basic modules: residual-in-residual dense blocks (RRDB) and convolutional block attention module (CBAM). A series of interior ablation experiments were conducted on TPSnet and we also compared it with some representative spatio-temporal fusion, Pansharpening, and STSF algorithms. TPSnet presented satisfactory performance on complicated meter-level ground surfaces according to the quantitative evaluation result, and it demonstrated excellent robustness to land cover change.