RAFnet: Recurrent attention fusion network of hyperspectral and multispectral images

Hyperspectral imaging can facilitate a better understanding of more knowledge under real scenes, compared with traditional image systems. However, due to hardware limitations, only low resolution hyperspectral (LrHs) and high resolution multispectral (HrMs) images can generally be acquired. This paper proposed a recurrent attention fusion network (RAFnet) under a variational probabilistic generative framework, in order to fuse the LrHs and HrMs images together to generate a high resolution hyperspectral (HrHs) image in an unsupervised manner. In specific, two variational autoencoders are designed to preserve both spectral and spatial information of LrHs and HrMs images, coupled through a shared decoder to generate hyperspectral images. Considering the spectra of each hyperspectral pixel is intrinsically a sequence based data structure, we construct a hierarchical recurrent neural network to extract the abundant spectral information. Moreover, self-attention and relation-attention mechanisms are adopted to capture long temporal dependencies through the spectral domain. The effectiveness and efficiency are evaluated based on several publicly available hyperspectral datasets, compared with many state-of-the-art methods for the unsupervised fusion task. (c) 2020 Elsevier B.V. All rights reserved.