Vector-Quantized Variational AutoEncoder for pansharpening

Pansharpening refers to the fusion of a multispectral image (MS) and a panchromatic image (PAN) to obtain a new image with the same spatial resolution as the PAN image and the same spectral resolution as the MS image. This paper describes a new, efficient, and accurate pansharpening architecture. The Vector-Quantized Variational AutoEncoder (VQ-VAE) is the foundation of the proposed method. The VQ-VAE model is trained to learn the non-linear mapping of degraded panchromatic image patches to high-resolution patches. This approach ensures that high-resolution patches can be recovered from low-resolution ones. After training on PAN patches, the VQ-VAE estimates high-resolution multispectral patches for each band of the original multispectral image before reconstructing the high-resolution multispectral image from the patches. The original multispectral image, the panchromatic image, and the estimated high-resolution multispectral image are combined through a modified Component Substitution (CS) process to obtain the pansharpened image. Three large satellite datasets from urban areas with 4-band spectral resolution (blue, green, red, and near-infrared) were used to evaluate the proposed pansharpening method's performance. The effectiveness of the proposed method is demonstrated by the quantitative and visual results obtained compared to several literature approaches.