Multi-scale RGB and NIR image Cross-fusion based on Generative Adversarial Network

Fusing the RGB and NIR images can improve the visibility and perception quality. In this task, enhancing details and keeping color fidelity are of the most importance. To achieve this goal, in this paper, an unsupervised dual-branch GAN model is proposed. In the generator, two branches are introduced to fuse texture and color information, respectively. Specifically, the upper branch uses full-scale skip connections to fuse texture details, while the lower branch learns color features within and across image channels to keep color fidelity. The features of the two branches are merged via cross-space attention blocks. As to discrimination, two discriminators are utilized to fully integrate and balance the contributions of the RGB map and the NIR map. Last but not the least, unsupervised loss functions are proposed in considerations of color, texture and adversary between the generator and the discriminator. The network is trained with a public dataset and a self-collected RGB-NIR dataset. Experimental results demonstrate that the algorithm fully fuses RGB and NIR images with fine details and plausible color, which is superior to most existing algorithms.