Down-scale simplified non-local attention networks with application to image denoising

Non-local (NL) attention modules or transformer-based methods have been widely applied in various image processing tasks. However, the computation of the long-range similarity is very expensive, which greatly limits the further application of the NL attention modules. Motivated by the recurrence law of image patches across different scales, we propose an efficient down-scale simplified NL (DSNL) attention module. In our method, the deep feature maps are divided into several feature maps in the coarse scales, which contain the cleaner version of feature patches in the original feature maps. Then the NL attention can be implemented on smaller and clearer feature maps. Numerical experiments ons image denoising demonstrate that the proposed attention module consistently outperforms the original patch-based NL attention modules on both visual quality and GPU time. The classical ResNet which integrates the proposed attention module can product favorable results compared to many state-of-the-art methods.