Neural network generation of adaptive filter and new applications in remote sensing image processing

Image adaptive filtering is a nonlinear image transformation, which has a wide range of applications. Traditional image adaptive filters are designed by experts, such as bilateral filter and shape adaptive filter. They can determine the shape, size, and weight of the filter based on the local structure and content of the image. They are commonly used to suppress noise while preserving the structural characteristics of the image. Convolutional Neural Networks (CNNs) are an effective tool for feature extraction and nonlinear expression. They can be used to learn and construct image adaptive filters. And this paper explores the application of nonlinear image adaptive filters generated by convolutional neural networks in image interpolation and image fusion. This paper introduces the generation network of an image adaptive filter, including its model structure and objective function. The common network structure usually employs an encoder-decoder architecture, which is mainly composed of three parts: feature extraction, feature recovery, and filter (convolution kernel) estimation. Then, the paper presents two different application scenarios of image adaptive filters: image interpolation and image fusion. The adaptive filter for images enables transformation between different phases during image interpolation and transformation between different bands during image fusion. In these two scenarios, the image adaptive filters are learned by the filter generation network based on the specific application scenario and then applied. In image interpolation applications, the image adaptive filter is used as a nonlinear transformation between two temporal images. The interpolated image is looked at as the mean of adaptive filtering of the previous temporal image and adaptive filtering of the latter temporal image. In image fusion applications, the image adaptive filter is used as a nonlinear fitting method to regress multispectral bands to the panchromatic band. It then extracts spatial details from the difference of the panchromatic band and the simulated panchromatic band, and finally adds spatial details to all the multispectral bands. We conducted experiments in two application scenarios. The first involved nonlinear transformation for image interpolation with different phases simultaneously. The second utilized an image adaptive filter as a nonlinear fitting method for multi-spectral band regression panchromatic band in image fusion. In image interpolation applications, the experimental results show that the interpolated results are consistent with the reference image in spatial and spectral characteristics, and the RMSE of the interpolated image with the reference image is relatively small. The experimental results for image fusion applications indicate that the low-resolution panchromatic band obtained through adaptive filter fitting of the multi-spectral band is more accurate than the traditional component replacement method. The fusion result obtained by nonlinear image adaptive filters has neither obvious spectral distortion nor obvious spatial distortion. From the application of nonlinear image adaptive filters generated by convolutional neural networks in image interpolation and image fusion, we have a glimpse of its application potential of image adaptive filter in constructing image nonlinear transformation. The filter generation network can generate adaptive filters for particular application scenarios, resulting in more accurate and visually pleasing images. © 2023 Science Press. All rights reserved.