RESIDUAL CONVOLUTIONAL NEURAL NETWORK BASED IN-LOOP FILTER WITH INTRA AND INTER FRAMES PROCESSED RESPECTIVELY FOR AVS3

Inspired by that deep learning based methods have made great progress in image enhancement tasks, we propose a deep residual convolutional neural network based in-loop filter to suppress compression artifacts for the third generation of Audio Video Standard (AVS3). The proposed network consists of several residual blocks, with quantization parameters (QPs) fed into the model. The QP map can help the network to learn the relationship between QPs and the degree of compression distortion. Therefore, different reconstructed frames with different distortion degrees can be enhanced by a single model. In addition, we analyze the difference between intra prediction and inter prediction in terms of compression distortion, and propose to process intra and inter frames respectively. Comparing to the AVS3 reference software (HPM-5.0), we achieve 8.66% and 8.75% BD-rate reduction on Y component under RA and LD configurations separately.