Interpreting Super-Resolution CNNs for Sub-Pixel Motion Compensation in Video Coding

被引:2
|
作者
Murn, Luka [1 ,2 ]
Smeaton, Alan F. [2 ]
Mrak, Marta [1 ]
机构
[1] BBC Res & Dev, London, England
[2] Dublin City Univ, Dublin, Ireland
来源
PROCEEDINGS OF THE 29TH ACM INTERNATIONAL CONFERENCE ON MULTIMEDIA, MM 2021 | 2021年
关键词
video compression; motion compensation; interpolation; machine learning; complexity reduction;
D O I
10.1145/3474085.3478326
中图分类号
TP18 [人工智能理论];
学科分类号
081104 ; 0812 ; 0835 ; 1405 ;
摘要
Machine learning approaches for more efficient video compression have been developed thanks to breakthroughs in deep learning. However, they typically bring coding improvements at the cost of significant increases in computational complexity, making them largely unsuitable for practical applications. In this paper, we present open-source software for convolutional neural networkbased solutions which improve the interpolation of reference samples needed for fractional precision motion compensation. Contrary to previous efforts, the networks are fully linear, allowing them to be interpreted, with a full interpolation filter set derived from trained models, making it simple to integrate in conventional video coding schemes. When implemented in the context of the state-of-the-art Versatile Video Coding (VVC) test model, the complexity of the learned interpolation schemes is significantly reduced compared to the interpolation with full neural networks, while achieving notable coding efficiency improvements on lower resolution video sequences. The open-source software package is available at https://github.com/bbc/cnn- fractional-motion-compensation under the 3-clause BSD license.
引用
收藏
页码:3803 / 3806
页数:4
相关论文
共 50 条
  • [41] Video Super-Resolution with Compensation in Feature Extraction
    Zhou, Yang
    Chen, Lei
    Zhao, Jiying
    2019 IEEE PACIFIC RIM CONFERENCE ON COMMUNICATIONS, COMPUTERS AND SIGNAL PROCESSING (PACRIM), 2019,
  • [42] Super-Resolution Reconstruction Algorithm for Infrared Image with Double Regular Items Based on Sub-Pixel Convolution
    Yu, Lei
    Zhang, Xuewei
    Chu, Yan
    APPLIED SCIENCES-BASEL, 2020, 10 (03):
  • [43] RGB-IR Cross Input and Sub-Pixel Upsampling Network for Infrared Image Super-Resolution
    Du, Juan
    Zhou, Huixin
    Qian, Kun
    Tan, Wei
    Zhang, Zhe
    Gu, Lin
    Yu, Yue
    SENSORS, 2020, 20 (01)
  • [44] Super-Resolution for Monocular Depth Estimation With Multi-Scale Sub-Pixel Convolutions and a Smoothness Constraint
    Zhao, Shiyu
    Zhang, Lin
    Shen, Ying
    Zhao, Shengjie
    Zhang, Huijuan
    IEEE ACCESS, 2019, 7 : 16323 - 16335
  • [45] High-Frequency Feature Learning in Image Super-Resolution with Sub-Pixel Convolutional Neural Network
    Jiang Xiao-yuan
    Chen Xi-hai
    ICVIP 2019: PROCEEDINGS OF 2019 3RD INTERNATIONAL CONFERENCE ON VIDEO AND IMAGE PROCESSING, 2019, : 251 - 255
  • [46] Super-resolution lensless on-chip microscopy based on array illumination and sub-pixel shift search
    Li, Yulong
    Ma, Jianshe
    Cao, Liangcai
    Su, Ping
    OPTICS LETTERS, 2024, 49 (06) : 1620 - 1623
  • [47] Scaled transverse translation by planar optical elements for sub-pixel sampling and remote super-resolution imaging
    Zhang, Qi
    Xu, Xin
    Guo, Yinghui
    Lu, Yuran
    He, Qiong
    Pu, Mingbo
    Li, Xiaoyin
    Xu, Mingfeng
    Zhang, Fei
    Luo, Xiangang
    NANOPHOTONICS, 2025,
  • [48] RSPCN: Super-Resolution of Digital Elevation Model Based on Recursive Sub-Pixel Convolutional Neural Networks
    Zhang, Ruichen
    Bian, Shaofeng
    Li, Houpu
    ISPRS INTERNATIONAL JOURNAL OF GEO-INFORMATION, 2021, 10 (08)
  • [49] Super-Resolution based Video Coding Scheme
    Cho, Hyun Min
    Choi, Kiho
    2022 IEEE/CVF CONFERENCE ON COMPUTER VISION AND PATTERN RECOGNITION WORKSHOPS, CVPRW 2022, 2022, : 1777 - 1779
  • [50] On scalable lossless video coding based on sub-pixel accurate MCTF
    Yea, S
    Pearlman, WA
    VISUAL COMMUNICATIONS AND IMAGE PROCESSING 2006, PTS 1 AND 2, 2006, 6077