Interpreting Super-Resolution CNNs for Sub-Pixel Motion Compensation in Video Coding

被引:2
|
作者
Murn, Luka [1 ,2 ]
Smeaton, Alan F. [2 ]
Mrak, Marta [1 ]
机构
[1] BBC Res & Dev, London, England
[2] Dublin City Univ, Dublin, Ireland
来源
PROCEEDINGS OF THE 29TH ACM INTERNATIONAL CONFERENCE ON MULTIMEDIA, MM 2021 | 2021年
关键词
video compression; motion compensation; interpolation; machine learning; complexity reduction;
D O I
10.1145/3474085.3478326
中图分类号
TP18 [人工智能理论];
学科分类号
081104 ; 0812 ; 0835 ; 1405 ;
摘要
Machine learning approaches for more efficient video compression have been developed thanks to breakthroughs in deep learning. However, they typically bring coding improvements at the cost of significant increases in computational complexity, making them largely unsuitable for practical applications. In this paper, we present open-source software for convolutional neural networkbased solutions which improve the interpolation of reference samples needed for fractional precision motion compensation. Contrary to previous efforts, the networks are fully linear, allowing them to be interpreted, with a full interpolation filter set derived from trained models, making it simple to integrate in conventional video coding schemes. When implemented in the context of the state-of-the-art Versatile Video Coding (VVC) test model, the complexity of the learned interpolation schemes is significantly reduced compared to the interpolation with full neural networks, while achieving notable coding efficiency improvements on lower resolution video sequences. The open-source software package is available at https://github.com/bbc/cnn- fractional-motion-compensation under the 3-clause BSD license.
引用
收藏
页码:3803 / 3806
页数:4
相关论文
共 50 条
  • [1] INTERPRETING CNN FOR LOW COMPLEXITY LEARNED SUB-PIXEL MOTION COMPENSATION IN VIDEO CODING
    Murn, Luka
    Blasi, Saverio
    Smeaton, Alan F.
    O'Connor, Noel E.
    Mrak, Marta
    2020 IEEE INTERNATIONAL CONFERENCE ON IMAGE PROCESSING (ICIP), 2020, : 798 - 802
  • [2] A video super-resolution reconstruction method based on sub-pixel registration
    School of Electronic and Information Engineering, Sichuan University, Chengdu 610064, China
    不详
    Guangdianzi Jiguang, 7 (972-976):
  • [3] Soccer Video Super-Resolution via Sub-Pixel Convolutional Neural Network
    Wang, Haoyu
    Lu, Yao
    2018 INTERNATIONAL JOINT CONFERENCE ON NEURAL NETWORKS (IJCNN), 2018,
  • [4] SUB-PIXEL MAPPING WITH HYPERSPECTRAL IMAGES USING SUPER-RESOLUTION
    Gaur, S.
    Buddhiraju, K. M.
    Porwal, A.
    IGARSS 2018 - 2018 IEEE INTERNATIONAL GEOSCIENCE AND REMOTE SENSING SYMPOSIUM, 2018, : 2459 - 2462
  • [5] Sub-pixel Layout for Super-Resolution with Images in the Octic Group
    Shi, Boxin
    Zhao, Hang
    Ben-Ezra, Moshe
    Yeung, Sai-Kit
    Fernandez-Cull, Christy
    Shepard, R. Hamilton
    Barsi, Christopher
    Raskar, Ramesh
    COMPUTER VISION - ECCV 2014, PT I, 2014, 8689 : 250 - 264
  • [6] Sub-Pixel Convolutional Neural Network for Image Super-Resolution Reconstruction
    Shao, Guifang
    Sun, Qiao
    Gao, Yunlong
    Zhu, Qingyuan
    Gao, Fengqiang
    Zhang, Junfa
    ELECTRONICS, 2023, 12 (17)
  • [7] Noniterative sub-pixel shifting super-resolution lensless digital holography
    Lee, Heejung
    Kim, Jongwu
    Kim, Junwoo
    Jeon, Philjun
    Lee, Seung Ah
    Kim, Dugyoung
    OPTICS EXPRESS, 2021, 29 (19) : 29996 - 30006
  • [8] Super-resolution image reconstruction algorithm based on sub-pixel shift
    Zhang, Dong-Xiao
    Lu, Lin
    Li, Cui-Hua
    Jin, Tai-Song
    Zidonghua Xuebao/Acta Automatica Sinica, 2014, 40 (12): : 2851 - 2861
  • [9] A Super-resolution Imaging System Based on Sub-Pixel Camera Shifting
    Liu, Gangping
    Zhou, Qun
    Zhang, Linxia
    Ke, Jun
    OPTOELECTRONIC IMAGING AND MULTIMEDIA TECHNOLOGY V, 2018, 10817
  • [10] Super-resolution compressed sensing imaging algorithm based on sub-pixel shift
    Xu, Bing
    Zhang, Xiaoping
    Wu, Xianjun
    CLUSTER COMPUTING-THE JOURNAL OF NETWORKS SOFTWARE TOOLS AND APPLICATIONS, 2019, 22 (Suppl 4): : S8407 - S8413