User Navigation Modeling, Rate-Distortion Analysis, and End-to-End Optimization for Viewport-Driven 360° Video Streaming

被引:4
|
作者
Chakareski, Jacob [1 ]
Corbillon, Xavier [2 ]
Simon, Gwendal [3 ]
Swaminathan, Viswanathan [4 ]
机构
[1] New Jersey Inst Technol, Coll Comp, Newark, NJ 07103 USA
[2] Tiledmedia, Rotterdam, Zuid Holland, Netherlands
[3] Synmedia, Networking, Rennes, France
[4] Adobe, Adobe Res, San Jose, CA USA
基金
美国国家科学基金会; 美国国家卫生研究院;
关键词
Omnidirectional video; quality of experience; viewport-adaptive 360 degrees video streaming; rate-distortion analysis and optimization; user navigation modeling;
D O I
10.1109/TMM.2022.3201397
中图分类号
TP [自动化技术、计算机技术];
学科分类号
0812 ;
摘要
The emerging technologies of Virtual Reality (VR) and 360 degrees video introduce new challenges for state-of-the-art video communication systems. Enormous data volume and spatial user navigation are unique characteristics of 360 degrees videos that necessitate a space-time effective allocation of the available network streaming bandwidth over the 360 degrees video content to maximize the Quality of Experience (QoE) delivered to the user. Towards this objective, we investigate a framework for viewport-driven rate-distortion optimized 360 degrees video streaming that integrates the user view navigation patterns and the spatiotemporal ratedistortion characteristics of the 360 degrees video content to maximize the delivered user viewport video quality, for the given network/system resources. The framework comprises a methodology for assigning dynamic navigation likelihoods over the 360 degrees video spatiotemporal panorama, induced by the user navigation patterns, an analysis and characterization of the 360 degrees video panorama's spatiotemporal rate-distortion characteristics that leverage preprocessed spatial tilling of the content, and an optimization problem formulation and solution that capture and aim to maximize the delivered expected viewport video quality, given a user's navigation patterns, the 360 degrees video encoding/streaming decisions, and the available system/network resources. We formulate a Markov model to capture the navigation patterns of a user over the 360 degrees video panorama and simultaneously extend our actual navigation datasets by synthesizing additional realistic navigation data. Moreover, we investigate the impact of using two different tile sizes for equirectangular tiling of the 360 degrees video panorama. Our experimental results demonstrate the advantages of our framework over the conventional approach of streaming a monolithic uniformly-encoded 360 degrees video and a state-of-the-art navigation-speed based reference method. Considerable average and instantaneous viewport video quality gains of up to 5 dB are demonstrated in the case of five popular 4 K 360 degrees videos. In addition, we explore the impact of two different popular 360 degrees video quality metrics applied to evaluate the streaming performance of our system framework and the two reference methods. Finally, we demonstrate that by exploiting the unequal rate-distortion characteristics of the different spatial sectors of the 360 degrees video panorama, we can enable spatially more uniform and temporally higher 360 degrees video viewport quality delivered to the user, relative to monolithic streaming.
引用
收藏
页码:5941 / 5956
页数:16
相关论文
共 50 条
  • [31] End-to-End Distortion Optimized Error Control for Real-time Wireless Video Streaming
    Peng, Guangchao
    Liu, Yanwei
    Hu, Yahui
    Ci, Song
    Tang, Hui
    2011 IEEE 13TH INTERNATIONAL WORKSHOP ON MULTIMEDIA SIGNAL PROCESSING (MMSP), 2011,
  • [32] Channel modeling and its effect on the end-to-end distortion in wireless video communications
    Soyak, E
    Eisenberg, Y
    Zhai, F
    Berry, R
    Pappas, TN
    Katsaggelos, AK
    ICIP: 2004 INTERNATIONAL CONFERENCE ON IMAGE PROCESSING, VOLS 1- 5, 2004, : 3253 - 3256
  • [33] SPHERICAL DOMAIN RATE-DISTORTION OPTIMIZATION FOR 360-DEGREE VIDEO CODING
    Li, Yiming
    Xu, Jizheng
    Chen, Zhenzhong
    2017 IEEE INTERNATIONAL CONFERENCE ON MULTIMEDIA AND EXPO (ICME), 2017, : 709 - 714
  • [34] Performability analysis of an adaptive-rate video-streaming service in end-to-end QoS scenarios
    Martín, IV
    Alins, JJ
    Aguilar-Igartua, M
    Mata-Díaz, J
    AMBIENT NETWORKS, 2005, 3775 : 157 - 168
  • [35] End-to-end stereoscopic video streaming with content-adaptive rate and format control
    Aksay, Anil
    Pehlivan, Selen
    Kurutepe, Engin
    Bilen, Cagdas
    Ozcelebi, Tanir
    Akar, Gozde Bozdagi
    Civanlar, M. Reha
    Tekalp, A. Murat
    SIGNAL PROCESSING-IMAGE COMMUNICATION, 2007, 22 (02) : 157 - 168
  • [36] Learning True Rate-Distortion-Optimization for End-To-End Image Compression
    Brand, Fabian
    Fischer, Kristian
    Kopte, Alexander
    Kaup, Andre
    DCC 2022: 2022 DATA COMPRESSION CONFERENCE (DCC), 2022, : 443 - 443
  • [37] Optimized video streaming over lossy networks with real-time estimation of end-to-end distortion
    Zhang, R
    Regunathan, SL
    Rose, K
    IEEE INTERNATIONAL CONFERENCE ON MULTIMEDIA AND EXPO, VOL I AND II, PROCEEDINGS, 2002, : 861 - 864
  • [38] End-to-End Distortion Modeling for Error-Resilient Screen Content Video Coding
    Tang, Tong
    Yin, Zhiyang
    Li, Jie
    Wang, Honggang
    Wu, Dapeng
    Wang, Ruyan
    IEEE TRANSACTIONS ON MULTIMEDIA, 2024, 26 : 4458 - 4468
  • [39] Spherical Position Dependent Rate-Distortion Optimization for 360-degree Video Coding
    Liu, Yuyang
    Guo, Hongwei
    Zhu, Ce
    Liu, Yipeng
    2019 ASIA-PACIFIC SIGNAL AND INFORMATION PROCESSING ASSOCIATION ANNUAL SUMMIT AND CONFERENCE (APSIPA ASC), 2019, : 992 - 996
  • [40] Saliency-driven rate-distortion optimization for 360-degree image coding
    Jui-Chiu Chiang
    Cheng-Yu Yang
    Bhishma Dedhia
    Yi-Fan Char
    Multimedia Tools and Applications, 2021, 80 : 8309 - 8329