Deep Learning and Video Quality Analysis: Towards A Unified VQA

Video makes up 80% of internet traffic today and is still rising. Most of it is meant for human consumption. But for 40 years, the video coding industry has been using mean-squared error-based PSNR, effectively the most basic full reference (FR) video quality measure, as the main tool for assessing video quality despite long known poor correlation to subjective video ratings. Moreover, in the encoder, the sum of absolute differences (SAD) is used instead of MSE to save multiplications. Meanwhile, many current video applications such as YouTube do not have access to a pristine reference and have had to develop ad hoc methods to attempt to monitor the volumes of video in their servers in a challenging no reference (NR) setting. For this, they have in part leaned on the Gaussianity of natural scene statistics (NSS), and evaluating how video distortions affect or alter those statistics to create a measure of quality. An entire cottage industry has sprung up to create both full-reference and no-reference video quality assessment (FR-, NR-VQA) measures, that can adequately meet the needs for monitoring and stream selection, in the massive worldwide video services industry. These two fields have so far gone their separate ways, as there seemed no sensible way to bring them under one roof. In this paper, we attempt a first synthesis of FR and NR VQA, which we simply call FastVDO Quality (FVQ). It incorporates all the lessons learned from the Video Multi-Assessment Fusion (VMAF) algorithm introduced by Netflix in 2016, the NSS-based assessment concepts developed by Univ. of Texas and Google to treat the NR case, culminating in the algorithms VIIDEO and SLEEQ, as well as our own research over the past several years in using learning-based methods in VQA. We provide some early indications that this approach can bear fruit for both NR and FR-VQA and may even offer state-of-the-art results in each field.