Performance Comparison of Subjective Assessment Methods for Stereoscopic 3D Video Quality

The International Telecommunication Union has standardized many subjective assessment methods for stereoscopic three-dimensional (3D) and 2D video quality. The same methods are used for 3D and 2D videos. The assessment time, stability, and discrimination ability, which means the ability to identify differences in video quality, are important factors in subjective assessment methods. Many studies on these factors have been done for 2D video quality. However, these factors for 3D video quality have not been sufficiently studied. To address this, we conduct subjective quality assessments for 3D and 2D videos using the absolute category rating (ACR), degradation category rating (DCR), and double stimulus continuous quality-scale (DSCQS) methods that are defined in ITU Recommendations. We first investigate the Pearson's correlation coefficients and Spearman's rank correlation coefficients between different pairings of the three methods to clarify which method is most efficient in terms of assessment time. The different pairings of the three methods exhibit high coefficients. These results indicate that the order relation of the mean opinion score (MOS) and the distance between the MOSs for these methods are almost the same. Therefore, for generally investigating the quality characteristics, the ACR method is most efficient because it has the shortest assessment time. Next, we analyze the stability of these subjective assessment methods. We clarify that the confidence intervals (CIs) of the MOSs for 3D video are almost the same as those for 2D video and that the stability of the DCR method is higher than that of the other methods. The DSCQS method has the smallest Cls for high-quality video. Finally, we investigate the discrimination ability of these subjective assessment methods. The results show that the DCR method performs better than the others in terms of the number of paired MOSs with a significant difference for low-quality video. However, we confirm that the DSCQS method performs better than the others for high-quality video.