Temporal resolution vs. visual saliency in videos: Analysis of gaze patterns and evaluation of saliency models

Temporal scalability of videos refers to the possibility of changing frame rate adaptively for efficient video transmission. Changing the frame rate may alter the spatial location that the viewers pay attention in the scene, which in turn significantly influences human's quality perception. Therefore, in order to effectively exploit the temporal scalability in applications, it is necessary to understand the relationship between frame rate variation and visual saliency. In this study, we answer the following three research questions: (1) Does the frame rate influence the overall gaze patterns (in an average sense over subjects)? (2) Does the frame rate influence the inter-subject variability of the gaze patterns? (3) Do the state-of-the-art saliency models predict human gaze patterns reliably for different frame rates? To answer the first two questions, we conduct an eye-tracking experiment. Under a free viewing scenario, we collect and analyze gaze-paths of human subjects watching high-definition (HD) videos having a normal or low frame rate. Our results show that both the average gaze-path and subject-wise variability of the gaze-path are influenced by frame rate variation. Then, we apply representative state-of-the-art saliency models to the videos and evaluate their performance by using the gaze pattern data collected from the eye-tracking experiment in order to answer the third question. It is shown that there exists a trade-off relation between accuracy in predicting the gaze pattern and robustness to frame rate variation, which raises necessity of further research in saliency modeling to simultaneously achieve both accuracy and robustness. (C) 2015 Elsevier B.V. All rights reserved.