Video saliency prediction via spatio-temporal reasoning

Video saliency detection often suffers from two issues: hard to disentangle the temporal motion patterns and spatial layout patterns, and hard to capture the temporal motion patterns. Thus a novel deep learning network architecture is proposed for video saliency in this paper. The proposed network consists of three parts: high-level representation module, attention module, and memory and reasoning module. The high-level representation module and attention module are used for capturing spatial saliency that is mainly learned from static images. The memory and reasoning module is used to infer the saliency from the information about spatial layout in frames and temporal motion between frames. Because high-level representation module and attention module could concentrate on high-level representation of spatial patterns, and the memory and reasoning module could concentrate on spatial and temporal saliency reasoning, the temporal patterns and spatial patterns could be disentangled efficiently. The quantitative and qualitative results show the proposed method achieves a promising results across a wide of metrics. (c) 2021 Elsevier B.V. All rights reserved.