Self-attention binary neural tree for video summarization

In this paper, we address the problem of shot-level video summarization, which aims at selecting a subset of video shots as a summary to represent the original video contents compactly and completely. Most existing methods rely on various network architectures to learn a single score predictor for shot ranking and selection. Different from these methods, we plug network feature learning into a binary neural tree to consider multi-path predictions for each shot, thus enabling the shot evaluation from different aspects. Due to the hierarchical structure of the tree, video shots can be coarse-to-fine encoded by imposing self-attention on them along branches, leading to favorable predictions. Extensive experiments were conducted on two real-world datasets, and the results reveal that the proposed method achieves superior performance in comparison with previous state-of-the-art methods. (c) 2020 Elsevier B.V. All rights reserved.