Identification of story units in audio-visual sequences by joint audio and video processing

In this paper, a novel technique, which uses a joint audio-visual analysis for scene identification and characterization, is proposed. The paper defines four different scene types: dialogues, stories, actions, and generic scenes. It then explains how any audio-visual material can be decomposed into a series of scenes obeying to the preview classification, by properly analyzing and then combining the underlying audio and visual information. A rule-based procedure is defined for such purpose. Before such rule-based decision can take place, a series of low-level pre-processing tasks care suggested to adequately measure audio and visual correlations. As far as visual information is concerned, it is proposed to measure similarities between non consecutive shots using a Learning Vector Quantization approach. An outlook on a possible implementation strategy for the overall scene identification task is suggested, and validated through a series of experimental simulations on real audio-visual data.