Towards understanding socio-cognitive behaviors of crowds from visual surveillance data

The problem of understanding socio-cognitive aspects of crowd behavior is a challenging yet critical task particularly for human-computer interaction applications. This issue is considered an important component of both current surveillance systems and futuristic interactions between intelligent agents and crowds of humans. In this paper, a probabilistic formulation of different categories of socio-cognitive crowd behavior is proposed. This framework models in a hierarchical manner relationships between the movements of entities within the crowd to differentiate between various crowd behaviors. Functionally, the framework can be considered a mid-level layer between purely visual analysis (i.e. detection and tracking of crowd components) and detailed semantics of crowd behaviors. In the presented study, the proposed framework extends a Gaussian Mixture Model of Dynamic Textures detection (GMM-of-DT) technique using Kalman filtering for the hierarchical motion representation of the crowds, simultaneously at micro (individual crowd members) and macro (groups of individuals) levels. Thereon, socio-cognitive crowd behaviors are categorized into individual, group, leader-follower and social interaction types. The proposed approach is validated on exemplary sequences from a benchmark PETS dataset (and, preliminarily, on other publicly available datasets), where the actual detection/tracking results are employed to evaluate probabilities of various socio-cognitive behaviors, and to compare the outputs of the proposed models to manually annotated ground-truth data.