Spatial-temporal Graph Inference with Granger Causality Relation for Group Activity Analysis

Causality reflects the directional effect from the active actor to the passive actor and commonly exists in group interactions. The difficulty in causality detection lies in the complex temporal dynamics of sequential features of the interacting actors. Existing methods use recurrent neural networks to describe the temporal dynamics of the interaction relations. Some methods use temporal attention mechanisms to describe temporal dependencies. They neglect to analyze the dependency between two actors， and are hard to distinguish the active actor and passive actor in the interaction. In this work，we design a Granger causality-based spatiotemporal graph model to learn the active-passive relations between interacting actors. To detect the Granger causality，the model designs an autoregression function for single individual temporal sequential features to describe the dependence of action on the individual itself. The model designs a correlative regression function for two individual temporal sequential features to describe the dependence of action on two individuals. The model detects the correlative individual as an active individual and the other as a passive individual by comparing the autoregressive error with the correlative regression error， when the autoregressive error is significantly larger than the correlative regression error， which indicates that the correlative individuals change the action of the other individual. The correlative regression function considers two individual temporal sequential features with multiple time delays，which can be used to learn the amount of time delay for actions between two individuals. This time delay amount is used to align the active individual time features with the passive individual time features. The temporally aligned active individual features provide the temporal and spatial contextual features of the passive individual and are fused with the passive individual features at the channel-wise level. The model constructs causal graphs of multi-scale spatiotemporal features to fully describe the interaction between appearance patterns， location constraints， and Granger causality among individuals. The multi-scale causal graph embeds the contextual features into the individual features and group features with graph inference. Experiments compare with state-of-the-art methods on Volleyball and Collective Activity datasets. （1）The spatial relation pooling model， such as Hierarchical Deep Temporal Model （HDTM）. （2） The spatial relation graph models include Social Scene Understanding model （SSU）， Convolutional Relational Machine （CRM）， Hierarchical Relational Machine model （HRN）， Actor Relation Graph model （ARG）， Graph Attention Interaction Model （GAIM）， Actor-Transformer （AT）， Position Distribution and Appearance Relation model （PDAR）， Multi-level Interaction Relation model（MLIR）. （3）The spatial-temporal relation model includes Confidence-Energy Recurrent Network （CERN）， spatial-temporal attentive graph network （stagNet）， Progressive Relation Learning model （PRL）， Graph LSTM-in-LSTM model （GLIL）， Visual Context model （VC）， GroupFormer （GF）， Partial context embedding（PCE），Coherence Constrained Graph LSTM（CCGLSTM）. Our Granger causality-based relation detector can describe the relations between potential active actors and passive actors. The channel-wise temporal causality graph inference module can enhance the feature of the passive actor by fusing a temporal delayed feature of the active actor. The graph model use Granger causality relation can describe effective interaction between actors and provide contextual features for group activity recognition. © 2023 Science Press. All rights reserved.