Predicting skeleton trajectories using a Skeleton-Transformer for video anomaly detection

Video anomaly detection detects video contents that do not conform to normal patterns offered by the training set. Because appearance-based features are susceptible to background interference, unlike most papers applying appearance-based methods, this paper proposes a novel Skeleton-Transformer (SkT) to predict future pose components in video frames and take errors between predicted pose components and corresponding expected values as anomaly scores. In SkT, we apply the multi-head self-attention (MSA) module and temporal convolutional layer (TCL), which are complementary because they focus on processing information from different viewpoints, to compose a skeleton attention (SkA) block. The MSA module can capture long-range dependencies between arbitrary pairwise pose components on spatial and temporal dimensions from different perspectives, while the TCL concentrates on local temporal information. Finally, multiple SkA blocks are stacked to form the major constituent of the SkT. To the best of our knowledge, the proposed approach is the first work applying Transformer framework to anomaly detection based on pose components, and we conduct experiments to determine the optimal structure. The proposed method achieves a frame-level AUC of 77.65% on the HR-ShanghaiTech dataset, exceeding state-of-the-art methods. Moreover, ablation studies validate each module’s effectiveness in the SkT, further verifying that the Transformer-based method is promising for anomaly detection.