Spatiotemporal consistency enhancement self-supervised representation learning for action recognition

Self-supervised learning has shown enormous potential in extracting valuable features from abundant unlabeled image data. However, for video, it requires models with powerful representation capabilities to exploit the rich spatiotemporal information to fully explore the internal relationships between different instances. This paper describes a novel spatiotemporal consistency enhancement self-supervised representation learning for action recognition. In contrast to typical contrastive learning methods, which merely use positive-negative pairs to learn invariant features, in this work, we design data augmentation of spatiotemporal information for feature similarity comparison. Specifically, we first extract the motion information from the video frames to keep the same action as those belonging to the original video. Further, we add static frames to these motion features to construct distracting video positive samples to mitigate the effect of irrelevant variables on model discrimination. In addition, we corrupt the sequence of video frames to generate extra categories of negative samples and distinguish them from the original frames by temporal differences. Ultimately, the learned helpful features are used for the downstream action recognition task, and the experimental results show that the method improves the recognition accuracy of the UCF101 and HMDB51 video action datasets.