Multi-scale Spatial and Temporal Feature Aggregation Graph Convolutional Network for Skeleton-Based Action Recognition

In the field of deep learning, skeleton data is widely used for action recognition. Currently, the recognition of human skeleton action based on Graph Convolutional Networks (GCNs) has occupied the main position and has achieved remarkable results. However, existing methods are not sufficiently expressive concerning temporal and spatial features. Therefore, we propose a Multi-scale Spatial and Temporal Feature Aggregation Graph Convolutional Network (MSTA-GCN) for skeleton-based action recognition, which can effectively aggregate features from spatial and temporal dimensions using a hierarchical structure. Specifically, we integrate the topology learning strategy with the edge convolution module to aggregate global and fine-grained features at the spatial dimension. On this basis, a multi-scale temporal convolution based on a temporal attention module is proposed to aggregate the node features that change within frames under the condition of guaranteeing the global temporal features. Finally, the feature refinement module of skeleton data is improved to enhance the ability of the network to represent spatial features. Our proposed MSTA-GCN outperforms most mainstream methods and achieves satisfactory performance on three large-scale datasets: NTU RGB+D 60, NTU RGB+D 120, and Northwestern-UCLA.