Dual-domain graph convolutional networks for skeleton-based action recognition

Skeleton-based action recognition is attracting more and more attention owing to the general representation ability of skeleton data. The Graph Convolutional Networks (GCNs) methods extended from Convolutional Neural Networks (CNNs) are proposed to directly extract spatial-temporal information from the graphs. Previous GCNs usually aggregate the skeleton information locally in the vertex domain. However, the focus on the local information brought about the limited representation ability in some actions containing overall dynamics in both spatial and temporal, which pulled down the overall accuracy of the model. Therefore, this paper proposes a more comprehensive two-stream GCN architecture containing the vertex-domain graph convolution and the spectral graph convolution based on Graph Fourier Transform (GFT). One stream utilizes an efficient vertex-domain graph convolution to obtain effective spatial-temporal information via Graph Shift Blocks (GSB), while the other brings the global spectral information from our improved Residual Spectral Blocks (RSB). According to the analysis of the experimental results, the action misalignment for certain actions is reduced. Moreover, along with other GCN methods that only focus on spatial-temporal information, our RSB strategies help improve their performance. DD-GCN is evaluated on three large skeleton-based datasets, NTU-RGBD 60, NTU-RGBD 120, and Kinetics-Skeleton. The experiment results demonstrate a comparable ability to the state-of-the-art.