HybridNet: Integrating GCN and CNN for skeleton-based action recognition

Graph convolutional networks (GCNs) can well-preserve the structure information of the human body. They have achieved outstanding performance in skeleton-based action recognition. Nevertheless, there are still some issues with existing GCN-based methods. First, all channels have the same adjacency matrix. However, the correlations between joints are complex and may drastically change depending on the actions. These correlations are difficult to fit by merely channel-shared adjacency matrices. Second, the interframe edges of graphs only connect the same joints, neglecting the dependencies between the different joints. Fortunately, convolutional neural networks (CNNs) can simultaneously establish the interdependence of all the points in a spatial-temporal patch. Furthermore, CNNs use different kernels among channels. They are more adaptable for modeling complicated dependencies. In this work, we design a hybrid network (HybridNet) to integrate GCNs and CNNs. The HybridNet not only utilizes structural information well but also models complicated relationships between interframe joints properly.Extensive experiments are conducted on three challenging datasets: NTU-RGB+D, NTU-RGB+D 120, and Skeleton-Kinetics. The proposed model achieves state-of-the-art performance on all these datasets by a considerable margin, demonstrating the superiority of our method.