Viewpoint guided multi-stream neural network for skeleton action recognition

Skeleton-based human action recognition has attracted considerable attention and succeeded in computer vision. However, one of the main challenges for skeleton action recognition is the complex viewpoint variations. Moreover, existing methods may be prone to develop the complicated networks with large model size. To this end, in this paper, we introduce a novel viewpoint-guided feature by adaptively selecting the optimal observation point to deal with the viewpoint variation problem. Furthermore, we present a novel multi-stream neural network for skeleton action recognition, namely Viewpoint Guided Multi-stream Neural Network (VGMNet). In particular, by incorporating four streams from spatial and temporal information, the proposed VGMNet can effectively learn the discriminative features of the skeleton sequence.We validate our method on three famous datasets, i.e., SHREC, NTU RGB+D, and Florence 3D. On SHREC, our proposed method has achieved better performance in terms of accuracy and efficiency against the state-of-the-art approaches. Furthermore, the highest scores on Florence 3D and NTU RGB+D show that our method is suitable for real application scenario with edge computing, and compatible to the case of multi-person action recognition.