Multi-modal Sign Language Recognition with Enhanced Spatiotemporal Representation

Sign language recognition (SLR) has become increasingly popular in recent years in computer vision. It is essential to extract discriminative spatiotemporal features to model the spatial and temporal evolutions of different signs. Also, local gesture and facial expression representations contribute to distinguishing signs with similar motion patterns but different meanings. In this paper, we propose a multi-modal sign language recognition framework, in the RGB representation model, we design the adaptive spatiotemporal attention modules to fulfill the visual cue definition in signing videos, and the adapter is designed for constructing the auxiliary task, which jointly learning with the SLR task to enhances the performance of the model. Given a signing video, a spatiotemporal attention-based Pseudo-3D Residual Networks (STA P3D ResNet) is used to learn spatiotemporal features mainly from the areas of interest and the key frames. After feature extraction, the attention-based Bidirectional Long Short-Term Memory Networks (Att-BLSTM) is utilized to select the significant motions. Meanwhile, from the skeletal data, we can obtain the texture image by color encoding and construct the spatial relation features, which are high level representations of human posture. The learnt skeleton-based features from skeletal data fused with the attention-aware video features to further provide more informative spatiotemporal information for SLR. Experiments are carried out on two large scale sign language datasets. And the experimental results demonstrate the effectiveness of our proposed method.