One-shot learning gesture recognition based on joint training of 3D ResNet and memory module

As a research hotspot in the field of human-machine interaction, a great progress of hand gesture recognition has been achieved with the development of deep learning of neural networks. However, in the deep learning based recognition methods, it is necessary to rely heavily on large-scale labeled dataset which is very hard to build in practical applications. In order to achieve a well performance under some strict constraint of few sample data, one-shot learning gesture recognition is studied and a joint deep training method by combination of 3D ResNet with a memory module is presented in this paper. In our scheme a combinatorial optimization of feature extraction by 3D ResNet with memory capacity of rare event by memory module is carried out with an effective strategy of optimal decision and two relative performance indices. In order to implement one-shot learning gesture recognition, the memory module is employed to remember the features extracted by well-trained 3D ResNet and the classification decision is performed by the nearest neighbor algorithm with cosine similarity measure. In view of real-world applications about human-machine interaction technology, its ability to deal with negative samples plays a significant role thus a mechanism based on the threshold of cosine similarity is built to realize effective classification and rejection respectively. In order to validate and evaluate the performance of our proposed method, a special hand gesture dataset containing 3045 gesture videos is built and a series of experiment results on our collected dataset and public datasets demonstrate the feasibility and effectiveness of our method.