Human Activity Recognition from RGB Video Streams Using 1D-CNNs

Human activity recognition is a challenging problem of classifying human activities based on video streams or accelerometer data. Most activity recognition methods use depth enabled data or multi-modal data to precisely predict movements. As majority of the real-world applications of activity recognition use RGB cameras, this paper aims at using RGB-only videos to classify actions. Motivated by recent advances of 1-Dimensional Convolutional Neural Networks (CNNs) to classify sequential data, this paper investigates the application of a deep 1D-CNN model for activity recognition as compared to a Bi-Directional Long Short-Term Memory (BLSTM) method. Instead of using raw RGB frames as input to the network, we utilize OpenPose API to detect 2D skeleton key points in video frames, which are then used to recognize human actions. This paper also addresses the challenges of training deep models with limited labeled data by using data augmentation and dynamic frame dropout techniques to increase the efficiency of the model and avoid overfitting. We verify the performance of our model on three popular and challenging benchmarks: UTD-MHAD, KTH and UCF-Sports. Extensive experiments demonstrate that our 1D-CNN model outperforms the BLSTM model and all the state-of-the-art methods consistently.