A hybrid CNN and BLSTM network for human complex activity recognition with multi-feature fusion

A hybrid convolutional neural network (CNN) and bidirectional long short-term memory (BLSTM) network for human complex activity recognition with multi-feature fusion is proposed in this paper. Specifically, a new CNN model is designed to extract the spatial features from the sensor data. Considering that in the process of activity recognition, the output at the current moment is not only related to the previous state, but also to the subsequent state. BLSTM network is further used to extract the temporal context of state information to improve the performance of activity recognition. In order to fully mine the features from the sensor data and further improve the performance of activity recognition, a new feature selection method named SFSANW (sequential forward selection and network weights), which is based on sequential forward selection algorithm and network weights is proposed to select features extracted by the traditional methods to obtain dominant features. The dominant features are then fused with the feature vectors extracted by the hybrid CNN and BLSTM network. Experiments are performed on two complex activity datasets, PAMAP2 and UT-Data, and 92.23% and 98.07% F1 scores are obtained, respectively. The experimental results demonstrate that the proposed method can achieve better performance of complex activity recognition, which is superior to the traditional machine learning algorithms and the state-of-the-art deep learning algorithms.