Facial Expression Recognition Using Salient Features and Convolutional Neural Network

A depth camera-based novel method is proposed here for efficient facial expression recognition. For each pixel in a depth image, eight local directional strengths are obtained and ranked. Once the rank of all pixels is obtained, eight histograms are developed for the eight surrounding directions. The histograms are then concatenated to represent features for a depth image of a face. This approach is named local directional rank histogram pattern (LDRHP). To combine with LDRHP features, one more robust feature extraction technique named local directional strength pattern (LDSP) is proposed. Typical local directional pattern (LDP) considers only absolute values of edge strengths for a pixel. This generalization in LDP may generate the same patterns for two different kinds of edge pixels. LDSP can overcome this problem. It considers the binary values of the position with the directions representing the highest and lowest original strengths. The highest strength indicates the strongest direction on the bright side of a pixel and the lowest one indicates the strongest direction in the dark side of that pixel. Hence, combining binary positions of these two directions can generate more robust patterns than LDP. Besides, LDSP pattern of a pixel is of six bits, whereas traditional LDP-based patterns are of eight bits (e.g., local directional deviation-based pattern and local directional position pattern). Thus, LDSP reduces the dimension of features with the same time adding robustness. For a depth image in a depth video, LDSP features are augmented with LDRHP features followed by Kernel principal component analysis and generalized discriminant analysis to generate more robust features. At last, the features are trained with a deep learning approach and convolutional neural network for successful facial expression recognition. The proposed approach is compared and shown to outperform the traditional expression recognition methods.