Virtual facial expression recognition using deep CNN with ensemble learning

In the current era, virtual environments and virtual characters have become popular. In the near future, recognition of virtual facial expressions plays an important role in virtual assistants, online video games, security systems, entertainment, psychological study, video conferencing, virtual reality, and online classes. The objective of this work is to recognize the facial emotions of virtual characters. Facial expression recognition (FER) from virtual characters is a difficult task due to its intra-class variation and inter-class similarity. The performances of existing FER systems are limited in this aspect. To address these challenges, we designed and developed a multi-block deep convolutional neural networks (DCNN) model to recognize the facial emotions from virtual, stylized and human characters. In multi-block DCNN, we defined four blocks with various computational elements to extract the discriminative features from facial images. To increase stability and to make better predictions two more models were proposed using ensemble learning which are bagging ensemble with SVM (DCNN-SVM), and the ensemble of three different classifiers with a voting technique (DCNN-VC). Image data augmentation was applied to expand the dataset to improve model performance and generalization. The accuracy of the proposed DCNN model was studied by tuning hyperparameters. Performances of the three proposed models were examined in contrast with pre-trained models such as VGGNet-19, ResNet50 with a voting technique for emotion recognition. The proposed models are evaluated and achieved the best accuracy when compared with other models on five publicly available facial emotion datasets that include UIBVFED, FERG, CK+, JAFFE, and TFEID.