Multilabel convolution neural network for facial expression recognition and ordinal intensity estimation

被引:3
|
作者
Ekundayo, Olufisayo [1 ]
Viriri, Serestina [1 ]
机构
[1] Univ KwaZulu Natal, Comp Sci Discipline, Durban, South Africa
关键词
Binary cross-entropy; Facial expression recognition; Island loss; Multilabel; Ordinal intensity estimation; FACE; EMOTION;
D O I
10.7717/peerj-cs.736
中图分类号
TP18 [人工智能理论];
学科分类号
081104 ; 0812 ; 0835 ; 1405 ;
摘要
Facial Expression Recognition (FER) has gained considerable attention in affective computing due to its vast area of applications. Diverse approaches and methods have been considered for a robust FER in the field, but only a few works considered the intensity of emotion embedded in the expression. Even the available studies on expression intensity estimation successfully assigned a nominal/regression value or classified emotion in a range of intervals. Most of the available works on facial expression intensity estimation successfully present only the emotion intensity estimation. At the same time, others proposed methods that predict emotion and its intensity in different channels. These multiclass approaches and extensions do not conform to man heuristic manner of recognising emotion and its intensity estimation. This work presents a Multilabel Convolution Neural Network (ML-CNN)-based model, which could simultaneously recognise emotion and provide ordinal metrics as the intensity estimation of the emotion. The proposed ML-CNN is enhanced with the aggregation of Binary Cross-Entropy (BCE) loss and Island Loss (IL) functions to minimise intraclass and interclass variations. Also, ML-CNN model is pre-trained with Visual Geometric Group (VGG-16) to control overfitting. In the experiments conducted on Binghampton University 3D Facial Expression (BU-3DFE) and Cohn Kanade extension (CK+) datasets, we evaluate ML-CNN's performance based on accuracy and loss. We also carried out a comparative study of our model with some popularly used multilabel algorithms using standard multilabel metrics. ML-CNN model simultaneously predicts emotion and intensity estimation using ordinal metrics. The model also shows appreciable and superior performance over four standard multilabel algorithms: Chain Classifier (CC), distinct Random K label set (RAKEL), Multilabel K Nearest Neighbour (MLKNN) and Multilabel ARAM (MLARAM).
引用
收藏
页数:24
相关论文
共 50 条
  • [1] Facial Expression Recognition Based on Convolution Neural Network
    Duan, Yue
    Zhou, Linli
    Wu, Yue
    [J]. PROCEEDINGS OF THE 2ND INTERNATIONAL CONFERENCE ON COMPUTER ENGINEERING, INFORMATION SCIENCE & APPLICATION TECHNOLOGY (ICCIA 2017), 2017, 74 : 339 - 343
  • [2] Finding Useful Features for Facial Expression Recognition and Intensity Estimation by Neural Network
    Imamura, Naoki
    Nomiya, Hiroki
    Hochin, Teruhisa
    [J]. INTERNATIONAL JOURNAL OF SOFTWARE INNOVATION, 2020, 8 (02) : 68 - 84
  • [3] Facial Expression Recognition Method Based on Cascade Convolution Neural Network
    Liu, Weida
    Fang, Jian
    [J]. IWCMC 2021: 2021 17TH INTERNATIONAL WIRELESS COMMUNICATIONS & MOBILE COMPUTING CONFERENCE (IWCMC), 2021, : 1012 - 1015
  • [4] Facial Expression Recognition Algorithm Based on Deep Convolution Neural Network
    Ivanovsky, Leonid
    Khryashchev, Vladimir
    Lebedev, Anton
    Kosterin, Igor
    [J]. PROCEEDINGS OF THE 2017 21ST CONFERENCE OF OPEN INNOVATIONS ASSOCIATION (FRUCT), 2017, : 141 - 147
  • [5] Facial Expression Intensity Estimation Using Ordinal Information
    Zhao, Rui
    Gan, Quan
    Wang, Shangfei
    Ji, Qiang
    [J]. 2016 IEEE CONFERENCE ON COMPUTER VISION AND PATTERN RECOGNITION (CVPR), 2016, : 3466 - 3474
  • [6] The Influence of the Activation Function in a Convolution Neural Network Model of Facial Expression Recognition
    Wang, Yingying
    Li, Yibin
    Song, Yong
    Rong, Xuewen
    [J]. APPLIED SCIENCES-BASEL, 2020, 10 (05):
  • [7] Conditional convolution neural network enhanced random forest for facial expression recognition
    Liu, Yuanyuan
    Yuan, Xiaohui
    Gong, Xi
    Xie, Zhong
    Fang, Fang
    Luo, Zhongwen
    [J]. PATTERN RECOGNITION, 2018, 84 : 251 - 261
  • [8] Facial Expression Recognition Using Enhanced Convolution Neural Network with Attention Mechanism
    Prabhu, K.
    SathishKumar, S.
    Sivachitra, M.
    Dineshkumar, S.
    Sathiyabama, P.
    [J]. COMPUTER SYSTEMS SCIENCE AND ENGINEERING, 2022, 41 (01): : 415 - 426
  • [9] Multi-output Laplacian Dynamic Ordinal Regression for Facial Expression Recognition and Intensity Estimation
    Rudovic, Ognjen
    Pavlovic, Vladimir
    Pantic, Maja
    [J]. 2012 IEEE CONFERENCE ON COMPUTER VISION AND PATTERN RECOGNITION (CVPR), 2012, : 2634 - 2641
  • [10] Human Facial Expression Recognition with Convolution Neural Networks
    Christou, Nikolaos
    Kanojiya, Nilam
    [J]. THIRD INTERNATIONAL CONGRESS ON INFORMATION AND COMMUNICATION TECHNOLOGY, 2019, 797 : 539 - 545