Multimodal Fusion Classification Network Based on Distance Confidence Score

被引：0
|
作者：
Zheng D. [1 ,2 ]
Yang Y. [1 ]
Huang H. [3 ]
Xie Z. [1 ,2 ]
Li W. [3 ]
机构：
[1] Laboratory for Medical Imaging Informatics, Shanghai Institute of Technical Physics, Chinese Academy of Sciences, Shanghai
[2] University of Chinese Academy of Sciences, Beijing
[3] Fudan University Shanghai Cancer Center, Shanghai
来源：
Shanghai Jiaotong Daxue Xuebao/Journal of Shanghai Jiaotong University | 2022年 / 56卷 / 01期
关键词：
Confidence; Multimodal fusion; Neural network;
D O I：
10.16183/j.cnki.jsjtu.2020.186
中图分类号：
学科分类号：
摘要：
Multimodal data modeling can effectively overcome the problem of insufficient information in a single mode and can greatly improve the performance of model. However, not much progress has been made in quantifying the confidence of neural network models, especially for multimodal fusion models. This paper proposes a method based on embedding, which calculates the local density estimation in the embedding space by calculating the distance between samples, and then calculates the confidence score of the model. The proposed method is scalable and can be used not only for a single modal model, but also for the confidence measurement of multimodal fusion model. In addition, it can also evaluate and quantify the influences of different modal data on the multimodal fusion model. © 2022, Editorial Board of Journal of Shanghai Jiao Tong University. All right reserved.
引用
收藏
页码：89 / 100
页数：11
相关论文
共 30 条
  • [1] MANDELBAUM A, WEINSHALL D., Distance-based confidence score for neural network classifiers
  • [2] SHAKED A, WOLF L., Improved stereo matching with constant highway networks and reflective confidence learning, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 6901-6910, (2017)
  • [3] ZHANG Y B, ZHANG Z F, MIAO D Q, Et al., Three-way enhanced convolutional neural networks for sentence-level sentiment classification, Information Sciences, 477, pp. 55-64, (2019)
  • [4] DEL-AGUA M A, GIMENEZ A, SANCHIS A, Et al., Speaker-adapted confidence measures for ASR using deep bidirectional recurrent neural networks, IEEE/ACM Transactions on Audio, Speech, and Language Processing, 26, 7, pp. 1198-1206, (2018)
  • [5] NADEEM U, BENNAMOUN M, SOHEL F, Et al., Learning-based confidence estimation for multi-modal classifier fusion, International Conference on Neural Information Processing, pp. 299-312, (2019)
  • [6] BERTHON A, HAN B, NIU G, Et al., Confidence scores make instance-dependent label-noise learning possible
  • [7] SZEGEDY C, ZAREMBA W, SUTSKEVER I, Et al., Intriguing properties of neural networks
  • [8] GAL Y, GHAHRAMANI Z., Dropout as a Bayesian approximation: Representing model uncertainty in deep learning, Proceedings of Machine Learning Research, 48, pp. 1050-1059, (2016)
  • [9] GUO C, PLEISS G, SUN Y, Et al., On calibration of modern neural networks, Proceedings of Machine Learning Research, 70, pp. 1321-1330, (2017)
  • [10] DENG B, JIA S, SHI D M., Deep metric learning-based feature embedding for hyperspectral image classification, IEEE Transactions on Geoscience and Remote Sensing, 58, 2, pp. 1422-1435, (2020)
123