Early vs. Late Multimodal Fusion for Recognizing Confusion in Collaborative Tasks

被引：0

作者：

Ashwath, Anisha ^{[1
]}

Peechatt, Michael ^{[1
]}

Alm, Cecilia ^{[1
,2
]}

Bailey, Reynold ^{[1
]}

机构：

[1] Rochester Inst Technol, Golisano Coll Comp & Informat Sci, Rochester, NY 14623 USA

[2] Rochester Inst Technol, Coll Liberal Arts, Rochester, NY 14623 USA

来源：

2023 11TH INTERNATIONAL CONFERENCE ON AFFECTIVE COMPUTING AND INTELLIGENT INTERACTION WORKSHOPS AND DEMOS, ACIIW | 2023年

基金：

美国国家科学基金会;

关键词：

data fusion; multimodal data; affective computing; early fusion; late fusion;

D O I：

10.1109/ACIIW59127.2023.10388144

中图分类号：

TP18 [人工智能理论];

学科分类号：

081104 ; 0812 ; 0835 ; 1405 ;

摘要：

There has been a rapid transformation in the medium of learning and communication due to the pandemic. Multitudes have adopted online video platforms to learn and work from any corner of the world. Emotion detection is vital for understanding how well instructions are communicated through online interactions and for building cognitive systems that can identify human behavior. Confusion is a key emotion that can impact online learning and can be used to verify whether students using an online platform understand the material being taught. Our research expands on previous work regarding confusion detection, focusing on data fusion techniques. We explore the impact of early fusion (feature-level) vs late fusion (decision-level) on modeling confusion identification during a collaborative block building task. Experimenting with different classifiers, our results show that late fusion performs better with larger time windows. This fusion approach can aid in model interpretability.

引用

页数：4

共 50 条

[1] Early vs Late Fusion in Multimodal Convolutional Neural Networks
Gadzicki, Konrad
Khamsehashari, Razieh
Zetzsche, Christoph
[J]. PROCEEDINGS OF 2020 23RD INTERNATIONAL CONFERENCE ON INFORMATION FUSION (FUSION 2020), 2020, : 292 - 297
[2] Affect recognition from face and body: Early fusion vs. late fusion
Gunes, H
Piccardi, M
[J]. INTERNATIONAL CONFERENCE ON SYSTEMS, MAN AND CYBERNETICS, VOL 1-4, PROCEEDINGS, 2005, : 3437 - 3443
[3] Multimodal Fusion Strategies: Human vs. Machine
Ko, Hanseok
[J]. AVSU'18: PROCEEDINGS OF THE 2018 WORKSHOP ON AUDIO-VISUAL SCENE UNDERSTANDING FOR IMMERSIVE MULTIMEDIA, 2018, : 1 - 1
[4] Age and Gender Recognition Based on Multiple Systems - Early vs. Late Fusion
Bocklet, Tobias
Stemmer, Georg
Zeissler, Viktor
Noeth, Elmar
[J]. 11TH ANNUAL CONFERENCE OF THE INTERNATIONAL SPEECH COMMUNICATION ASSOCIATION 2010 (INTERSPEECH 2010), VOLS 3 AND 4, 2010, : 2834 - +
[5] Fusion vs. Two-Stage for Multimodal Retrieval
Arampatzis, Avi
Zagoris, Konstantinos
Chatzichristofis, Savvas A.
[J]. ADVANCES IN INFORMATION RETRIEVAL, 2011, 6611 : 759 - 762
[6] Early vs. late orthodontic treatment
Solomon, F
[J]. JOURNAL OF THE AMERICAN DENTAL ASSOCIATION, 2000, 131 (10): : 1406 - +
[7] Serial vs. Concurrent Scheduling of Transmission and Processing Tasks in Collaborative Systems
Junuzovic, Sasa
Dewan, Prasun
[J]. COLLABORATIVE COMPUTING: NETWORKING, APPLICATIONS AND WORKSHARING, 2009, 10 : 746 - 759
[8] Early vs. late orthodontic treatment - Response
Kluemper, GT
Beeman, CS
Hicks, EP
[J]. JOURNAL OF THE AMERICAN DENTAL ASSOCIATION, 2000, 131 (10): : 1410 - +
[9] Multimodal early fusion operators for temporal video scene segmentation tasks
Beserra, Antonio A. R.
Goularte, Rudinei
[J]. MULTIMEDIA TOOLS AND APPLICATIONS, 2023, 82 (20) : 31539 - 31556
[10] Multimodal early fusion operators for temporal video scene segmentation tasks
Antonio A. R. Beserra
Rudinei Goularte
[J]. Multimedia Tools and Applications, 2023, 82 : 31539 - 31556

← 1 2 3 4 5 →