An Immersive System with Multi-modal Human-computer Interaction

被引：21

作者：

Zhao, Rui ^{[1
]}

Wang, Kang ^{[1
]}

Divekar, Rahul ^{[2
]}

Rouhani, Robert ^{[3
]}

Su, Hui ^{[3
,4
]}

Ji, Qiang ^{[1
]}

机构：

[1] RPI, Dept Elect Comp & Syst Engn, Troy, NY 12180 USA

[2] RPI, Dept Comp Sci, Troy, NY USA

[3] RPI, Cognit Immers Syst Lab, Troy, NY USA

[4] IBM Corp, TJ Watson Res Ctr, Armonk, NY 10504 USA

来源：

PROCEEDINGS 2018 13TH IEEE INTERNATIONAL CONFERENCE ON AUTOMATIC FACE & GESTURE RECOGNITION (FG 2018) | 2018年

关键词：

human-computer interaction system; multi-modal sensor fusion;

D O I：

10.1109/FG.2018.00083

中图分类号：

TP18 [人工智能理论];

学科分类号：

081104 ; 0812 ; 0835 ; 1405 ;

摘要：

We introduce an immersive system prototype that integrates face, gesture and speech recognition techniques to support multi-modal human-computer interaction capability. Embedded in an indoor room setting, a multi-camera system is developed to monitor the user facial behavior, body gesture and spatial location in the room. A server that fuses different sensor inputs in a time-sensitive manner so that our system knows who is doing what at where in real-time. When correlating with speech input, the system can better understand the user intention for interaction purpose. We evaluate the performance of core recognition techniques on both benchmark and self-collected datasets and demonstrate the benefit of the system in various use cases.

引用

页码：517 / 524

页数：8

共 50 条

[1] Multi-modal based human-computer interaction system for children
[J]. Yang, Z.-H., 1600, Asian Network for Scientific Information (12):
[2] Multi-modal Human-Computer Virtual Fusion Interaction In Mixed Reality
Jia, Shengying
[J]. JOURNAL OF APPLIED SCIENCE AND ENGINEERING, 2023, 26 (11): : 1609 - 1618
[3] On Annotation and Evaluation of Multi-modal Corpora in Affective Human-Computer Interaction
Kaechele, Markus
Schels, Martin
Meudt, Sascha
Kessler, Viktor
Glodek, Michael
Thiam, Patrick
Tschechne, Stephan
Palm, Guenther
Schwenker, Friedhelm
[J]. MULTIMODAL ANALYSES ENABLING ARTIFICIAL AGENTS IN HUMAN-MACHINE INTERACTION, 2015, 8757 : 35 - 44
[4] An application of multi-modal human-computer interaction -: The chess player Turk 2
Sajo, Levente
Kovacs, Gyoergy
Fazekas, Attila
[J]. 2008 IEEE INTERNATIONAL CONFERENCE ON AUTOMATION, QUALITY AND TESTING, ROBOTICS (AQTR 2008), THETA 16TH EDITION, VOL II, PROCEEDINGS, 2008, : 316 - 319
[5] Pen-based gesture recognition in multi-modal human-computer interaction
Wang, Y.J.
Yuan, B.Z.
[J]. Beifang Jiaotong Daxue Xuebao/Journal of Northern Jiaotong University, 2001, 25 (02):
[6] Multi-modal hand gesture designing in multi-screen touchable teaching system for human-computer interaction
Liu, Tingting
Chen, Zengzhao
Liu, Hai
Zhang, Zhaoli
Chen, Yingying
[J]. ICAIP 2018: 2018 THE 2ND INTERNATIONAL CONFERENCE ON ADVANCES IN IMAGE PROCESSING, 2018, : 198 - 202
[7] Human-computer interaction for alert warning and attention allocation systems of the Multi-Modal Watchstation
Obermayer, RW
Nugent, WA
[J]. INTEGRATED COMMAND ENVIRONMENTS, 2000, 4126 : 14 - 22
[8] Experimental Study on Appropriate Reality of Agents as a Multi-modal Interface for Human-Computer Interaction
Tanaka, Kaori
Matsui, Tatsunori
Kojima, Kazuaki
[J]. HUMAN-COMPUTER INTERACTION: INTERACTION TECHNIQUES AND ENVIRONMENTS, PT II, 2011, 6762 : 613 - 622
[9] Immersive 3D Human-Computer Interaction System
Wang, Po-Hsien
Wang, Ting-Ying
Chang, Ya-Chu
Huang, Ching-Chun
[J]. 2018 IEEE INTERNATIONAL CONFERENCE ON CONSUMER ELECTRONICS-TAIWAN (ICCE-TW), 2018,
[10] Multi-modal hands-free human computer interaction: A prototype system
Frangeskides, Frangiskos
Lanitis, Andreas
[J]. ICEIS 2006: PROCEEDINGS OF THE EIGHTH INTERNATIONAL CONFERENCE ON ENTERPRISE INFORMATION SYSTEMS: HUMAN-COMPUTER INTERACTION, 2006, : 19 - +

← 1 2 3 4 5 →