Asynchronous Multimodal Text Entry using Speech and Gesture Keyboards

被引：0

作者：

Kristensson, Per Ola ^{[1
]}

Vertanen, Keith ^{[2
]}

机构：

[1] Univ St Andrews, Sch Comp Sci, St Andrews KY16 9AJ, Fife, Scotland

[2] Princeton Univ, Dept Comp Sci, Princeton, NJ USA

来源：

12TH ANNUAL CONFERENCE OF THE INTERNATIONAL SPEECH COMMUNICATION ASSOCIATION 2011 (INTERSPEECH 2011), VOLS 1-5 | 2011年

基金：

英国工程与自然科学研究理事会;

关键词：

mobile text entry; multimodal interfaces;

D O I：

暂无

中图分类号：

TP18 [人工智能理论];

学科分类号：

081104 ; 0812 ; 0835 ; 1405 ;

摘要：

We propose reducing errors in text entry by combining speech and gesture keyboard input. We describe a merge model that combines recognition results in an asynchronous and flexible manner. We collected speech and gesture data of users entering both short email sentences and web search queries. By merging recognition results from both modalities, word error rate was reduced by 53% relative for email sentences and 29% relative for web searches. For email utterances with speech errors, we investigated providing gesture keyboard corrections of only the erroneous words. Without the user explicitly indicating the incorrect words, our model was able to reduce the word error rate by 44% relative.

引用

页码：588 / +

页数：2

共 50 条

[41] Multimodal speech-gesture interface for handfree painting on a virtual paper using partial recurrent neural networks as gesture recognizer
Corradini, A
Cohen, PR
PROCEEDING OF THE 2002 INTERNATIONAL JOINT CONFERENCE ON NEURAL NETWORKS, VOLS 1-3, 2002, : 2293 - 2298
[42] Multimodal Speech Emotion Recognition using Cross Attention with Aligned Audio and Text
Lee, Yoonhyung
Yoon, Seunghyun
Jung, Kyomin
INTERSPEECH 2020, 2020, : 2717 - 2721
[43] Understanding Gesture and Speech Multimodal Interactions for Manipulation Tasks in Augmented Reality Using Unconstrained Elicitation
Williams A.S.
Ortega F.R.
Proceedings of the ACM on Human-Computer Interaction, 2020, 4 (ISS)
[44] Understanding Multimodal User Gesture and Speech Behavior for Object Manipulation in Augmented Reality Using Elicitation
Williams, Adam S.
Garcia, Jason
Ortega, Francisco
IEEE TRANSACTIONS ON VISUALIZATION AND COMPUTER GRAPHICS, 2020, 26 (12) : 3479 - 3489
[45] The impacts of limited visual feedback on mobile text entry for the twiddler and mini-QWERTY keyboards
Clawson, J
Lyons, K
Starner, T
Clarkson, E
NINTH IEEE INTERNATIONAL SYMPOSIUM ON WEARABLE COMPUTERS, PROCEEDINGS, 2005, : 170 - 177
[46] GesGPT: Speech Gesture Synthesis With Text Parsing From ChatGPT
Gao, Nan
Zhao, Zeyu
Zeng, Zhi
Zhang, Shuwu
Weng, Dongdong
Bao, Yihua
IEEE ROBOTICS AND AUTOMATION LETTERS, 2024, 9 (03) : 2718 - 2725
[47] Gesture objects detection and tracking for virtual text entry keyboard interface
Kuldeep Singh Yadav
Rabul Hussain Anish Monsley K.
Multimedia Tools and Applications, 2023, 82 : 5317 - 5342
[48] A new concept of assistive virtual keyboards based on a systematic review of text entry optimization techniques
Gomide R.S.
Loja L.F.B.
Lemos R.P.
Flôres E.L.
Melo F.R.
Teixeira R.A.G.
Research on Biomedical Engineering, 2016, 32 (02) : 176 - 198
[49] Exploring Word-gesture Text Entry Techniques in Virtual Reality
Chen, Sibo
Wang, Junce
Guerra, Santiago
Mittal, Neha
Prakkamakul, Soravis
CHI EA '19 EXTENDED ABSTRACTS: EXTENDED ABSTRACTS OF THE 2019 CHI CONFERENCE ON HUMAN FACTORS IN COMPUTING SYSTEMS, 2019,
[50] Gesture objects detection and tracking for virtual text entry keyboard interface
Yadav, Kuldeep Singh
Monsley, Anish K.
Laskar, Rabul Hussain
MULTIMEDIA TOOLS AND APPLICATIONS, 2023, 82 (04) : 5317 - 5342

← 1 2 3 4 5 →