Generating Body Motions using Spoken Language in Dialogue

被引:15
|
作者
Ishii, Ryo [1 ]
Katayama, Taichi [1 ]
Higashinaka, Ryuichiro [1 ]
Tomita, Junji [1 ]
机构
[1] NTT Corp, NTT Media Intelligence Labs, Yokosuka, Kanagawa, Japan
关键词
body motion; generation method; natural language; language; dialogue; virtual agent; HEAD MOVEMENT; TURN-TAKING; ANIMATION;
D O I
10.1145/3267851.3267866
中图分类号
TP18 [人工智能理论];
学科分类号
081104 ; 0812 ; 0835 ; 1405 ;
摘要
We propose a model to automatically generate whole body motions accompanying utterances at appropriate times, similar to humans, by using various types of natural-language-analysis information obtained from spoken language. Specifically, we focus on the co-occurrence relationship between various types of natural-language-analysis information such as words included in the spoken language, parts of speech, a thesaurus, word positions, dialogue acts of the spoken language, and human motions. Our model automatically generates nods, head postures, facial expressions, hand gestures, and upper-body posture using such information. We first recorded a two-person dialogue and constructed a multimodal corpus including utterance and whole body motion information. Next, using the constructed corpus, we constructed our model for generating a motion for each phrase unit using machine learning and using words, parts of speech, a thesaurus, word positions, and speech acts of the entire spoken language as inputs. These types of natural-language-analysis information were useful for motion generation. The effectiveness of our model was verified through a subjective experiment using a virtual conversational agent. As a result, the agent's body motions and impressions regarding naturalness of motion, degree of coincidence between utterance and motion, humanness of the agent, and likability of the agent improved with our model.
引用
收藏
页码:87 / 92
页数:6
相关论文
共 50 条
  • [1] Spoken language dialogue systems
    Giachin, E
    McGlashan, S
    CORPUS-BASED METHODS IN LANGUAGE AND SPEECH PROCESSING, 1997, 2 : 69 - 117
  • [2] Analysis of head motions and speech in spoken dialogue
    Ishi, Carlos T.
    Ishiguro, Hiroshi
    Hagita, Norihiro
    INTERSPEECH 2007: 8TH ANNUAL CONFERENCE OF THE INTERNATIONAL SPEECH COMMUNICATION ASSOCIATION, VOLS 1-4, 2007, : 1485 - 1488
  • [3] Using prosodic information to constrain language models for spoken dialogue
    Taylor, P
    Shimodaira, H
    Isard, S
    King, S
    Kowtko, J
    ICSLP 96 - FOURTH INTERNATIONAL CONFERENCE ON SPOKEN LANGUAGE PROCESSING, PROCEEDINGS, VOLS 1-4, 1996, : 216 - 219
  • [4] Generative Spoken Dialogue Language Modeling
    Nguyen, Tu Anh
    Kharitonov, Eugene
    Copet, Jade
    Adi, Yossi
    Hsu, Wei-Ning
    Elkahky, Ali
    Tomasello, Paden
    Algayres, Robin
    Sagot, Benoit
    Mohamed, Abdelrahman
    Dupoux, Emmanuel
    TRANSACTIONS OF THE ASSOCIATION FOR COMPUTATIONAL LINGUISTICS, 2023, 11 : 250 - 266
  • [5] The SENECA spoken language dialogue system
    Minker, W
    Haiber, U
    Heisterkamp, P
    Scheible, S
    SPEECH COMMUNICATION, 2004, 43 (1-2) : 89 - 102
  • [6] A conversation acts model for generating spoken dialogue contributions
    Stent, AJ
    COMPUTER SPEECH AND LANGUAGE, 2002, 16 (3-4): : 313 - 352
  • [7] Spoken language understanding method using confidence measure and dialogue history
    Fujiwara, Noriki
    Itoh, Toshihiko
    Araki, Kenji
    Kai, Atsuhiko
    Konishi, Tatsuhiro
    Itoh, Yukihiro
    Systems and Computers in Japan, 2007, 38 (09): : 21 - 31
  • [8] Evaluation of spoken language understanding and dialogue systems
    Hildebrandt, B
    Rautenstrauch, H
    Sagerer, G
    ICSLP 96 - FOURTH INTERNATIONAL CONFERENCE ON SPOKEN LANGUAGE PROCESSING, PROCEEDINGS, VOLS 1-4, 1996, : 685 - 688
  • [9] Robust numeric recognition in spoken language dialogue
    Rahim, M
    Riccardi, G
    Saul, L
    Wright, J
    Buntschuh, B
    Gorin, A
    SPEECH COMMUNICATION, 2001, 34 (1-2) : 195 - 212
  • [10] Adaptive language models for spoken dialogue systems
    Solsona, RA
    Fosler-Lussier, E
    Kuo, HKJ
    Potamianos, A
    Zitouni, I
    2002 IEEE INTERNATIONAL CONFERENCE ON ACOUSTICS, SPEECH, AND SIGNAL PROCESSING, VOLS I-IV, PROCEEDINGS, 2002, : 37 - 40