Sentence Generation from IMU-based Human Whole-Body Motions in Daily Life Behaviors

This paper presents a probabilistic approach to-ward integrating human whole-body motions with natural language. Human whole-body motions in daily life are recorded by inertial measurement units (IMU) and subsequently encoded into motion symbols. Sentences are manually attached to the human motion primitives for their annotation. Two aspects of semantics and syntactics are represented by probabilistic graphical models. One probabilistic model trains the linking of motion symbols to words, and the other model represents sentence structure as word sequences. These two models are useful toward translating human whole-body motions into descriptions, where multiple words are associated from the human motions by the first model, and the second model searches for syntactically consistent sentences consisting of the associated words. The proposed approach was tested on a large dataset of human whole-body motions and sentences to annotate these motions. The linking of human motions to natural language enables robots to understand observations of human behavior as sentences.