Geometry-Incorporated Posing of a Full-Body Avatar From Sparse Trackers

Accurately rendering a user's full body in a virtual environment is crucial for embodied mixed reality (MR) experiences. Conventional MR systems provide sparse trackers such as a headset and two hand-held controllers. Recent studies have intensively investigated learning methods to regress untracked joints from sparse trackers and have produced plausible poses in real time for MR applications. However, most studies have assumed that they either know the position of the root joint or constrain it, yielding stiff pelvis motions. This paper presents the first geometry-incorporated learning method to generate the position and rotation of all joints, including the root joint, from the head and hands information for a wide range of motions. We split the problem into identifying a reference frame and a pose inference with respect to the new reference frame. Our method defines an avatar frame by setting a non-joint as the origin and transforms joint data in a world coordinate system into the avatar coordinate system. Our learning builds on a propagating long short-term memory (LSTM) network exploiting prior knowledge of the kinematic chains and the previous time domain. The learned joints are transformed back to obtain the positions with respect to the world frame. In our experiments, our method achieves competitive accuracy and robustness with the state-of-the-art speed of approximately 130 fps on motion capture datasets and the wild tracking data obtained from commercial MR devices. Our experiments confirm that the proposed method is practically applicable to MR systems.