Virtual reality interaction based on visual attention and kinesthetic information

Emerging multimedia technologies significantly enhance the naturalness and immersion of human computer interaction. Currently, research on kinesthetic information has gained increasing attentions of multimedia community. However, effective interaction between kinesthetic and other multimedia signals remains a challenging task. In this paper, we propose a visual-kinesthetic interaction in Virtual Reality (VR) and real-world control tasks. First, we model the correlation between user’s visual attention and kinesthetic positions under different tasks. Second, we utilize an attention-based Long Short-Term Memory network to predict the kinesthetic positions. Third, we build a VR system with robotic car control, which validates our model in VR interaction and control tasks. With a high task achievement rate, we envision the implementation of kinesthetic information in a more natural interaction system. The VR interaction system based on the proposed model can also provide guidance for the design of immersive robot teleoperation systems.