Predictive Shoulder Kinematics of Rehabilitation Exercises Through Immersive Virtual Reality

Objective: The adoption of telehealth has rapidly accelerated owing to the global COVID19 pandemic disrupting communities and in-person healthcare practices. While telehealth had initial benefits in enhancing accessibility for remote treatment, physical rehabilitation has been heavily limited owing to the loss of hands-on evaluation tools. This paper presents an immersive virtual reality (iVR) pipeline for replicating physical therapy success metrics through applied machine learning of patient observation. Methods: We demonstrate a method of training gradient boosted decision-trees for kinematic estimation to replicate mobility and strength metrics using an off-the-shelf iVR system. During the two-month study, training data were collected while a group of users completed physical rehabilitation exercises in an iVR game. Utilizing this data, we trained on iVR-based motion capture data and OpenSim biomechanical simulations. Results: Our final model indicates that upper-extremity kinematics from OpenSim can be accurately predicted using the HTC Vive head-mounted display system with a Mean Absolute Error less than 0.78 degrees for joint angles and less than 2.34 Nm for joint torques. Additionally, these predictions are viable for runtime estimation, with approximately a 0.74 ms rate of prediction during exercise sessions. Conclusion: These findings suggest that iVR paired with machine learning can serve as an effective medium for collecting evidence-based patient success metrics for telehealth. Significance: Our approach can help increase the accessibility of physical rehabilitation with off-the-shelf iVR head-mounted display systems by providing therapists with the metrics needed for remote evaluation.