Deep Reinforcement Learning for EMG-based Control of Assistance Level in Upper-limb Exoskeletons

In this paper, we propose a deep reinforcement learning (DRL) method to control the assistance level of an upper-limb exoskeleton in real-time based on the electromyographic (EMG) activity of human muscles in 3D point-to-point reaching movements. The proposed autonomous assistive device would enhance the force exertion capability of individuals by resolving major challenges such as identifying scaling factors for personalized amplification of their effort and not requiring lengthy offline training/adjustment periods to perform their manual tasks comfortably. To this end, we employed the Twin Delayed Deep Deterministic Policy Gradient (TD3) method for rapid learning of the appropriate controller's gain values and delivering personalized assistive torques by the exoskeleton to different joints to assist the wearer in a weight handling task. A nonlinear reward function is defined in terms of the EMG activity level and the position deviation from the destination point to simultaneously minimize the muscle effort and maximize the positioning accuracy. This facilitates autonomous and individualized physical assistance by rapid exploration of reward values and adopting various action gains within a safe range to exploit the ones that maximize the reward. Based on experimental studies on an exoskeleton with soft actuators that we have developed, the proposed DRL method is able to identify the most appropriate assistive gain for each joint of the exoskeleton in real-time for the user with a fast rate of convergence (during the first two minutes). Optimum assistive gains are identified for each degree of freedom (DOF) in a 4 kg weight handling task in 3D space, which required less than 15% of the muscle contraction level (EMG activity).