Prediction of Metacarpophalangeal Joint Angles and Classification of Hand Configurations Based on Ultrasound Imaging of the Forearm

With the advancement in computing and robotics, it is necessary to develop fluent and intuitive methods for interacting with digital systems, augmented/virtual reality (AR/VR) interfaces, and physical robotic systems. Hand movement recognition is widely used to enable such interaction. Hand configuration classification and metacarpophalangeal (MCP) joint angle detection are important for a comprehensive reconstruction of hand motion. Surface electromyography (sEMG) and other technologies have been used for the detection of hand motions. Ultrasound images of the forearm offer a way to visualize the internal physiology of the hand from a musculoskeletal perspective. Recent works have shown that these images can be classified using machine learning to predict various hand configurations. In this paper, we propose a Convolutional Neural Network (CNN) based deep learning pipeline for predicting the MCP joint angles. We supplement our results by using a Support Vector Classifier (SVC) to classify the ultrasound information into several predefined hand configurations based on activities of daily living (ADL). Ultrasound data from the forearm were obtained from six subjects who were instructed to move their hands according to predefined hand configurations relevant to ADLs. Motion capture data was acquired as the ground truth for hand movements at three speeds (0.5 Hz, 1 Hz, and 2 Hz) for the index, middle, ring, and pinky fingers. We demonstrated the perfect prediction of hand configurations through SVC classification and a correspondence between the predicted MCP joint angles and the actual MCP joint angles for the fingers, with an average root mean square error of 7.35 degrees. A low latency (6.25 - 9.10 Hz) pipeline was implemented for the prediction of both MCP joint angles and hand configuration estimation aimed for real-time implementation.