Hand Gesture Recognition with Convolutional Neural Networks for the Multimodal UAV Control

We introduce a robust wearable sensor suite fusing arm motion and hand gesture recognition for operator control of UAVs. The sensor suite fuses mechanomyography (MMG) and an inertial measurement unit (IMU) to capture multimodal (arm movement and hand gesture) command signals simultaneously. The IMU produces world referenced orientation and acceleration data while concomitant MMG tracks muscle activation through surface vibration. The use of surface muscle vibration for gesture recognition removes the need for electrical contact with the skin, which has impeded other forms of muscle measurement for gesture recognition in the field. This investigation presents hardware design, inertial recognition of arm movement, and the detailed structure of a convolutional neural network (CNN) system used for real-time hand gesture recognition based on MMG signals. The system achieved 94% accuracy for five gestures with simple calibration for each user, thereby providing an intuitive gesture based UAV control system. To our knowledge this is the first wearable system enabling multimodal control of UAVs through intuitive gestures that does not require electrical skin contact. Future work involves testing the system with larger UAV swarms.