mXception and dynamic image for hand gesture recognition

Gesture detection has recently attracted a lot of attention due to its wide range of applications, notably in human–computer interaction (HCI). However, when it comes to video-based gesture recognition, elements in the background unrelated to gestures slow down the system’s classification rate. This paper presents an algorithm designed for the recognition of large-scale gestures. In the training phase, we utilize RGB-D videos, where the depth modality videos are derived from RGB modality videos using UNET and subsequently employed for testing. However, it’s worth noting that in real-time applications of the proposed dynamic hand gesture recognition (DHGR) system, only RGB modality videos are needed. The algorithm begins by creating two dynamic images: one from the estimated depth video and the other from the RGB video. Dynamic images generated from RGB video excel in capturing spatial information; while, those derived from depth video excel in encoding temporal aspects. These two dynamic images are merged to form an RGB-D dynamic image (RDDI). The RDDI is then fed into a modified Xception-based CNN model for the purpose of gesture classification and recognition. In order to evaluate the system’s performance, we conducted experiments using the EgoGesture and MSR Gesture datasets. The results are highly promising, with a reported classification accuracy of 91.64% for the EgoGesture dataset and an impressive 99.41% for the MSR Gesture dataset. The results demonstrated that the suggested system outperformed some existing techniques.