Explicit quaternion krawtchouk moment invariants for finger-spelling sign language recognition

Sign recognition is a difficult task due to the complexity of its composition which uses signs of different levels, words, facial expression, body posture and finger-spelling to convey meaning. With the development of recent technologies, such as Kinect sensor, new opportunities have emerged in the field of human computer interaction and sign language, allowing to capture both RGB and Depth (RGB-D) information. In the regard to feature extraction, the traditional methods process the RGB and Depth images independently. In this paper, we propose a robust static finger-spelling sign language recognition system adopting the Quaternion algebra that provide a more robust and holistical representation, based on fusing RGB images and Depth information simultaneously. Indeed, we propose, for the first time, a new sets of Quaternion Krawtchouk moments(QKMs) and Explicit Quaternion Krawtchouk Moment Invariants (EQKMIs). The proposed system is evaluated on three well-known finger-spelling datasets, demonstrate the performance of the novel method compared to other methods used in the literature, against geometrical distortion, noisy conditions and complex background, indicating that it could be highly effective for many other computer vision applications.