ASL Hand Gesture Classification and Localization Using Deep Ensemble Neural Network

The growing number of deaf and hearing-impaired populations and the evolution of vision-based application devices present enormous opportunities for research and modelling in the domain of gesture classification and recognition. Since hand gesture recognition is crucial in sign language interpretation, an efficient gesture recognition system should consider maximum sign characters for recognition. In this work, a robust hand gesture recognition model is proposed by employing deep ensemble neural networks. Initially, a pre-trained network is designed by adopting the VGG16 architecture with a self-attention layer embedded with the VGG16 architecture. This self-attention module enables to learn the potentially distinguishing image features for better differentiability among gesture categories. After that, a weighted ensemble model is introduced, which uses the complementary information contributed by the base model to magnify the overall network performance. A detailed experimental investigation of the individual pre-trained models and the weighted ensemble of these models is conducted to validate the efficacy of the proposed network. Moreover, this work shows that the proposed model can also be employed for weakly supervised object segmentation. Additionally, the performance of the suggested methodology is evaluated on publicly available two complex hand gesture datasets and obtains an accuracy of 99.76% and 95.10%, which relatively outperforms other state-of-the-art approaches.