A neural-network based web application on real-time recognition of Pakistani sign language

Communication between deaf and non-deaf individuals is affected by a language barrier, as the hearing-impaired rely on sign language to communicate. This two-way process requires fluency in language to be effective. In Pakistan, there are approximately 10 million hearing-impaired citizens at present, and a lack of assistive technology hinders this process. This led us to the objective of our study: to develop a web application that allows real-time recognition of Pakistani Sign Language (PSL) letters. The application backend was developed using Python, and the frontend was developed with Cascading Style Sheets (CSS) and Bootstrap libraries. The dataset consisted of static and dynamic gestures of PSL and the study employed deep learning techniques using the Mediapipe library to model two recognition systems. The offline testing accuracy was 2% and 4% higher than previous significant works on static and dynamic PSL recognition respectively. Real-time testing of both models achieved accurate prediction of all 39 letters.