In-Air Handwriting for Chinese Character Recognition from Monocular Camera: A Deep Learning based Approach with Fingertip Detection and Virtual Strokes Elimination

This study presents a deep learning-based approach for recognizing Chinese characters written in the air. Chinese characters, which are composed of numerous strokes in a square shape, pose a challenge for recognizing real and virtual strokes when writing in the air. Existing Optical Character Recognition (OCR) models often result in inaccurate recognition due to this difficulty. Depth cameras can improve recognition accuracy, but they are more expensive than traditional optical cameras. To address these challenges, this study employs a deep learning model to track fingertips from a single optical camera. The approach incorporates an intuitive interface that distinguishes between real and virtual strokes while writing. Gesture recognition serves as commands to enable users to start and stop writing and to make corrections during input. This approach makes in-air handwriting behavior similar to writing on paper. To evaluate the effectiveness of the proposed approach, a comparative experiment was conducted to evaluate the model's performance for recognizing characters with and without virtual strokes. Eliminating virtual strokes enabled the OCR model to achieve over 90% recognition accuracy, which was 37% higher than the accuracy with virtual strokes. Moreover, the recognition accuracy of characters with virtual strokes fluctuated significantly in a stability test conducted one week later, while the input method without virtual strokes remained stable at over 90% recognition accuracy. These experimental results indicate that the proposed method can not only effectively remove virtual strokes in air writing but also make in-air handwriting easier for people.