Fingertips Detection in Egocentric Video Frames using Deep Neural Networks

In recent years, there has been much advancement in Augmented Reality technologies. Also, there has been a rise in the usage of wearable cameras. These technologies allow us to interact with the virtual world and the real world simultaneously. Hand gestures or finger gestures can be used to provide input instructions replacing conventional tools like a keyboard or a mouse. This paper introduces an improvement over the YOLSE (You Only Look what You Should See) model towards multiple fingertip position estimation. We propose a regression-based technique to locate fingertip(s) in a multi-gesture condition. First, the hand gesture is segmented from the scene using a deep neural network (DNN) based object detection model. Next, fingertip(s) positions are estimated using MobileNetv2 architecture. It is difficult to use direct regression when the varying number of visible fingertips are present in different egocentric hand gestures. We used the multi-label classification concept to identify all the visible extended fingers in the image. Average errors on RGB image with a resolution of 640 x 480 is 6.1527 pixels. The processing time of 9.072 ms is achieved on Nvidia GeForce GTX 1080 GPU.