3D gesture based real-time object selection and recognition

Among other naturally performed gestures by users, pointing gesture is one of the most intuitive interfaces for selection. Many systems have been proposed on pointing gesture for HCl systems. The detection and selection of pointed object can be helpful in many applications. In this paper, we approximate the pointing location of the user by localizing the 3D position of upper human body skeletal joints and tracking the skeleton. This was achieved with real time constraints, using Microsoft Kinect sensor, which has the ability to estimate human joint location invariant to pose, clothing, body shape etc. The pointing direction is based on the line of sight connecting the shoulder joint to hand joint location. The developed system senses intentional arm pointing gesture. Pointing of the user in the earth ordinal direction is also approximated with respect to the user location in 3D space. The target pointed to is extracted, localized, zoomed and recognized. The recognition of object is performed by Hue-Saturation Histogram matching. The developed method allows selecting the object using both hands, either separately or simultaneously. (C) 2017 Elsevier B.V. All rights reserved.