Hand Pose Estimation for Robot Programming by Demonstration in Object Manipulation Tasks

Observing human demonstration is an efficient way to transfer skills to robots and increasing their capabilities. In this paper, a vision-based markerless Progamming by Demonstration (PbD) system is developed for object manipulation tasks of robots. The system entails the use of KCF algorithm for tracking human hand movements and PSO algorithm for reliably estimating hand posture, using a single noisy and low-cost RGB-D sensor. In particular, 27 DoF hand pose including palm and finger joints is recovered for providing more precise information of how an specific object is grasped. Moreover, hand trajectories and postures in the manipulation demonstration are converted from human workspace to robot workspace using pre-calibrated coordinate relations, and thus can directed guide a robot end-effector. Implementation is conducted to cope with two typical object manipulation tasks, namely Handover and Tool-use. Experimental results validate the accuracy and stability of the proposed system.