Real-time Hand-object Occlusion for Augmented Reality Using Hand Segmentation and Depth Correction

Hand-object occlusion is crucial to enhance the realism of Augmented Reality, especially for egocentric hand-object interaction scenes. In this paper, a hand segmentation-based depth correction approach is proposed, which can help to realize real-time hand-object occlusion. We introduce a lightweight convolutional neural network to quickly obtain real hand segmentation mask. Based on the hand mask, different strategies are adopted to correct the depth data of hand and non-hand regions, which can implement hand-object occlusion and object-object occlusion simultaneously to deal with complex hand situations during interaction. The experimental results demonstrate the feasibility of our approach presenting visually appealing occlusion effects.