Hybrid Tracking for Improved Registration of Laparoscopic Ultrasound and Laparoscopic Video for Augmented Reality

Laparoscopic augmented reality (AR), improves the surgeon's experience of using multimodal visual data during a procedure by fusion of medical image data (e.g., ultrasound images) onto live laparoscopic video. The majority of AR studies are based on either computer vision-based or hardware-based (e.g., optical and electromagnetic tracking) approaches. However, both approaches introduce registration errors because of variable operating conditions. To alleviate this problem, we propose a novel approach of hybrid tracking which comprises of both hardware-based and computer vision-based approaches. It consists of the registration of an ultrasound image with a time-matched video frame using electromagnetic tracking followed by a computer vision-based refinement of the registration and subsequent fusion. Experimental results demonstrate not only the feasibility of the proposed concept but also improved tracking accuracy that it provides and the potential for its integration into a future clinical AR system.