From Macro to Micro Autonomous Multiscale Image Fusion for Robotic Surgery

An automated scanning framework for probe-based confocal laser endomicroscopy (pCLE) and optical coherence tomography (OCT) optical biopsy using the da Vinci surgical robot is reported. It is capable of generating large-area mosaics of both pCLE and OCT images. A crucial feature is that pCLE images are used to close the control loop, and mosaicing results from both static and deforming phantoms demonstrated that this effectively compensates kinematic errors. Furthermore, by using OCT images to maintain a constant distance to the tissue, and hence ensure consistent contact between the pCLE probe and the tissue, the system is able to compensate for target motion along the axial direction. This visual servoing allows for the correction of errors due to tissue deformation, robot positioning, and grasping of the pickup probe. The accuracy of this correction is better than the field of view (FoV) of the pCLE probe, resulting in continuous 2-D mosaics without gaps or discontinuities, which represent a common problem for open-loop control. Finally, the resulting high-resolution tissue maps at microscale can be fused in real time with a stereo reconstruction at macroscale from the laparoscopic image feed, providing the surgeon with a multiscale 3-D view of the operating site. This augmented visualization provides a range of potential benefits for intraoperative tissue characterization and surgical planning.