GRAPHICAL USER INTERFACE FOR HUMAN INTERVENTION IN 2D-3D REGISTRATION OF MEDICAL IMAGES

Image-guided medical therapies and image-guided biomechanical measurement systems often combine 2D and 3D imaging modalities. Determination of relations between the 2D and 3D imaging data is known as 2D-3D registration. Motivated by an ongoing project aimed at non-invasive, marker-free measurement of the kinematics of the bones in the foot during gait, we consider a registration approach that involves (1) computing projections of the 3D data set, (2) computing a quality measure to describe the agreement/discrepancy between the simulated projections and actual 2D images, and (3) optimization of the quality measure relative to the kinematic degrees of freedom to determine the optimal registration. For our particular project, the 3D imaging modality is CT scan, the 2D modality is bi-plane fluoroscopy, the computed projection is a digitally reconstructed radiograph (DRR), the quality measure is normalized cross-correlation (NCC) between a pair of DRRs and a pair of corresponding fluoroscope images, and the 2D imaging includes a sequence of several hundred stereo image pairs. We have recently released a software toolkit, DRRACC, that accelerates both the DRR and NCC computations via GPU-based parallel processing to enable more efficient automated determination of kinematic relations for optimal registration. While fully automated 2D-3D registration is desirable, there are situations (such as creating a reasonable starting configuration for optimization, re-starting after the optimizer fails to converge, and visual verification of registration relations) when it is desirable/necessary to have a human in the loop. In this paper, we present an OpenGL-based graphical user interface that employs the DRRACC toolkit to allow the user to manipulate the kinematics of individual objects (bones) segmented from the 3D imaging and to view the corresponding DRR and the associated correlation with a reference image in real time. We also present plots showing initial results for the dependence of the registration measure on pairs of kinematic parameters. The plots show well-defined peaks that support the hope for automated registration, but they also contain large relatively flat regions that may prove problematic for gradient-based optimizers and necessitate the sort of interface presented in this paper.