3D tracking using 2D-3D line segment correspondence and 2D point motion

In this paper, we propose a 3D tracking method which integrates two kinds of 2D feature tracking techniques. Our tracker searches 2D-3D correspondences used to estimate camera pose on the next frame from detected straight edges and projected 3D-CAD model on the current frame, and tracks corresponding edges on the consecutive frames. By tracking those edges, our tracker can keep correct correspondences even when large camera motion occurs. Furthermore, when the estimated pose seems incorrect, our tracker brings back to the correspondences of the previous frame and proceeds tracking of corresponding edges. Then, on the next frame, our tracker estimates the pose from those correspondences and can recover to the correct pose. Our tracker also detects and tracks corners on the image as 2D feature points, and estimates the camera pose from 2D-3D line segment correspondences and the motions of feature points on the consecutive frames. As the result, our tracker can suppress the influence of incorrect 2D-3D correspondences and can estimate the pose even when the number of detected correspondences is not enough. We also propose an approach which estimates both the camera pose and the correspondences. With this approach, our tracker can estimate the pose and the correspondence on the initial frame of the tracking. From experimental results, we confirmed our tracker can work in real-time with enough accuracy for various applications even with a less accurate CAD model and noisy low resolution images.