Uncertainty Estimation for KLT Tracking

The Kanade-Lucas-Tomasi tracker (KLT) is commonly used for tracking feature points due to its excellent speed and reasonable accuracy. It is a standard algorithm in applications such as video stabilization, image mosaicing, egomotion estimation, structure from motion and Simultaneous Localization and Mapping (SLAM). However, our understanding of errors in the output of KLT tracking is incomplete. In this paper, we perform a theoretical error analysis of KLT tracking. We first focus our analysis on the standard KLT tracker and then extend it to the pyramidal KLT tracker and multiple frame tracking. We show that a simple local covariance estimate is insufficient for error analysis and a Gaussian Mixture Model is required to model the multiple local minima in KLT tracking. We perform Monte Carlo simulations to verify the accuracy of the uncertainty estimates.