Appearance management and cue fusion for 3D model-based tracking

This paper presents a systematic approach to acquiring model appearance information online for monocular model-based tracking. The acquired information is used to drive a set of complementary imaging cues to obtain a highly discriminatory observation model. Appearance is modeled as a Markov random field of color distributions over the model surface. The online acquisition process estimates appearance-based on uncertain image measurements and is designed to greatly reduce the chance of mapping non-object image data onto the model. Confidences about the different appearance driven imaging cues are estimated in order to adaptively balance the contributions of the different cues. The discriminatory power of the resulting model is good enough to allow long-duration single-hypothesis model-based tracking with no prior appearance information. Careful evaluation based on real and semi-synthetic video sequences shows that the presented algorithm is able to robustly track a wide variety of targets under challenging conditions.