Object based segmentation of video using color, motion and spatial information

Video segmentation is different from segmentation of a single image. While several correct solutions may exist for segmenting a single image, there needs to be a consistency among segmentations of each frame for video segmentation. Previous approaches of video segmentation concentrate on motion, or combine motion and color information in a batch fashion. We propose a maximum a posteriori probability (MAP) framework that uses multiple cues, like spatial location, color and motion, for segmentation. We assign weights to color and motion terms, which are adjusted at every pixel, based on a confidence measure of each feature. We also discuss the appropriate modeling of pdfs of each feature of a region. The correct modeling of the spatial pdf imposes temporal consistency among segments in consecutive frames. This approach unifies the strengths of both color segmentation and motion segmentation in one framework, and shows good results on videos that are not suited for either of these approaches.