Kernel-based multiple cue algorithm for object segmentation

This paper proposes a novel algorithm to solve the problem of segmenting foreground-moving objects from the background scene. The major cue used for object segmentation is the motion information, which is initially extracted from MPEG motion vectors. Since the MPEG motion vectors are generated for simple video compression without any consideration of visual objects, they may not correspond to the true motion of the macroblocks. We propose a Kernel-based Multiple Cue (KMC) algorithm to deal with the above inconsistency of MPEG motion vectors and use multiple cues to segment moving objects. KMC detects and calibrates camera movements; and then finds the kernels of moving objects. The segmentation starts from these kernels, which are textured regions with credible motion vectors. Beside motion information, it also makes use of color and texture to help achieving a better segmentation. Moreover, KMC can keep track of the segmented objects over multiple frames, which is useful for object-based coding. Experimental results show that KMC combines temporal and spatial information in a graceful way, which enables it to segment and track the moving objects under different camera motions. Future work includes object segmentation in compressed domain, motion estimation from raw video, etc.