Integrated segmentation and depth ordering of motion layers in image sequences

We describe a method to segment and depth order motion layers simultaneously in an image sequence. Previous approaches to motion segmentation and layering have tended to ignore the depth ordering issue or treat it as a post-processing operation. We argue here that motion estimation and segmentation are crucially dependent on depth order and hence that the latter should form an integral part of any layering scheme. Using an explicit model of boundary ownership allowing simultaneous assignment of motions to regions and extraction of depth order, the method fuses colour region segmentations with motion estimates obtained via block correlation. A novel depth-dependent partial correlation technique then provides improved motion estimates in the vicinity of motion boundaries. The segmentation and layering are implemented within a region-adjacency graph framework-creating moving object graphs-which is both flexible and computationally tractable. Results of experiments on real sequences show the approach to be effective. (C) 2002 Elsevier Science B.V. All rights reserved.