Semantic Background Estimation in Video Sequences

We present a method that estimates the scene background in videos by utilizing semantic segmentation to extract foreground objects, such as people or cars, and stitching background regions to reconstruct the background. Inspired by recent developments in deep learning, we utilize semantic segmentation based on Conditional Random Field as Recurrent Neural Networks (CRF as RNN) to detect the regions of important objects in each frame and generate a foreground-background map. We use these segmentation maps to extract the background regions from each frame and then stitch them over consecutive frames to obtain the full background for the video sequence. Our foreground/background estimation approach has potential applications in change detection, video surveillance, video compression and video privacy. We illustrate the effectiveness of our method on example videos from the Change Detection (CDNET) dataset.