Efficient Online Surveillance Video Processing Based on Spark Framework

In the current surveillance video processing systems, the video processing algorithms and the physical resources are highly coupled, and a video stream is usually used as a basic task scheduling unit. With the expansion of the scale of the system, the traditional systems will cause the large resource fragments that cannot be utilized adequately. In this paper, we propose a novel online surveillance video processing system architecture that combines the distributed Kafka message queue and Spark computing framework. Our system decouples the video stream collection and the video stream processing, and further decouples the video processing tasks and the physical resources. This loosely coupled architecture can quickly recover the failed tasks without data loss for the large-scale video surveillance, and can provide the more scalable distributed computing ability. In addition, a fine-grained online video task management method, which uses the cached video data blocks as the scheduling units, is proposed to increase the resource utilization. Experimental results show that our system has the higher resource utilization and the higher task capacity compared with the traditional systems.