Data Parallelism for Distributed Streaming Applications

Streaming applications can analyze vast data streams and requires both high throughput and low latency. They are comprised of operator graphs which produce and consume data tuples where operators are stateful, selective and user-defined. The streaming programming model logically exposes task and pipeline parallelism, enabling it to develop parallel systems. Naturally it doesnot expose data parallelism, which must be extracted from streaming applications. This paper presents a compiler and runtime system that automatically extract data parallelism for distributed stream processing. Our approach is safety guarantee in presence of stateful, selective and user-defined operators. Data parallelization is secure if the sequential semantics of the applications are preserved, also the compiler ensures safety by considering dependencies on other operators in the graph and selectivity, state, partitioning of operator. The distributed runtime system ensures that tuples always exit parallel regions in the same order they would without data parallelism, using the most efficient strategy as identified by the compiler.