Duality-Based Locality-Aware Stream Partitioning in Distributed Stream Processing Engines

In this paper, we propose duality-based locality-aware stream partitioning (LSP) in distributed stream processing engines (DSPEs). In general, LSP directly uses the locality concept of distributed batch processing engines (DBPEs). This concept does not fully take into account the characteristics of DSPEs and therefore does not maximize cluster resource utilization. To solve this problem, we first explain the limitations of existing LSP, and we then propose a duality relationship between DBPEs and DSPEs. We finally propose a simple but efficient ping-based mechanism to maximize the locality of DSPEs based on the duality. The insights uncovered in this paper can maximize the throughput and minimize the latency in stream partitioning.