Alps: An Adaptive Load Partitioning Scaling Solution for Stream Processing System on Skewed Stream

The distributed stream processing system suffers from the rate variation and skewed distribution of input stream. The scaling policy is used to reduce the impact of rate variation, but cannot maintain high performance with a low overhead when input stream is skewed. To solve this issue, we propose Alps, an Adaptive Load Partitioning Scaling system. Alps exploits adaptive partitioning scaling algorithm based on the willingness function to determine whether to use a partitioning policy. To our knowledge, this is the first approach integrates scaling policy and partitioning policy in an adaptive manner. In addition, Alps achieves the outstanding performance of distributed stream processing system with the least overhead. Compared with state-of-the-art scaling approach DS2, Alps reduces the end-to-end latency by 2 orders of magnitude on high-speed skewed stream and avoids the waste of resources on low-speed or balanced stream.