Model-driven scheduling for distributed stream processing systems

Distributed Stream Processing Systems (DSPS) are "Fast Data" platforms that allow streaming applications to be composed and executed with low latency on commodity clusters and Clouds. Such applications are composed as a Directed Acyclic Graph (DAG) of tasks, with data parallel execution using concurrent task threads on distributed resource slots. Scheduling such DAGs for DSPS has two parts-allocation of threads and resources for a DAG, and mapping threads to resources. Existing schedulers often address just one of these, make the assumption that performance linearly scales, or use ad hoc empirical tuning at runtime. Instead, we propose model-driven techniques for both mapping and allocation that rely on low-overhead a priori performance modeling of tasks. Our scheduling algorithms are able to offer predictable and low resource needs that is suitable for elastic pay-as-you-go Cloud resources, support a high input rate through high VM utilization, and can be combined with other mapping approaches as well. These are validated for micro and application benchmarks, and compared with contemporary schedulers, for the Apache Storm DSPS. (C) 2018 Elsevier Inc. All rights reserved.