The Non-Expert Tax: Quantifying the cost of auto-scaling in Cloud-based data stream analytics

All major Cloud providers offer Data Stream Analytics as managed services, allowing non-expert users to easily extract knowledge from data streams. These services lift the burden of job deployment and maintenance off users by provisioning resources automatically. However, this automation often comes at a cost, as auto-scaling policies may choose to over-provision to meet performance goals. We term this cost the Non-Expert Tax: the relative error between the ideal cost of deployment if an expert user would carefully configure resource allocation and the actual cost incurred by executing the same job with auto-scaling enabled. We conduct an empirical evaluation study of auto-scaling in two popular Cloud-based data stream analytics services and we find that they aggressively scale out, allocating resources quickly to meet high demand, but are conservative when scaling down, thus, charging users for underutilized resources. We quantify the Non-Expert Tax and show it can be as high as 544% for short-term jobs and up to 332% per month for periodic workloads.