Flux: Decoupled Auto-Scaling for Heterogeneous Query Workload in Alibaba AnalyticDB

Modern cloud data warehouses are integral to processing heterogeneous query workloads, which range from quick online transactions to intensive ad-hoc queries and extract, transform, load (ETL) processes. The synchronization of heterogeneous workloads, particularly the blend of short and long-running queries, often degrades performance due to intricate concurrency controls and cooperative multi-tasking execution models. Additionally, the auto-scaling mechanisms for mixed workloads can lead to spikes in demand and underutilized resources, impacting both performance and cost-efficiency. This paper introduces the Flux, a cloud-native workload auto-scaling platform designed for Alibaba AnalyticDB, which implements a pioneering decoupled auto-scaling architecture. By separating the scaling mechanisms for short and long-running queries, Flux not only resolves performance bottlenecks but also harnesses the elasticity of serverless container instances for ondemand resource provisioning. Our extensive evaluations demonstrate Flux's superiority over traditional scaling methods, with up to a 75% reduction in query response time (RT), a 19.0% increase in resource utilization ratio, and a 77.8% decrease in cost overhead.