Auto-scaling for real-time stream analytics on HPC cloud

There are very-high-volume streaming data in the cyber world today. With the popularization of 5G technology, the streaming Big Data grows larger. Moreover, it needs to be analyzed in real time. We propose a new strategy HPC2-ARS to enable streaming services on HPC platforms. This strategy includes a three-tier high-performance cloud computing (HPC2) platform and a novel autonomous resource-scheduling (ARS) framework. The HPC2 platform is our de facto base platform for research on streaming service. It has three components: Tianhe-2 high-performance computer, custom OpenStack cloud computing software, and Apache Storm stream data analytic system. Our ARS framework ensures real-time response on unpredictable and fluctuating stream, especially streaming Big Data in the 5G era. This strategy addresses an essential problem in the convergence of HPC Cloud, Big Data, and streaming service. Specifically, Our ARS framework provides theoretical and practical solutions for resource provisioning, placement, and scheduling optimization. Extensive experiments have validated the effectiveness of the proposed strategy.