Monitoring Data Reduction in Data Centers: A Correlation-Based Approach

Monitoring data are collected and stored in a wide range of domains, especially in data centers, which integrate myriads of services and massive data. To handle the inevitable challenges brought by increasing volume of monitoring data, this paper proposes a correlation-based reduction method for streaming data that derives quantitative formulas between correlated indicators, and reduces the sampling rate of some indicators by replacing them with formulas predictions. This approach also revises formulas through iterations of the reduction process to find an adaptive solution in dynamic environments of data centers. One highlight of this work is the ability to work on upstream side, i.e., it can reduce volume requirements for data collection of monitoring systems. This work also tests the approach with both simulated and real data, showing that our approach is capable of data reduction in complex data centers.