On-line predictive load shedding for network monitoring

Building robust network monitoring applications is hard given the unpredictable nature of network traffic. Complex analysis on streaming network data usually leads to overload situations when presented with anomalous traffic, extreme traffic mixes or highly variable rates. We present an on-line predictive load shedding scheme for monitoring systems that quickly reacts to overload situations by gracefully degrading the accuracy of analysis methods. The main novelty of our approach is that it does not require any knowledge of the monitoring applications. This way we preserve a high degree of flexibility, increasing the potential uses of these systems. We implemented our scheme in an existing network monitoring system and deployed it in a research ISP network. Our experiments show a 10-fold improvement in the accuracy of the results during long-lived executions with several concurrent monitoring applications. The system efficiently handles extreme load situations, while being always responsive and without undesired packet losses.