Real-time detection of hidden traffic patterns

We address the problem of fast automatic identification of traffic patterns in core networks with high speed links carrying large numbers of flows. This problem has applications in detecting DoS attacks, traffic management, and network security. The typical measurement and identification objective is to determine flows that use up a disproportionate fraction of network resources. Several schemes have been devised to measure large flows efficiently assuming that the notion of what constitutes a flow is well defined a priori. However there are many scenarios where traffic patterns are hidden in the sense that there is no clear knowledge of what exactly to look for and there is no natural a priori definition of flow. In this paper, we develop an effective scheme to identify and measure hidden traffic patterns. The approach is flexible enough to automatically identify interesting traffic patterns for further evaluation. The basic idea is to extend the runs based approach proposed in [1] to the case where flow definitions are not known a priori. A straightforward extension is both memory and processing intensive. We develop an efficient scheme that has good theoretical properties and does extremely well in practice.