Understanding Increasing Traffic Levels for Internet Abuse Detection

Internet misuse is a serious concern which undermines the burgeoning e-business environment. Several mature technologies exist for combating Internet misuse at the edges of the Internet, but certain classes of misuse can only be effectively prevented within the network core. Unfortunately, as communication data rates increase, it becomes increasingly difficult to process all the network traffic using conventional approaches. This paper discusses mechanisms to overcome this problem based on processing summaries of network traffic, rather than individual packets. An experimental implementation of the approach is described, and its integration with advanced processing tools to identify abuse situations is discussed. In the approach, packets are captured from the network and summaries of specific characteristics of the headers are generated. These are then analysed using data mining tools in order to identify abuse or unusual characteristics. The results of the analysis phase can then be used to refine the summarization process both to improve efficiency and to provide a means of responding to new threats.