Distributed System for Botnet Traffic Analysis and Anomaly Detection

As the ICT technologies evolve and new stacks are being proposed and created, the complexity of cyber security has dramatically increased, making the traditional signature-based approach ineffective. Many of nowadays existing solutions have never been deeply tested from the security point of view and thus being a target of cyber criminals. On the other hand, the Big Data technologies give the network administrators wide spectrum of tools to combat cyber threats. This paper presents one of such a systems for network traffic analysis and anomalies detection. The core of the system bases on the Big Data processing framework, data mining and machine learning techniques. So far, the proposed system implements two pattern extraction strategies leveraging batch processing methods. The presented experiments are focused on the problem of the botnet detection by means of data in form of NetFlows. The results analysis focus on performance evaluation of the proposed algorithms. In particular, different setups are considered in order to evaluate such aspects as detection effectiveness. The obtained results are promising and show that the proposed system can be considered as a useful tool for the network administrator.