BotMark: Automated botnet detection with hybrid analysis of flow-based and graph-based traffic behaviors

The Botnets have become one of the most serious threats to cyber infrastructure. Most existing work on detecting botnets is based on flow-based traffic analysis by mining their communication patterns. There also exists related work based on anomaly detection in communication graphs. As bots have continuously evolved and become increasingly sophisticated, only using flow-based traffic analysis or graph-based analysis for the detection would result in false negatives or false positives, or can even be evaded. In this work, we propose BotMark, an automated model that detects botnets with hybrid analysis of flow-based and graph-based network traffic behaviors. We extract 15 statistical flow-based traffic features as well as 3 graph-based features in building the detection model. For flow-based detection, we consider the similarity and stability of C-flow as measurements in the detection. In particular, we employ k-means to measure the similarity of C-flows and assign similarity scores, and calculate stability score of C-flows through the distribution of packet length within a C-flow. The graph-based detection is based on the observation that the neighborhoods of anomalous nodes significantly differ from those of normal nodes in communication graphs. In particular, we use least-square technique and Local Outlier Factor (LOF) to calculate anomaly scores that measure the differences of their neighborhoods. Our models use the scores to mark bots. BotMark performs automated botnet detection with hybrid analysis of flow-based and graph-based traffic behaviors by ensemble of the detection results based on similarity scores, stability scores and anomaly scores. We collect a very large size of network traffic by simulating 5 newly propagated botnets, including Mirai, Black energy, Zeus, Athena and Ares in a real computing environment. Extensive experimental results demonstrate the effectiveness of BotMark. It achieves 99.94% in terms of detection accuracy, outperforming any individual detector with flow-based detection or graph-based detection. (C) 2019 Elsevier Inc. All rights reserved.