Big-Data Analysis of Multi-Source Logs for Anomaly Detection on Network-based System

Log data are important audit basis to record routine events occurring on computer or network system, which are also critical data source for detecting system anomalies. By analyzing the data from multi-source logs, it is helpful to detect abnormal system behaviors and discover intruder attacks in real time. In this paper, a Spark-based log data security platform is designed and built to analyze the large-scale log data and detect abnormal network behaviors. By integrating data mining, machine learning, and statistical analysis technologies, our proposed framework can quickly analyze large-scale multi-source log data and accurately discriminate the abnormal behaviors. Furthermore, the association analysis is applied to detect abnormal behaviors or potential threats in the system. Under a real-world network environment, extensive experiments are conducted to evaluate the system performance, which can achieve a fast and accurate detection for abnormal network behaviors, and significantly improve the accuracies under various types of network attack scenarios.