Anomaly Detection of Software System Logs based on Natural Language Processing

System logs record the daily status of operating systems, application software, firewalls, etc. Analyzing system logs can help to prevent and eliminate information security events in real time. In this paper, we propose to analyze the system logs for anomalous event detection based on natural language processing. First, we use doc2vec of natural language processing algorithm to construct sentence vectors, then apply several state-of-the-art classification algorithms on the sentence vectors for anomaly detection. The system logs generated by the Thunderbird supercomputer are adopted here to verify the proposed method. The results show that doc2vec combined with machine learning classification algorithms could not only effectively extract the semantic information of the logs, but also perform excellent anomaly detection.