Anomaly States Monitoring of Large-Scale Systems with Intellectual Analysis of System Logs

The article analyzes the paths and algorithms for automating the monitoring of computer system states by means of intellectual analysis of unstructured system log data in order to detect and diagnose abnormal states. This information is necessary for technical support to locate the problem and diagnose it accurately. Because of the ever-growing log size, mining data mining models are used to help developers extract system information. At the first stage, logs are collected with records of system states and information on the execution of processes. At the second stage, the log parser is used to retrieve a group of event templates, with the result that the raw logs are structured. At the third stage, after the logs are parsed into separate patterns, they are additionally represented as numerical vectors of attributes (attributes). The set of all vectors forms a matrix of signs. In the fourth stage, the feature matrix is used to detect anomalies of machine learning methods to determine whether the new incoming log sequence is abnormal or not. A decision tree was used as a classification method for machine learning. Using the example of a distributed HDFS data set, the effectiveness of the considered method for detecting anomalous system states is shown.