Fault Detection of ICT systems with Deep Learning Model for Missing Data

Fault detection is one of the most important tasks in information and communications technology (ICT) systems. Unsupervised anomaly detection methods, which are based on machine learning for fault detection in the ICT systems, use various kinds of data such as traffic data, memory usage data, CPU usage data, and text log data. The problem of deploying unsupervised anomaly detection methods in real ICT systems is that these data may have missing values. When a record has missing values, existing unsupervised anomaly detection ignores the records or imputes missing values with specific values. However, both operations lead to decreased performance of the anomaly detection methods. In this paper, we propose an unsupervised anomaly detection method that can handle records with missing values without imputation by using a neural network that can process variable length inputs. We experimented with 22 benchmark datasets to evaluate the performance of the proposed method for various kinds of data. The experimental results reveal that the proposed method performs better than existing methods in terms of area under the receiver operating characteristic (AUROC) on average for two cases in which 1) neither training nor test data include incomplete data, and 2) both training and test data include incomplete data. Moreover, we experimented with data from a Wi-Fi service that have missing values. The results show that the proposed method outperformed existing unsupervised anomaly detection methods.