Sample-Correlation-Aware Unsupervised Deep Anomaly Detection Model

The goal of anomaly detection is to identify abnormal patterns within normal patterns. Anomaly detection is applied in different forms in different application scenarios, such as network security, medical image and video monitoring. How to make full use of all kinds of characteristic information of data to identify the anomaly is one of the hot spots of anomaly detection. Many intelligent algorithms and models in data mining, machine learning and deep learning have been used to training anomaly detection rules to improve their detection performances. The training methods of anomaly detection rules are divided into three kinds: supervised, semi-supervised and unsupervised methods, and the third kind is the most popular. Formally, unsupervised deep anomaly detection can be viewed as density estimation from the data distribution. At present, deep anomaly detection models have achieved remarkable results in different application scenarios. But these methods are mainly based on the original features of data for training, and ignore the complex correlation among data samples: there are usually some kinds of correlation among normal samples that abnormal samples do not have. This makes these methods not ideal for anomaly analysis of complex data with characteristics of large data size, high dimension, unbalanced abnormal proportion, etc. In view of this, in this paper, a sample-correlation-aware deep learning model is proposed and used for anomaly detection, named sample-correlation-aware unsupervised deep anomaly detection model(SCA-UDLM): First of all, through in-depth analysis of the original sample features and the correlation features among samples, the model uses the K-nearest neighbors algorithm to search for similar samples to extract the correlation features among samples and store them in an undigraph structure, where nodes represent the samples and the edges represent the correlation between the two samples. Secondly, the original features and correlation features are fused based on the dual autoencoder composed of feature encoder and graph encoder, and generated high-quality data embedding in the low-dimensional feature space. Thirdly, the decoder decodes the low-dimensional embedding into reconstruction samples with original dimensional, and calculates the reconstruction error and features. Finally, an estimation network based on Gaussian mixture model is designed to estimate the probability density of samples based on the input of the reconstruction features and the low-dimensional embedding which are fused by the additive fusion method, and judge whether these samples are abnormal or not based on the given judgment threshold. A large number of experiments and analyses were made on this model and relevant representative machine learning methods and up-to-date deep learning methods, such as, one class support vector machines (OC-SVM), isolation forests(IF), deep structured energy based models(DSEBM), deep autoencoding GMM(DAGMM), Anomaly generative adversarial network(AnoGAN), adversarially learned anomaly detection(ALAD). The experimental results show that the detection performances of this model are improved by about 2% compared with other related methods, and its experimental results with different parameters, modules and noises are more stable than other methods, which prove the validity of the correlation module. The visualization experimental results can also highlight the advantages of this model in data feature extraction and full utilization. © 2021, Science Press. All right reserved.