Anomaly Detection Using Agglomerative Hierarchical Clustering Algorithm

Intrusion detection is becoming a hot topic of research for the information security people. There are mainly two classes of intrusion detection techniques namely anomaly detection techniques and signature recognition techniques. Anomaly detection techniques are gaining popularity among the researchers and new techniques and algorithms are developing every day. However, no techniques have been found to be absolutely perfect. Clustering is an important data mining techniques used to find patterns and data distribution in the datasets. It is primarily used to identify the dense and sparse regions in the datasets. The sparse regions were often considered as outliers. There are several clustering algorithms developed till today namely K-means, K-medoids, CLARA, CLARANS, DBSCAN, ROCK, BIRCH, CACTUS etc. Clustering techniques have been successfully used for the detection of anomaly in the datasets. The techniques were found to be useful in the design of a couple of anomaly based Intrusion Detection Systems (IDS). But most of the clustering techniques used for these purpose have taken partitioning approach. In this article, we propose a different clustering algorithm for the anomaly detection on network datasets. Our algorithm is an agglomerative hierarchical clustering algorithm which discovers outliers on the hybrid dataset with numeric and categorical attributes. For this purpose, we define a suitable similarity measure on both numeric and categorical attributes available on any network datasets.