Detecting malicious use with unlabelled data using clustering and outlier analysis

Most commercial intrusion detection systems (IDSs) presently available are signature-based network IDSs. Organisations Using these IDSs are still experiencing difficulties in detecting intrusive activity oil their networks since novel new attacks are consistently being encountered, and analysts call miss legitimate alarms when reviewing large alarm logs that contain a high number of false positives. There has been research investigating the Use Of data Mining techniques to effectively detect malicious activity in all enterprise network. The results of many of these projects have demonstrated that these techniques can be effective when trained/calibrated using labelled datasets. Labelled datasets identify and characterize normal and malicious traffic for Use In training/calibrating the detection sensor. However, the creation of labelled datasets is resource intensive. It requires a significant effort by security analysts to create a data set that characterises the traffic in a specific enterprise network environment. This research simulates and analyses malicious activity oil an enterprise network to explore the detection of malicious activity with data mining techniques Using unlabelled datasets. Semi-discrete decomposition (SDD) is Used as a clustering and outlier analysis technique to characterize network traffic as either normal or anomalous.