Host-based intrusion detection by monitoring windows registry accesses

In this study, we propose a host-based intrusion detection system for Microsoft Windows. The proposed system detects attacks on a host machine by monitoring anomalous accesses to the Windows Registry. First a model of normal registry behavior is trained for a host and then this model is used to detect abnormal registry accesses. The system trains a normal model using data that contains no attacks and then checks each access to the registry to determine whether or not the behavior is abnormal and corresponds to an attack. Here a new approach to the Register Anomaly Detection (RAD) is proposed in the meaning of model generator and anomaly detector. Self Organizing Map (SOM), a type of artificial neural network model, is used as an anomaly detection algorithm. The system is trained on a set of normal registry accesses by using SOM algorithm and then used to detect the behaviors of malicious software. The results of this study shows that the proposed system is effective in detecting the behaviors of malicious software and has a low rate of false alarms compared to other host-based intrusion detection systems.