Visualizing Syscalls using Self-organizing Maps for System Intrusion Detection

Monitoring syscall logs provides a detailed view on almost all processes running on a system. Existing approaches therefore analyze sequences of executed syscall types for system behavior modeling and anomaly detection in cyber security. However, failures and attacks that do not manifest themselves as type sequences violations remain undetected. In this paper we therefore propose to incorporate syscall parameter values with the objective of enriching analysis and detection with execution context information. Our approach thereby first selects and encodes syscall log parameters and then visualizes the resulting high-dimensional data using self-organizing maps to enable complex analysis. We thereby display syscall occurrence frequencies and transitions of consecutively executed syscalls. We employ a sliding window approach to detect changes of the system behavior as anomalies in the SOM mappings. In addition, we use SOMs to cluster aggregated syscall data for classification of normal and anomalous system behavior states. Finally, we validate our approach on a real syscall data set collected from an Apache web server. Our experiments show that all injected attacks are represented as changes in the SOMs, thus enabling visual or semi-automatic anomaly detection.