Generating Behavior-based Malware Detection Models with Genetic Programming

Malware remains a major IT security threat and current detection approaches struggle to cope with a professionalized mal ware development industry. We propose the use of genetic programming to generate effective and robust malware detection models which we call FrankenMods. These are sets of graph metrics that capture characteristic mal ware behavior. Evolution of FrankenMods with good detection capabilities yields continuously improved detection effectiveness. FrankenMods are operationalized by evaluating them on quantitative data flow graphs that model mal ware behavior as data flows between system resources caused by issued system calls. We show that FrankenMods are substantially more robust and effective than a state-of-the-art graph metric-based detection approach.