Metamorphic Malware and Obfuscation: A Survey of Techniques, Variants, and Generation Kits

The competing landscape between malware authors and security analysts is an ever-changing battlefield over who can innovate over the other. While security analysts are constantly updating their signatures of known malware, malware variants are changing their signature each time they infect a new host, leading to an endless game of cat and mouse. This survey looks at providing a thorough review of obfuscation and metamorphic techniques commonly used by malware authors. The main topics covered in this work are (1) to provide an overview of string-scanning techniques used by antivirus vendors and to explore the impact malware has had from a security and monetary perspective; (2) to provide an overview of the methods of obfuscation during disassembly, as well as methods of concealment using a combination of encryption and compression; (3) to provide a comprehensive list of the datasets we have available to us in malware research, including tools to obfuscate malware samples, and to finally (4) discuss the various ways Windows APIs are categorized and vectorized to identify malicious binaries, especially in the context of identifying obfuscated malware variants. This survey provides security practitioners a better understanding of the nature and makeup of the obfuscation employed by malware. It also provides a review of what are the main barriers to reverse-engineering malware for the purposes of uncovering their complexity and purpose. © 2023 Kenneth Brezinski and Ken Ferens.