The Role of Expert Systems in Reverse Code Engineering Tasks

Reverse Code Engineering (RCE) as a Computer Science discipline has been stifled for a number of reasons. Legal issues, liability and general lack of the prerequisite knowledge of necessary subjects have all contributed to holding the science back. As a result, RCE tools and techniques have not matured at the same rate as their counterparts in other areas. Many of the most popular and widely used RCE tools available are freeware or shareware programs, most without any developer support. Additionally, advanced RCE techniques are rarely taught in classrooms, instead a small collection of books and internet forums provide the main sources for advanced education. For these reasons, RCE skills exist as a craft discipline rather than a science. Expert systems (ES) are artificial intelligence systems with application in areas such as computer-based instruction and task automation. ES offer to integrate the expert knowledge of a person with a software system. This approach has the ability to provide solutions to interesting problems where an experts' knowledge is critical for success. The purpose of this paper is to prepare a framework for application of ES in RCE tasks. First, the fundamentals and challenges of RCE are introduced. Next, an enumeration of the descriptions, uses, strengths and weaknesses of ES are offered toward the goal of application in RCE tasks. Finally, this paper proposes the Rule-Engine Detection by Intermediate Representation (REDIR) system for automating the static detection of obfuscated anti-debugging techniques in software samples. Through the application of a rule-based ES, the process of detection is broken down into two phases: expert clue recognition of anti-debugging techniques and programmatic verification of detections. In this system, the ES looks for the clues that can lead to detection of anti-debug techniques. For each clue, REDIR attempts to extract a corresponding sub-program, instrumenting it to simulate a debugging condition and evaluate. Once complete, the user receives a list of confirmed detections to mitigate as desired.