Detect Compiler Inserted Run-time Security Checks in Binary Software

Our goal in this work is to develop a mechanism to determine the presence of targeted compiler-based or automated rules-based runtime security checks in any given binary. Our generalized approach relies on several key insights. First, instructions added by automated checks likely follow just one or only a few fixed patterns or templates at every insertion point. Second, any security check will guard some interesting or vulnerable program structure, like return addresses, indirect jumps/calls, etc., and the placement of the security check will inform about the nature of the check. By contrast, we would not expect ordinary user code to follow any single pattern at every such interesting program location. Our technique to detect automated security checks in binary code does not rely on known code signatures that can change depending on the language, the compiler, and the security check. We implement and evaluate our technique, and present our results, observations, and challenges in this work.