Software Research Laboratory

Introduction

DOC is a static analysis suite that detects obfuscations in executable, particularly procedure call and call-return obfuscations. It uses abstract interpretation (AI) to find instances where explicit call or call-return instructions are not used. A prototype is implemented as an Eclipse plugin for browsing X86 executables.

How It Works

Embedded within every executable program are tell-tale signs of the program’s intent located in a table of system calls that are needed by the program. This list of system calls can be used to determine a program’s behavior prior to executing the program. Consider a program that makes use of the system calls bind, send, and closesocket. It is quite clear that data transfer over a network will take place if this program is run. Malicious code writers, being wary of such transparency in their code, defend their programs’ privacy by obfuscating the table of required system calls, thus obfuscating the program’s true behavior. A common approach to this form of obfuscation is known as “call obfuscation.”

In call obfuscation, the detection of used system calls is made difficult by replacing the existing call and ret instructions with a different, but equivalent, set of instructions. For instance, the two sequences of code shown below are equivalent; they both call the method DeleteFileA. However, the second sequence contains no call statement, making it difficult to know a call even exists at this program point. This rather simple trick is enough to confound many analysis tools.

Our tool, DOC (Detector of Obfuscated Calls), is capable of statically detecting such code obfuscation by interpreting the executable and building an abstract representation of the registers and stack, which allows us to detect pieces of code that violate standard calling conventions. The result is a more complete listing of system calls used by the program. Using this improved list of system calls, one is in a better position to accurately gauge a program’s runtime behavior, a crucial first step in detecting malicious behavior.

In the screenshot of DOC below, the results of interpreting an executable are displayed. The in the left column marks an instruction that is a valid call statement, meaning it is not obfuscated. The marks instructions that have been found to be obfuscated. Manually tracing the code reveals that the instructions marked with do indeed simulate a call.

Eclipse Interface for DOC