Analyzing network traffic to detect self-decrypting exploit code

Remotely-launched software exploits are a common way for attackers to intrude into vulnerable computer systems. As detection techniques improve, remote exploitation techniques are also evolving. Recent techniques for evasion of exploit detection include polymorphism (code encryption) and metamorphism (code obfuscation). This paper addresses the problem of detecting in network traffic polymorphic remote exploits that are encrypted, and that self-decrypt before launching the intrusion. Such exploits pose a great challenge to existing malware detection techniques, partly due to the non-obvious starting location of the exploit code in the network payload. We describe a new method for detecting self-decrypting exploit codes. This method scans network traffic for the presence of a decryption routine, which is characteristic of such exploits. The proposed method uses static analysis and emulated instruction execution techniques. This improves the accuracy of determining the starting location...

• Type: Conference
• Year: 2007
• Where: CCS
• Authors: Qinghua Zhang, Douglas S. Reeves, Peng Ning, S. Purushothaman Iyer
• Keywords: CCS 2007, Decryption Routine, Detection, Detection Techniques, Security Privacy