Skip to main content

Resurrecting anti-virtualization and anti-debugging: Unhooking your hooks.

Apostolopoulos, T., Katos, V., Choo, R. and Patsakis, C., 2021. Resurrecting anti-virtualization and anti-debugging: Unhooking your hooks. Future Generation Computer Systems, 116 (March), 393-405.

Full text available as:

Anti_forensics.pdf - Accepted Version
Available under License Creative Commons Attribution Non-commercial No Derivatives.


DOI: 10.1016/j.future.2020.11.004


Dynamic malware analysis involves the debugging of the associated binary files and the monitoring of changes in sandboxed environments. This allows the investigator to manipulate the code execution path and environment to develop an understanding of the malware’s internal workings, aims and modus operandi. However, the latest state of the art malware may incor- porate anti-virtual environment (VM) and anti-debugging countermeasures (i.e. to determine whether the malware is being executed in a VM or us- ing a debugger prior to payload execution). We argue that for the malware to be effective, it will need to support an array of anti-detection and eva- sion mechanisms. In essence, from the malware’s perspective, it needs to adopt a “defence in depth” paradigm to achieve its underlying business logic functionality. Beyond the malicious uses, software vendors to preserve the intellectual property rights of their products often resort to similar methods to deter competitors from gaining intelligence from the binaries or prevent customers from using their products in unauthorised hardware. In this work, we illustrate how Windows architecture impedes the work of debuggers when they analyse with armoured binaries. The debugger and the malware have the same privileges, so the attacker may manipulate theaddress space that the debugger operates and, e.g. bypass detection. We showcase this by presenting a new framework (ANTI), which automates the procedure of integrating anti-debugging and anti-VM in the binary. Specifi- cally, ANTI introduces an anti-hooking method targeting Windows binaries, where hooks applied by state of the art debuggers are removed and injects its code in other processes. This significantly compounds the challenge of binary analysis. Our extensive evaluation also demonstrates that ANTI successfully circumvents detection from state-of-the-art detection methods. Therefore, ANTI illustrates that current tools for dynamic analysis have serious implementation gaps that allow for binaries to bypass them. More alarmingly, ANTI shows how one can use well-known methods to “resurrect” old attacks.

Item Type:Article
Uncontrolled Keywords:Malware; Windows hooking; dynamic analysis; anti-debugging; anti-virtualization
Group:Faculty of Science & Technology
ID Code:34823
Deposited By: Symplectic RT2
Deposited On:13 Nov 2020 14:52
Last Modified:14 Mar 2022 14:25


Downloads per month over past year

More statistics for this item...
Repository Staff Only -