This blog post comes from Stewart Sentanoe of the University of Passau. Stewart is a PhD student and he was recently a Google Summer of Code Intern working on the Honeynet Project.
Project Introduction
Virtual Machine Introspection
Virtual Machine Introspection (VMI) is the process of examining and monitoring a virtual machine from the hypervisor or virtual machine monitor (VMM) point of view. Using this approach, we can get the untainted information of the monitored virtual machine. There are three main problems with VMI currently:
- Semantic gap: How do you interpret low level data into useful information?
- Performance impact: How big is the overhead?
- Stealthiness: How to make the monitoring mechanism hard to be detected by the adversary?
This project focused on the third problem, and specifically on how to hide the breakpoint that has been set. We do not want the adversary to be able to detect whether there is a breakpoint that has been set to some of the memory addresses. If they are able to detect the breakpoint, most likely the adversary will not continue the attack and we will learn nothing. By leveraging VMI, we are able to build high interaction honeypot where the adversary can do whatever they want with the system. Thus, we can gather as much information as we can and we get the big picture of what’s going on in the system and learn from it.
Setting a Breakpoint Implemented by Drakvuf
DRAKVUF is a virtualization based agentless black-box binary analysis system developed by Tamas K Lengyel. DRAKVUF allows for in-depth execution tracing of arbitrary binaries (including operating systems), all without having to install any special software within the virtual machine used for analysis (https://drakvuf.com and https://github.com/tklengyel/drakvuf).
There are two ways to set a breakpoint implemented by DRAKVUF using INT3 (0xCC opcode) and Xen altp2m.
These are the following steps by DRAKVUF to inject breakpoint using INT3:
- Inject 0xCC into the target
- Mark pages Execute-only in the EPT (Extended Page Tables)
- If anything tries to read the page:
- Remove 0xCC and mark page R/W/X
- Singlestep
- Place 0xCC back and mark page X-only
- When 0xCC traps to Xen
- Remove 0xCC
- Singlestep
- Place 0xCC back
Sounds good right? But, there is a big problem when INT3 is used.
To make the breakpoint mechanism work well with multiple vCPUs, DRAKVUF uses Xen altp2m. At the normal runtime of a VM, each guest’s physical memory (GFN – Guest Frame Number) will be mapped one to one to the machine (host) physical memory (MFN – Machine Frame Number) as shown in the image below.
Next, to set a breakpoint, DRAKVUF will copy the target page to the end of the guest’s physical memory and add the trap there. DRAKVUF will also make an empty page (the purposes will be explained later) as shown below.
Now, during the runtime, the pointer of the original target will be switched h to the copy page as shown below and marked as execute only.
If a process tries to execute those pages, it can simply switch the pointer back to the original, single step and then switch the pointer to the copy page again. You might be thinking that if an adversary is able to scan “beyond” the physical memory, the adversary will detect a page that contains the copy. This where the empty page kicks in, whenever a process tries to read or write to the copy page, DRAKVUF will simply change the pointer to the empty page as shown below.
Sounds cool doesn’t it? Of course it is! But, there are several problems with this process, which led to this GSOC project. The sections below will cover them piece by piece.
Problems of DRAKVUF
There are three problems that I found out during this project:
- There’s a M:1 relation between the shadow copy and the empty page, which means that if we set breakpoints to two addresses, it will create two shadow copy and only one empty page.
- If an adversary is able to write “something” to a shadow copy, the “something” will also appear on the other shadow copy which can raise their suspicious level.
- The current implementation of DRAKVUF will use ’00’ for the empty page, but the real behaviour never been observed.
Proposed Milestones
There are two milestones for this project:
- Create a proof of concept (kernel module) that detects the presence of DRAKVUF by trying to write “something” to one of the shadow copy and probe the second shadow copy to check the existence of the “something”
- Patch DRAKVUF
The Process and the Findings
At the beginning of this project, I had no idea how to read the memory beyond the physical address space, but then I found this article which describes a function (ioremap) that I used for my kernel module (available here). The drawback is that it requires some debug information generated by DRAKVUF, for example the address of the shadow copy.
https://gist.github.com/scorpionzezz/d4f19fb9cc61ff4b50439e9b1846a17a
When I executed the code without the writing part, I got this: https://gist.github.com/scorpionzezz/6e4bdd0b22d5877057823a045c784721
As expected, it gave me empty result. Then, when I wrote “something” to the first address which in this point is letter ‘A’ (in hex is 41). The ‘A’ also appears on the second address: https://gist.github.com/scorpionzezz/ce6623f1176e99de61617222ceba462a
Bingo! Something fishy there. Alright, then I tried to print more addresses: https://gist.github.com/scorpionzezz/22bdb3c727dd130bb59b28cf717d9bac
Did you see something weird there? Yes, the ‘FF’, actually the empty is ‘FF’ instead ’00’. So actually, an adversary does not need to write “something” to the empty page, it just simply detects if there are ’00’ then it reveals the presence of DRAKVUF.
But where is the ‘FF’ comes from? Architecturally, all physical addresses defined by CPUID EAX=80000008h bits 15-8 (more here) are considered “valid” In Linux, it checks the address validity when it sets up the memory page table (see here). It is up to the firmware to tell the OS and the hypervisor what range are valid with the E820 map (see here). When a process requests a memory address that is not valid (assuming the new page table is made), it goes through the Memory Management Unit (MMU) and then Platform Controller Hub (PCH). The PCH tries to find the valid physical memory but could not found it then, if it involves write, the written value will be ignored and if it involves read, it will return all 1s. This behaviour is written into this (page 32) Intel document and anyway VMI (for now) just works on Intel processor.
Alright, now time to fix this.
First is pretty easy where I just write ‘FF’ to the shadow page: https://gist.github.com/scorpionzezz/9853d836b38b82c2961c1d437390c8a3
It solved the simple problem. But now let’s talk about the bigger problem about the writing. The idea is to simply ignore write attempt to the shadow page and also to the empty page. For both cases, we can use features provided by Xen, which emulate the write. Sounds easy, but actually there was another problem: LibVMI (library that used by DRAKVUF) does not support the write emulation flag, so I needed to patch it up (see here).
Alright, now I check whenever a process tries to write to the shadow copy, then just do the emulation: https://gist.github.com/scorpionzezz/3a12bebdd43d5717d671136f0fc0069c
Now, we also need to add TRAP to the shadow copy so we can also do emulation whenever a process tries to write to it. https://gist.github.com/scorpionzezz/763cd6b9f257105f2941e104cf6f2d8e
Now every time a process tries to write to either the empty page and the shadow copy, the written value will be not “stored” in the memory. Thus, it hides DRAKVUF better.
Conclusion
This project increases the stealthiness level of DRAKVUF. With a high level of stealthiness, it opens up the potential for a new generation honeypots, intrusion detection systems and dynamic malware analysis where it will be hard for the adversary to detect the presence of the monitoring system.
Acknowledgement
Thanks to Tamas K Lengyel and Varga-Perke Balint you rock! Thank you for your help and patience. Thank you also for Benjamin Taubmann for the support and of course Honeynet and Google for GSOC 2018 🙂