Today we're going to be doing some rootkit debugging, specifically regarding runtime2, with a bit of a twist! I have a ton of rootkit debugging posts coming in the next few weeks, as I've decided to break them up rather than throwing them together in one giant mess of a post.
I've shown various scenarios in which I've debugged a rootkit before (0x7A, etc), but this time we're going to use various extensions to help us, other methods, and overall go a lot more in-depth. The postmortem runtime2 rootkit KMD that will be used in this post was generated by our beloved niemiro, so a big thanks to him! He aimed to make it a good example of some things a rootkit/malware developer can do to make things not as obvious when you resort to methods such as hooking the SSDT, which is rather old and very detectable these days.
Right, so here's our bug check. Most if not all of these 'older' rootkits will use Direct Kernel Object Manipulation (DKOM) to hook low-level routines/functions within the System Service Dispatch Table (SSDT). When this is occurring, assuming the developer of the rootkit didn't do a very good job in writing their rootkit, a lot can go wrong when disabling write protection, carelessly swapping memory, and inserting hooks. Malware scans from many AV programs can also cause crashes when they detect that the SSDT is hooked under certain circumstances. The rootkit can also intentionally call a bug check if written this way when it has detected a scan has initiated.
However, if the driver is well written, you may not crash at all. What do you do if you're suspicious of a rootkit infection/hooking, yet a bug check isn't occurring naturally due to proper programming of the rootkit? This is when in some cases (like in this specific example here), you may need to take matters into your own hands and either:
The first option is much less effective than the second, and that's due to the fact that the rootkit may not be doing any obvious hooking of the SSDT, etc, at the time of you forcing the crash. If you're investigating during a live session, it's much more effective and there are a lot more commands you can use. However, for learning purposes, I will show both.
1st argument - The value in the 2nd argument (0x3)implies it was a process as opposed to a thread that unexpectedly terminated. If it was a thread however, it would have instead been 0x6.
2nd argument - The value in the 2nd argument (86664d90) is a pointer to the _EPROCESS object that unexpectedly terminated. We can dump it using !object, which I will show below:
3rd argument - The value in the 3rd argument (86664edc) is the process image file name. Essentially, it's the process name that unexpectedly terminated. We can dump it by using dc, which I will show below:
The process that unexpectedly terminated was csrss.exe, which is the Client/Server Runtime Subsystem.
For some extra food for thought, we can get a lot of the information above manually if we'd like to. If we take the 2nd argument (_EPROCESS object) and run the following:
dt will display the type, which will show us the offset, etc. More specifically it displays information about a local variable, global variable or data type:
If we take the _EPROCESS object and add it to the offset (+0x14c), we get the following:
Look familiar? It's our 3rd argument, the process image name (csrss.exe).
4th argument - The value in the 3rd argument (819e91f0) is the explanatory message regarding the reason for the bug check, specifically in ASCII. To dump this, we'll need to use the dc command as we used earlier on the 3rd argument:
As we can see, this is reiterating the 1st argument.
So far all we know is that this bug check occurred because csrss.exe, a critical Windows process, unexpectedly terminated. Why? If you weren't able to tell whilst reading through by now, we purposely terminated it via Task Manager to force the bug check.With that said, for this post we're choosing the first method (forcing a bug check).
Pretending we ourselves didn't purposely force the bug check for a moment, what would we at this point do if this was a crash dump we were looking at from a system that isn't ours, and/or from our doing? Among many things, one of the first few things I do in 'not so obvious' 0xF4's is I check the summary and stats regarding virtual memory on the system at the time of the crash:
We can see right away that we don't have insufficient non-paged pool, which is a pretty popular cause of most 0xF4's as it at that point cannot handle I/O operations, etc. It's generally due to buggy drivers causing pool related memory leaks, etc.
With the above said, assuming we are looking at a crash dump that wasn't ours, we can almost entirely rule out this specific 0xF4 being caused by a buggy driver (MSFT or 3rd party). To be extra sure, double-check the modules list and see if there's anything that jumps out as problematic.
At this point I would start becoming suspicious of a rootkit, as I would not right away suggest the hard disk (whether HDD or SSD) is the immediate problem given we'd probably see obvious NT_STATUS codes for that. For example, possibly 0xc0000006. Either that, or an entirely different bug check (perhaps 0x7A). With that said, now we get to have some fun!
There are many ways to go about detecting a rootkit hooking the SSDT, and we will discuss extensions/scripts for the moment:
1st Method - SwishDbgExt
The first method we will be using in this postmortem debugging example is the wonderful SwishDbgExt, which was developed/created by a friend of mine (Matt Suiche). I've made various contributions to the help file considering the love I have gathered for this extension. It makes a lot of our lives as debuggers much easier.
Once you have the extension loaded, we're going to be using the !ms_ssdt command. This command displays the System Service Dispatch Table, which is extremely helpful in the investigation of suspected rootkit hooks through using what is known as Direct Kernel Object Manipulation (DKOM).
-- Chopping some of the SSDT output as it's fairly large, let's skip to what's important:
We can see a module (Gjglly.sys), and nt!NtFlushWriteBuffer is hooked. Let's not jump to conclusions just yet as this could be a completely legitimate hook.
First of all, what's Gjglly.sys? This is a driver in relation to AntiDebugLIB.
This is a legitimate driver, but we can also at the same time not jump to conclusions on it being safe as said above (hint: it's not - I will discuss later).
2nd Method - Script
The second method we will be using in this postmortem debugging example is an older script developed by Lionel d'Hauenens of Laboskopia. This script will only work with the x86 WinDbg client. Regardless, as we know, it's always best to debug a crash dump in the client based off of the system's architecture. Given the system that generated this crash dump was 32-bit, we'll be debugging it with the x86 WinDbg client.
This script is incredibly helpful in not only checking the SSDT for hooking, but also detecting what is known as a Shadow SSDT hook. You can find a great article on Shadow SSDT hooking from my very good friend Harry - Shadow SSDT Hooking with Windbg.
Nevertheless, if you don't understand French instructions, to install the script, simply drag and drop the script folder into your x86 Debuggers folder.
Once it's installed, type the following command into WinDbg with your postmortem rootkit crash dump:
So long as you installed the script properly, you should see the following:
Once you see that which has verified the script is loaded properly, type the following command:
Here's a few excerpts:
Here's Gjglly.sys again, and we can see it's hooking. At this point, we would be suspicious enough to run an ARK tool such as GMER. Of course, if we ran GMER at this point, it would show that the system is in fact infected with runtime2, and Gjglly.sys is our rootkit driver. Normally, by itself and by default, the runtime2 driver is runtime2.sys, not Gjglly.sys. In this specific scenario, niemiro used a different loader to inject the driver than the original, and renamed runtime2.sys to Gjglly.sys (a legitimate module name). Although GMER if run would still say the system is infected with a rootkit, and it would label Gjglly.sys as the rootkit driver, it's done not to trick various ARK tools, but the user.
If you've ever been infected with runtime2, you know it additionally drops startdrv.exe in the Temp directory of the current Windows install. startdrv.exe is the part that infuriates those who become infected with runtime2, because it's the part that immediately makes you suspicious of an infection as it's a trojan dropper. This is the specific part of the rootkit that will cause slowness, garbage popups, etc.
In this specific case, niemiro intentionally corrupted startdrv.exe which stopped it from executing entirely. You may be saying to yourself, what's the point of a rootkit's protection driver with no execution of payload, etc, by its dropped trojan? Well, there really is no point (unless your goal was to use something else that's malicious and not as obvious as a trojan dropper)! This was just an example to show how you can better hide a rootkit (protection driver, really) if you wanted to. Although ARK tools would still pick it up, it's not anywhere near as obvious to the user.
Thanks for reading, and I will get around to doing a live debugging of runtime2 as soon as I can.
References
Hunting rootkits with Windbg.
Some notes about System Service Dispatch Table hook.
I've shown various scenarios in which I've debugged a rootkit before (0x7A, etc), but this time we're going to use various extensions to help us, other methods, and overall go a lot more in-depth. The postmortem runtime2 rootkit KMD that will be used in this post was generated by our beloved niemiro, so a big thanks to him! He aimed to make it a good example of some things a rootkit/malware developer can do to make things not as obvious when you resort to methods such as hooking the SSDT, which is rather old and very detectable these days.
Code:
CRITICAL_OBJECT_TERMINATION (f4)
A process or thread crucial to system operation has unexpectedly exited or been
terminated.
Several processes and threads are necessary for the operation of the
system; when they are terminated (for any reason), the system can no
longer function.
Arguments:
Arg1: 00000003, [COLOR=#ff0000]Process [/COLOR]
Arg2: 86664d90, [COLOR=#0000cd]Terminating object [/COLOR]
Arg3: 86664edc, [COLOR=#800080]Process image file name[/COLOR]
Arg4: 819e91f0, [COLOR=#006400]Explanatory message (ascii)[/COLOR]Right, so here's our bug check. Most if not all of these 'older' rootkits will use Direct Kernel Object Manipulation (DKOM) to hook low-level routines/functions within the System Service Dispatch Table (SSDT). When this is occurring, assuming the developer of the rootkit didn't do a very good job in writing their rootkit, a lot can go wrong when disabling write protection, carelessly swapping memory, and inserting hooks. Malware scans from many AV programs can also cause crashes when they detect that the SSDT is hooked under certain circumstances. The rootkit can also intentionally call a bug check if written this way when it has detected a scan has initiated.
However, if the driver is well written, you may not crash at all. What do you do if you're suspicious of a rootkit infection/hooking, yet a bug check isn't occurring naturally due to proper programming of the rootkit? This is when in some cases (like in this specific example here), you may need to take matters into your own hands and either:
- Force a bug check.
- Live kernel debugging session.
- Run an ARK (Anti-Rootkit) tool. This is no fun... we want to have fun and learn : )
The first option is much less effective than the second, and that's due to the fact that the rootkit may not be doing any obvious hooking of the SSDT, etc, at the time of you forcing the crash. If you're investigating during a live session, it's much more effective and there are a lot more commands you can use. However, for learning purposes, I will show both.
1st argument - The value in the 2nd argument (0x3)implies it was a process as opposed to a thread that unexpectedly terminated. If it was a thread however, it would have instead been 0x6.
2nd argument - The value in the 2nd argument (86664d90) is a pointer to the _EPROCESS object that unexpectedly terminated. We can dump it using !object, which I will show below:
Code:
kd> !object [COLOR=#0000cd]86664d90 [/COLOR]
Object: [COLOR=#0000cd]86664d90 [/COLOR]Type: (841537e8) Process
ObjectHeader: 86664d78 (old version)
HandleCount: 4 PointerCount: 1273rd argument - The value in the 3rd argument (86664edc) is the process image file name. Essentially, it's the process name that unexpectedly terminated. We can dump it by using dc, which I will show below:
Code:
kd> dc 86664edc
86664edc 73727363 78652e73 00000065 00000000 [COLOR=#ff0000]csrss.exe[/COLOR].......
86664eec 00000000 00000000 00000000 8605a278 ............x...
86664efc 869356d8 00000000 00000000 0000000a .V..............
86664f0c 8c04d631 00000000 00000000 7ffdc000 1...............
86664f1c 00000000 00000099 00000000 00000000 ................
86664f2c 00000000 000005c6 00000000 0003cc50 ............P...
86664f3c 00000000 00000000 00000000 00006d77 ............wm..
86664f4c 00000000 00000000 00000162 00000000 ........b.......The process that unexpectedly terminated was csrss.exe, which is the Client/Server Runtime Subsystem.
For some extra food for thought, we can get a lot of the information above manually if we'd like to. If we take the 2nd argument (_EPROCESS object) and run the following:
Code:
dt _EPROCESS [COLOR=#0000cd]86664d90 [/COLOR]imageFileNamedt will display the type, which will show us the offset, etc. More specifically it displays information about a local variable, global variable or data type:
Code:
kd> dt _EPROCESS [COLOR=#0000cd]86664d90 [/COLOR]imageFileName
nt!_EPROCESS
[COLOR=#a52a2a]+0x14c[/COLOR] ImageFileName : [16] "[COLOR=#ff0000]csrss.exe[/COLOR]"If we take the _EPROCESS object and add it to the offset (+0x14c), we get the following:
Code:
kd> ? 86664d90+0x14c
Evaluate expression: -2040115492 = [COLOR=#800080]86664edc[/COLOR]Look familiar? It's our 3rd argument, the process image name (csrss.exe).
4th argument - The value in the 3rd argument (819e91f0) is the explanatory message regarding the reason for the bug check, specifically in ASCII. To dump this, we'll need to use the dc command as we used earlier on the 3rd argument:
Code:
kd> dc 819e91f0
819e91f0 6d726554 74616e69 20676e69 74697263 [COLOR=#ff8c00]Terminating crit[/COLOR]
819e9200 6c616369 6f727020 73736563 25783020 [COLOR=#ff8c00]ical process[/COLOR] 0x%As we can see, this is reiterating the 1st argument.
So far all we know is that this bug check occurred because csrss.exe, a critical Windows process, unexpectedly terminated. Why? If you weren't able to tell whilst reading through by now, we purposely terminated it via Task Manager to force the bug check.With that said, for this post we're choosing the first method (forcing a bug check).
Pretending we ourselves didn't purposely force the bug check for a moment, what would we at this point do if this was a crash dump we were looking at from a system that isn't ours, and/or from our doing? Among many things, one of the first few things I do in 'not so obvious' 0xF4's is I check the summary and stats regarding virtual memory on the system at the time of the crash:
Code:
kd> !vm
*** Virtual Memory Usage ***
Physical Memory: 917370 ( 3669480 Kb)
Page File: \??\C:\pagefile.sys
Current: 3976680 Kb Free Space: 3976676 Kb
Minimum: 3976680 Kb Maximum: 4193280 Kb
Available Pages: 769568 ( 3078272 Kb)
ResAvail Pages: 869504 ( 3478016 Kb)
Locked IO Pages: 0 ( 0 Kb)
Free System PTEs: 348283 ( 1393132 Kb)
Modified Pages: 11884 ( 47536 Kb)
Modified PF Pages: 11830 ( 47320 Kb)
[COLOR=#ff0000]NonPagedPool Usage: 8265 ( 33060 Kb)[/COLOR]
[COLOR=#ff0000]NonPagedPool Max: 522998 ( 2091992 Kb)[/COLOR]
PagedPool 0 Usage: 5501 ( 22004 Kb)
PagedPool 1 Usage: 10013 ( 40052 Kb)
PagedPool 2 Usage: 623 ( 2492 Kb)
PagedPool 3 Usage: 631 ( 2524 Kb)
PagedPool 4 Usage: 726 ( 2904 Kb)
PagedPool Usage: 17494 ( 69976 Kb)
PagedPool Maximum: 523264 ( 2093056 Kb)
Session Commit: 3758 ( 15032 Kb)
Shared Commit: 9951 ( 39804 Kb)
Special Pool: 0 ( 0 Kb)
Shared Process: 1260 ( 5040 Kb)
Pages For MDLs: 2 ( 8 Kb)
PagedPool Commit: 17550 ( 70200 Kb)
Driver Commit: 2335 ( 9340 Kb)
Committed pages: 115256 ( 461024 Kb)
Commit limit: 1882649 ( 7530596 Kb)We can see right away that we don't have insufficient non-paged pool, which is a pretty popular cause of most 0xF4's as it at that point cannot handle I/O operations, etc. It's generally due to buggy drivers causing pool related memory leaks, etc.
With the above said, assuming we are looking at a crash dump that wasn't ours, we can almost entirely rule out this specific 0xF4 being caused by a buggy driver (MSFT or 3rd party). To be extra sure, double-check the modules list and see if there's anything that jumps out as problematic.
At this point I would start becoming suspicious of a rootkit, as I would not right away suggest the hard disk (whether HDD or SSD) is the immediate problem given we'd probably see obvious NT_STATUS codes for that. For example, possibly 0xc0000006. Either that, or an entirely different bug check (perhaps 0x7A). With that said, now we get to have some fun!
There are many ways to go about detecting a rootkit hooking the SSDT, and we will discuss extensions/scripts for the moment:
1st Method - SwishDbgExt
The first method we will be using in this postmortem debugging example is the wonderful SwishDbgExt, which was developed/created by a friend of mine (Matt Suiche). I've made various contributions to the help file considering the love I have gathered for this extension. It makes a lot of our lives as debuggers much easier.
Once you have the extension loaded, we're going to be using the !ms_ssdt command. This command displays the System Service Dispatch Table, which is extremely helpful in the investigation of suspected rootkit hooks through using what is known as Direct Kernel Object Manipulation (DKOM).
-- Chopping some of the SSDT output as it's fairly large, let's skip to what's important:
Code:
|-------|--------------------|--------------------------------------------------------|---------|--------|
| Index | Address | Name | Patched | Hooked |
|-------|--------------------|--------------------------------------------------------|---------|--------|
[COLOR=#ff8c00]*** ERROR: Module load completed but symbols could not be loaded for Gjglly.sys [/COLOR]
| 126 | 0xFFFFFFFF91F054C2 | [COLOR=#ff0000]Gjglly [/COLOR]| | |
| 127 | 0xFFFFFFFF81A34F80 | nt!NtDeviceIoControlFile | | |
| 128 | 0xFFFFFFFF81949B44 | nt!NtDisplayString | | |
| 129 | 0xFFFFFFFF81A2117F | nt!NtDuplicateObject | | |
| 130 | 0xFFFFFFFF81A18134 | nt!NtDuplicateToken | | |
| 131 | 0xFFFFFFFF81AB14E8 | nt!NtEnumerateBootEntries | | |
| 132 | 0xFFFFFFFF81AB278A | nt!NtEnumerateDriverEntries | | |
| 133 | 0xFFFFFFFF91F04FFA | [COLOR=#ff0000]Gjglly [/COLOR]| | |
| 134 | 0xFFFFFFFF81AB10B7 | nt!NtEnumerateSystemEnvironmentValuesEx | | |
| 135 | 0xFFFFFFFF81A9F073 | nt!NtEnumerateTransactionObject | | |
| 136 | 0xFFFFFFFF91F051B6 | [COLOR=#ff0000]Gjglly [/COLOR]| | |
| 137 | 0xFFFFFFFF81A802D5 | nt!NtExtendSection | | |
| 138 | 0xFFFFFFFF819A113A | nt!NtFilterToken | | |
| 139 | 0xFFFFFFFF819B39FC | nt!NtFindAtom | | |
| 140 | 0xFFFFFFFF819DCA86 | nt!NtFlushBuffersFile | | |
| 141 | 0xFFFFFFFF819AD0F6 | nt!NtFlushInstructionCache | | |
| 142 | 0xFFFFFFFF819781EB | nt!NtFlushKey | | |
| 143 | 0xFFFFFFFF818B11C1 | nt!NtFlushProcessWriteBuffers | | |
| 144 | 0xFFFFFFFF819C175B | nt!NtFlushVirtualMemory | | |
| 145 | 0xFFFFFFFF81A82D64 | [COLOR=#0000cd]nt!NtFlushWriteBuffer[/COLOR] | | [COLOR=#ff0000]Yes [/COLOR]|We can see a module (Gjglly.sys), and nt!NtFlushWriteBuffer is hooked. Let's not jump to conclusions just yet as this could be a completely legitimate hook.
First of all, what's Gjglly.sys? This is a driver in relation to AntiDebugLIB.
This is a legitimate driver, but we can also at the same time not jump to conclusions on it being safe as said above (hint: it's not - I will discuss later).
2nd Method - Script
The second method we will be using in this postmortem debugging example is an older script developed by Lionel d'Hauenens of Laboskopia. This script will only work with the x86 WinDbg client. Regardless, as we know, it's always best to debug a crash dump in the client based off of the system's architecture. Given the system that generated this crash dump was 32-bit, we'll be debugging it with the x86 WinDbg client.
This script is incredibly helpful in not only checking the SSDT for hooking, but also detecting what is known as a Shadow SSDT hook. You can find a great article on Shadow SSDT hooking from my very good friend Harry - Shadow SSDT Hooking with Windbg.
Nevertheless, if you don't understand French instructions, to install the script, simply drag and drop the script folder into your x86 Debuggers folder.
Once it's installed, type the following command into WinDbg with your postmortem rootkit crash dump:
Code:
kd> $$><script\@@init_cmd.wdbgSo long as you installed the script properly, you should see the following:
Code:
SysecLabs Windbg Script : Ok :)
('al' for display all commands)Once you see that which has verified the script is loaded properly, type the following command:
Code:
!!display_system_callHere's a few excerpts:
Code:
007D 0003 OK nt!NtDeleteObjectAuditAlarm (81a46500)
[COLOR=#ff8c00]007E 0002 [/COLOR][COLOR=#ff0000]HOOK[/COLOR][COLOR=#ff8c00]-> *** ERROR: Module load completed but symbols could not be loaded for Gjglly.sys [/COLOR]
007F 000A OK nt!NtDeviceIoControlFile (81a34f80)
0080 0001 OK nt!NtDisplayString (81949b44)
0081 0007 OK nt!NtDuplicateObject (81a2117f)
0082 0006 OK nt!NtDuplicateToken (81a18134)
0083 0002 OK nt!NtEnumerateBootEntries (81ab14e8)
0084 0002 OK nt!NtEnumerateDriverEntries (81ab278a)
[COLOR=#ff8c00]0085 0006 [/COLOR][COLOR=#ff0000]HOOK[/COLOR][COLOR=#ff8c00]-> Gjglly+0x1ffa (91f04ffa) [/COLOR]
0086 0003 OK nt!NtEnumerateSystemEnvironmentValuesEx (81ab10b7)
0087 0005 OK nt!NtEnumerateTransactionObject (81a9f073)
[COLOR=#ff8c00]0088 0006 [/COLOR][COLOR=#ff0000]HOOK[/COLOR][COLOR=#ff8c00]-> Gjglly+0x21b6 (91f051b6)[/COLOR]
Code:
00BC 0003 OK nt!NtOpenJobObject (81a912df)
[COLOR=#ff8c00]00BD 0003 [/COLOR][COLOR=#ff0000]HOOK[/COLOR][COLOR=#ff8c00]-> Gjglly+0x1f4c (91f04f4c)[/COLOR]
00BE 0004 OK nt!NtOpenKeyTransacted (819727e1)
Code:
0143 0001 OK nt!NtSetUuidSeed (8195977b)
[COLOR=#ff8c00]0144 0006 [/COLOR][COLOR=#ff0000]HOOK[/COLOR][COLOR=#ff8c00]-> Gjglly+0x2372 (91f05372) [/COLOR]
0145 0005 OK nt!NtSetVolumeInformationFile (81a6c5de)Here's Gjglly.sys again, and we can see it's hooking. At this point, we would be suspicious enough to run an ARK tool such as GMER. Of course, if we ran GMER at this point, it would show that the system is in fact infected with runtime2, and Gjglly.sys is our rootkit driver. Normally, by itself and by default, the runtime2 driver is runtime2.sys, not Gjglly.sys. In this specific scenario, niemiro used a different loader to inject the driver than the original, and renamed runtime2.sys to Gjglly.sys (a legitimate module name). Although GMER if run would still say the system is infected with a rootkit, and it would label Gjglly.sys as the rootkit driver, it's done not to trick various ARK tools, but the user.
If you've ever been infected with runtime2, you know it additionally drops startdrv.exe in the Temp directory of the current Windows install. startdrv.exe is the part that infuriates those who become infected with runtime2, because it's the part that immediately makes you suspicious of an infection as it's a trojan dropper. This is the specific part of the rootkit that will cause slowness, garbage popups, etc.
In this specific case, niemiro intentionally corrupted startdrv.exe which stopped it from executing entirely. You may be saying to yourself, what's the point of a rootkit's protection driver with no execution of payload, etc, by its dropped trojan? Well, there really is no point (unless your goal was to use something else that's malicious and not as obvious as a trojan dropper)! This was just an example to show how you can better hide a rootkit (protection driver, really) if you wanted to. Although ARK tools would still pick it up, it's not anywhere near as obvious to the user.
Thanks for reading, and I will get around to doing a live debugging of runtime2 as soon as I can.
References
Hunting rootkits with Windbg.
Some notes about System Service Dispatch Table hook.
Last edited:
