Debugging Stop 0x122 - WHEA_INTERNAL_ERROR

x BlueRobot

Administrator
Staff member
Joined
May 7, 2013
Posts
10,396
This is quite an obscure bugcheck, which doesn't really require much investigation, but since I haven't posted anything useful to Sysnative in a while (assignments etc.), I thought I would consider writing a small article regarding this bugcheck.

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Background:

Stop 0x122 is primarily caused, as the name suggests, by a internal error occurring in the WHEA error reporting system implemented in Windows since Windows Vista. The parameters of the bugcheck give an indication into the the type of internal error, and thus what might need investigating.

Microsoft has provided some documentation on the WHEA system, which will not provided here; I'll save it for a Stop 0x124 Complete Guide Debugging post.


~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Debugging Stop 0x122:

As discussed earlier, this is mainly a parameter checking bugcheck, and then investigating the stack to find a little more information. Firstly, let's dump the bugcheck parameters with .bugcheck or .dumpdebug.

Code:
2: kd> [COLOR=#008000].bugcheck[/COLOR]
Bugcheck code [COLOR=#ff0000]00000122[/COLOR]
Arguments 00000000`000000[COLOR=#0000ff]0a[/COLOR] 00000000`000000[COLOR=#ff8c00]04[/COLOR] 00000000`00000000 00000000`00000000

The first parameter dictates the meaning of the second parameter, in case 0xA indicates that the second parameter is an error source type.

Alternatively, we could have gathered the same information with the undocumented .dumpdebug command:

Code:
2: kd> [COLOR=#008000].dumpdebug[/COLOR]
----- 64 bit Kernel Mini Dump Analysis

DUMP_HEADER64:
MajorVersion        0000000f
MinorVersion        00001db1
KdSecondaryVersion  00000000
DirectoryTableBase  00000001`67910000
PfnDataBase         fffff800`0350b228
PsLoadedModuleList  fffff800`0349fe90
PsActiveProcessHead fffff800`03481b90
MachineImageType    00008664
NumberProcessors    00000008
BugCheckCode        [COLOR=#ff0000]00000122[/COLOR]
BugCheckParameter1  00000000`000000[COLOR=#0000cd]0a[/COLOR]
BugCheckParameter2  00000000`0000000[COLOR=#ffa500]4[/COLOR]
BugCheckParameter3  00000000`00000000
BugCheckParameter4  00000000`00000000
KdDebuggerDataBlock fffff800`0344b0a0
ProductType         00000001
SuiteMask           00000110
WriterStatus        00000000
MiniDumpFields      00000cff 

TRIAGE_DUMP64:
ServicePackBuild      00000100 
SizeOfDump            00040000 
ValidOffset           0003fffc 
ContextOffset         00000348 
ExceptionOffset       00000f00 
MmOffset              00002080 
UnloadedDriversOffset 000020d0 
PrcbOffset            00002298 
ProcessOffset         00006f98 
ThreadOffset          00007468 
CallStackOffset       00007900 
SizeOfCallStack       00001000 
DriverListOffset      00008c40 
DriverCount           000000d1 
StringPoolOffset      000101d0 
StringPoolSize        00004698 
BrokenDriverOffset    00000000 
TriageOptions         ffffffff 
TopOfStack            fffff880`0cbc7d48 
BStoreOffset          00000000 
SizeOfBStore          00000000 
LimitOfBStore         00000000`00000000 
DebuggerDataOffset    00008900 
DebuggerDataSize      00000340 
DataBlocksOffset      00014868 
DataBlocksCount       0000001a 
  fffff800`034a0320 - fffff800`034a03bb at offset 00014a08
  fffff800`0348812c - fffff800`0348812f at offset 00014aa4
  fffff800`03488128 - fffff800`0348812b at offset 00014aa8
  fffff800`03410880 - fffff800`0341097f at offset 00014aac
  fffff800`034d6ca0 - fffff800`034d6ca3 at offset 00014bac
  fffff800`034d68a0 - fffff800`034d69bf at offset 00014bb0
  fffff800`0350b074 - fffff800`0350b077 at offset 00014cd0
  fffff800`0350b068 - fffff800`0350b06b at offset 00014cd4
  fffffa80`06cdb490 - fffffa80`06cdb677 at offset 00014cd8
  fffffa80`06cdc010 - fffffa80`06cdc1f7 at offset 00014ec0
  fffffa80`06cdcc10 - fffffa80`06cdcdf7 at offset 000150a8
  fffffa80`06cdc810 - fffffa80`06cdc9f7 at offset 00015290
  fffffa80`06cdc410 - fffffa80`06cdc5f7 at offset 00015478
  fffffa80`06cdd010 - fffffa80`06cdd1f7 at offset 00015660
  fffffa80`06cddc10 - fffffa80`06cdddf7 at offset 00015848
  fffffa80`06cdd810 - fffffa80`06cdd9f7 at offset 00015a30
  fffffa80`06cdd410 - fffffa80`06cdd5f7 at offset 00015c18
  fffffa80`06cde010 - fffffa80`06cde1f7 at offset 00015e00
  fffffa80`07303000 - fffffa80`073039f7 at offset 00015fe8
  fffff800`03473610 - fffff800`03473617 at offset 000169e0
  fffff800`03473618 - fffff800`0347361f at offset 000169e8
  fffffa80`0ebf4918 - fffffa80`0ebf5917 at offset 000169f0
  fffff800`03473620 - fffff800`03473627 at offset 000179f0
  fffff800`03473628 - fffff800`0347362f at offset 000179f8
  fffffa80`06ce3000 - fffffa80`06ce3fff at offset 00017a00
  fffff800`032da000 - fffff800`032dafff at offset 00018a00
  Max offset 19a00, 2e2f8 from end of file

By using the MSDN bugcheck reference, we can see that the WHEA reporting system failed to allocate a error record for the a PCI express related error. All error records have the following structure:

Code:
2: kd> [COLOR=#008000]dt hal!_WHEA_ERROR_RECORD[/COLOR]
   +0x000 Header           : _WHEA_ERROR_RECORD_HEADER
   +0x080 SectionDescriptor : [1] _WHEA_ERROR_RECORD_SECTION_DESCRIPTOR

The error record contains important debugging information which we usually use in relation to a Stop 0x124.

Now, let's investigate the call stack:

Code:
2: kd> [COLOR=#008000]knL[/COLOR]
 # Child-SP          RetAddr           Call Site
00 fffff880`0cbc7d48 fffff800`033e7693 nt!KeBugCheckEx
01 fffff880`0cbc7d50 fffff880`00e73bcf [COLOR=#0000ff]nt!WheaReportHwError+0x123[/COLOR]
02 fffff880`0cbc7db0 fffff880`00e735f6 [COLOR=#ff0000]pci!ExpressRootPortAerInterruptRoutine+0x27f[/COLOR]
03 fffff880`0cbc7e10 fffff800`032d647c [COLOR=#ff0000]pci!ExpressRootPortInterruptRoutine+0x36[/COLOR]
04 fffff880`0cbc7e80 fffff800`03310d99 nt!KiInterruptDispatch+0x16c
05 fffff880`0cbc8010 fffff800`0330fced nt!KxFlushEntireTb+0xa5
06 fffff880`0cbc8050 fffff800`033339e0 nt!KeFlushTb+0x119
07 fffff880`0cbc80d0 fffff800`032e809d [COLOR=#800080]nt! ?? ::FNODOBFM::`string'+0xae02[/COLOR] << [I]nt!MiInitializePfn[/I]
08 fffff880`0cbc8110 fffff800`032d876e nt!MmAccessFault+0xa7d
09 fffff880`0cbc8270 fffff800`035bb5d4 nt!KiPageFault+0x16e
0a fffff880`0cbc8400 fffff880`01776809 nt!NtQueryObject+0xa4
0b fffff880`0cbc8510 00000000`00000001 aswSnx+0x1f809
0c fffff880`0cbc8518 00000000`00000000 0x1

We can certainly see some PCI related calls in the stack, which fits perfectly with the context of the thread we're investigating. The nt!WheaReportHwError indicates that we attempted to alert the rest of the operating system to a potential hardware failure, but then failed to do so for some reason, thereby leading to the bugcheck which we just experienced. The pci!ExpressRootPortAerInterruptRoutine gives a small indication to where the error may have occurred, this will require breaking into leaked PCI documentation if wish to understand more about potentially this function might be doing.

Using the !dpx extension, we can see a much more informative stack, we gives light on some suspicious security drivers, which knowing McAfee and their poor range of products, could be meddling with Windows. Furthermore, there is signs of two security programs being run, which always usually means disaster!

Code:
2: kd> [COLOR=#008000]!dpx[/COLOR]
Start memory scan  : 0xfffff8800cbc7d48 ($csp)
End memory scan    : 0xfffff8800cbc9000 (Stack Base)

               rsp : 0xfffff8800cbc7d48 : 0xfffff800033e7693 : nt!WheaReportHwError+0x123
0xfffff8800cbc7d48 : 0xfffff800033e7693 : [COLOR=#0000ff]nt!WheaReportHwError+0x123[/COLOR]
0xfffff8800cbc7d78 : 0xfffff80003364f82 : [COLOR=#0000ff]nt!WheaGetErrorSource+0x12[/COLOR]
0xfffff8800cbc7da8 : 0xfffff88000e73bcf :[COLOR=#ff0000] pci!ExpressRootPortAerInterruptRoutine+0x27f[/COLOR]
0xfffff8800cbc7e08 : 0xfffff88000e735f6 : [COLOR=#ff0000]pci!ExpressRootPortInterruptRoutine+0x36[/COLOR]
0xfffff8800cbc7e48 : 0xfffff80003218ae7 : hal!HalRequestSoftwareInterrupt+0x58
0xfffff8800cbc7e78 : 0xfffff800032d647c : nt!KiInterruptDispatch+0x16c
0xfffff8800cbc7e80 : 0x0000000000000111 :  Trap @ fffff8800cbc7e80
0xfffff8800cbc7fe8 : 0xfffff80003310d99 : nt!KxFlushEntireTb+0xa5
0xfffff8800cbc8048 : 0xfffff8000330fced : nt!KeFlushTb+0x119
0xfffff8800cbc8068 : 0xfffff800034051de : nt!ExAllocatePoolWithTag+0xfe
0xfffff8800cbc8078 : 0xfffff800033032de : nt!MiInsertPrivateVad+0x86
0xfffff8800cbc8098 : 0xfffff80003303246 : nt!PsChargeProcessNonPagedPoolQuota+0x22
0xfffff8800cbc8108 : 0xfffff800032e809d : nt!MmAccessFault+0xa7d
0xfffff8800cbc8158 : 0xfffff800035d23eb : nt!NtAllocateVirtualMemory+0x75b
0xfffff8800cbc8268 : 0xfffff800032d876e : nt!KiPageFault+0x16e
0xfffff8800cbc8270 : 0x0000000000000000 :  Trap @ fffff8800cbc8270
0xfffff8800cbc83d8 : 0xfffff800035bb5d4 : nt!NtQueryObject+0xa4
0xfffff8800cbc8478 : 0xfffff80003680436 : nt!CmpCallCallBacks+0x4f6
0xfffff8800cbc84d8 : 0xfffff800032d5e70 : nt!KiServiceLinkage
*** WARNING: Unable to verify timestamp for [COLOR=#800080]aswSnx.SYS[/COLOR] << avast!
*** ERROR: Module load completed but symbols could not be loaded for aswSnx.SYS
0xfffff8800cbc87c8 : 0xfffff800032ec2fc : nt!IopFreeIrp+0x11c
0xfffff8800cbc8808 : 0xfffff800032efa7c : nt!IopCompleteRequest+0x54c
0xfffff8800cbc8828 : 0xfffff800035b3397 : nt!CmpQueryKeyData+0xef
0xfffff8800cbc8848 : 0xfffff80003405350 : nt!ExAllocatePoolWithTag+0x270
0xfffff8800cbc8888 : 0xfffff80003405350 : nt!ExAllocatePoolWithTag+0x270
0xfffff8800cbc88c8 : 0xfffff800034051de : nt!ExAllocatePoolWithTag+0xfe
*** WARNING: Unable to verify timestamp for mfehidk.sys
*** ERROR: Module load completed but symbols could not be loaded for [COLOR=#800080]mfehidk.sys[/COLOR] << McAfee
0xfffff8800cbc8978 : 0xfffff80003680436 : nt!CmpCallCallBacks+0x4f6
0xfffff8800cbc8a08 : 0xfffff800035f4f97 : nt!IopXxxControlFile+0x607
0xfffff8800cbc8a48 : 0xfffff800035aa7cd : nt!NtQueryKey+0x4eb
0xfffff8800cbc8a88 : 0xfffff800035d1198 : nt!ObpFreeObject+0x128
0xfffff8800cbc8b20 : 0xfffff8800cbc8bc8 : 0xfffff800035d1a34 : nt!ObpCloseHandle+0x94
0xfffff8800cbc8b38 : 0xfffff800035f57f6 : nt!NtDeviceIoControlFile+0x56
0xfffff8800cbc8ba0 : 0xfffff8800cbc8bc8 : 0xfffff800035d1a34 : nt!ObpCloseHandle+0x94
0xfffff8800cbc8ba8 : 0xfffff800032d98d3 : nt!KiSystemServiceCopyEnd+0x13
0xfffff8800cbc8bc8 : 0xfffff800035d1a34 : nt!ObpCloseHandle+0x94
0xfffff8800cbc8c18 : 0xfffff800032d98d3 : nt!KiSystemServiceCopyEnd+0x13

However, to conclude this breif article, there isn't much we can do with simply a Minidump. I would potentially investigate the pool resources and any other signs of resource stress on the system.

References:

https://msdn.microsoft.com/en-us/library/windows/hardware/ff557313(v=vs.85).aspx
 
Last edited:
Back
Top