[SOLVED] BSODs and DPC Latency issue.. - Windows 7 x64
- Thread starter eox08
- Start date
Hey guys, for whatever reason I had no idea you all have been chiming in here since last month, thank you all for your support. However, in the last 48 hours I have had a BSOD happen three times. I just rebooted from one of the occurrences just now and this is what i saw. I was going to include the MEMORY.dmp file but it is over a gig big :huh:
Thanks!
CLOCK_WATCHDOG_TIMEOUT (101)
This indicates that an expected clock interrupt on a secondary processor, in a multi-processor system, was not received within the allocated interval.
^^ 19 clock ticks in regards to the timeout.
fffff880009ef180 is the PRCB address of the hung processor, let's keep this address in mind.
As this matches the 3rd parameter of the bug check, processor #1 is the responsible processor. Now with the information we have here thus far, we know that processor #1 reached 19 clock ticks without responding, therefore the system crashed. Before we go further, what is a clock tick? A clock interrupt is a form of interrupt which involves counting the the cycles of the processor core, which is running a clock on the processors to keep them all in sync. A clock interrupt is handed out to all processors and then they must report in, and when one doesn't report in, you then crash.
It looks like nt!KiDeferredReadyThread+0x695 is in a tight loop waiting for a lock to likely release. In fact, save a lot of posting space, all of the cores except #1 are the same. Example:
On processor #2, we can see we have a timer expiration and then hang on nt!KiDeferredReadyThread+0x6ab.
From here we can go ahead and run !timer to view all of the DPCs that were pending:
We had 572 DPCs pending, very bad!
nt!PopCheckForIdleness fires in 100 second intervals to check whether or not a DPC has been completed. If a DPC has not been completed in a set amount of time, the bug check is called. This is a particularly difficult 0x101, probably one of the more difficult ones I've analyzed. It requires a lot of knowledge regarding thread context switching, etc. It appears though in this case that avast! had a role in blocking a network packet, therefore not allowing the network subsystem to do its job.
With that said, I dove into the modules list and noted COMODO backup + avast! which is no good. You have two active security softwares running, certainly causing a potential amount of network conflicts.
1. Remove and replace avast! and COMODO with Microsoft Security Essentials for temporary troubleshooting purposes as they are causing conflicts:
avast! removal - avast! Uninstall Utility | Download aswClear for avast! Removal
MSE - Microsoft Security Essentials - Microsoft Windows
2. AppleCharger.sys is listed and loaded which is the GIGABYTE On/Off Charge driver. See here for details - GIGABYTE ON/OFF Charge
Very troublesome software, so please uninstall ASAP!
3.
UltraMon's multi monitor drivers are very old (2008) and often cause many problems. Please uninstall.
4. wdcsam64.sys is listed and loaded which is the Western Digital SES (SCSI Enclosure Services) driver. Please remove this software ASAP as it's very troublesome and is also not necessary to the functionality of your system.
Regards,
Patrick
CLOCK_WATCHDOG_TIMEOUT (101)
This indicates that an expected clock interrupt on a secondary processor, in a multi-processor system, was not received within the allocated interval.
Code:
BugCheck 101, {[COLOR=#ff0000]19[/COLOR], 0, [COLOR=#006400]fffff880009ef180[/COLOR], 1}^^ 19 clock ticks in regards to the timeout.
fffff880009ef180 is the PRCB address of the hung processor, let's keep this address in mind.
Code:
0: kd> !prcb 1
PRCB for Processor 1 at [COLOR=#006400]fffff880009ef180[/COLOR]:
Current IRQL -- 0
Threads-- Current fffffa80116c4770 Next 0000000000000000 Idle fffff880009fa0c0
Processor Index 1 Number (0, 1) GroupSetMember 2
Interrupt Count -- 02239c56
Times -- Dpc 00000022 Interrupt 00000296
Kernel 000a41ed User 0001416eAs this matches the 3rd parameter of the bug check, processor #1 is the responsible processor. Now with the information we have here thus far, we know that processor #1 reached 19 clock ticks without responding, therefore the system crashed. Before we go further, what is a clock tick? A clock interrupt is a form of interrupt which involves counting the the cycles of the processor core, which is running a clock on the processors to keep them all in sync. A clock interrupt is handed out to all processors and then they must report in, and when one doesn't report in, you then crash.
Code:
0: kd> kv
Child-SP RetAddr : Args to Child : Call Site
fffff800`00b9c0c8 fffff800`0451ca4a : 00000000`00000101 00000000`00000019 00000000`00000000 fffff880`009ef180 : nt!KeBugCheckEx
fffff800`00b9c0d0 fffff800`044cf6f7 : 0000057f`00000000 fffff800`00000001 00000000`00002710 fffff880`00f36cec : nt! ?? ::FNODOBFM::`string'+0x4e3e
fffff800`00b9c160 fffff800`04411895 : fffff800`04437460 fffff800`00b9c310 fffff800`04437460 fffffa80`00000000 : nt!KeUpdateSystemTime+0x377
fffff800`00b9c260 fffff800`044c2113 : fffff880`03890180 fffff800`00b9c301 fffffa80`0c12ef00 fffffa80`14b22b50 : hal!HalpHpetClockInterrupt+0x8d
fffff800`00b9c290 fffff800`044cb5ae : fffffa80`0d245270 fffffa80`0d471c50 fffff800`00b9c460 fffffa80`0cfda190 : nt!KiInterruptDispatchNoLock+0x163 ([COLOR=#006400]TrapFrame @ fffff800`00b9c290[/COLOR])
fffff800`00b9c420 fffff800`044d0787 : fffffa80`15412c10 fffffa80`14b22c58 fffffa80`14b22c58 fffffa80`1541d100 : nt!KiDeferredReadyThread+0x6a7
fffff800`00b9c4a0 fffff800`044d05de : 0000001b`68142479 fffff800`00b9cb18 00000000`000b8374 fffff800`04644108 : nt!KiProcessExpiredTimerList+0x157
fffff800`00b9caf0 fffff800`044d03c7 : 00000009`17658fc8 00000009`000b8374 00000009`17658fee 00000000`00000074 : nt!KiTimerExpiration+0x1be
fffff800`00b9cb90 fffff800`044bd8ca : fffff800`04640e80 fffff800`0464ecc0 00000000`00000000 fffff880`00000000 : nt!KiRetireDpcList+0x277
fffff800`00b9cc40 00000000`00000000 : fffff800`00b9d000 fffff800`00b97000 fffff800`00b9cc00 00000000`00000000 : nt!KiIdleLoop+0x5a
Code:
0: kd> u @rip
nt!KiDeferredReadyThread+0x6a7:
fffff800`044cb5ae 498b4530 mov rax,qword ptr [r13+30h]
fffff800`044cb5b2 4885c0 test rax,rax
fffff800`044cb5b5 75e5 [COLOR=#ff0000]jne[/COLOR] [COLOR=#ff0000] nt!KiDeferredReadyThread+0x695 (fffff800`044cb59c)[/COLOR]
fffff800`044cb5b7 f0490fba6d3000 lock bts qword ptr [r13+30h],0
fffff800`044cb5be 72dc [COLOR=#ff0000] jb [/COLOR] [COLOR=#ff0000]nt!KiDeferredReadyThread+0x695 (fffff800`044cb59c)[/COLOR]
fffff800`044cb5c0 4c8d15394af8ff lea r10,[nt!KiSelectNextThread <PERF> (nt+0x0) (fffff800`04450000)]
fffff800`044cb5c7 e97dfbffff [COLOR=#ff0000]jmp[/COLOR] [COLOR=#ff0000]nt!KiDeferredReadyThread+0x239 (fffff800`044cb149)[/COLOR]
fffff800`044cb5cc 440fb6477b movzx r8d,byte ptr [rdi+7Bh]
Code:
0: kd> u fffff800`044cb5b5 fffff800`044cb5cc
nt!KiDeferredReadyThread+0x6ae:
fffff800`044cb5b5 75e5 [COLOR=#4b0082]jne[/COLOR] [COLOR=#4b0082]nt!KiDeferredReadyThread+0x695 (fffff800`044cb59c)[/COLOR]
fffff800`044cb5b7 f0490fba6d3000 lock bts qword ptr [r13+30h],0
fffff800`044cb5be 72dc [COLOR=#4b0082]jb[/COLOR] [COLOR=#4b0082]nt!KiDeferredReadyThread+0x695 (fffff800`044cb59c)[/COLOR]
fffff800`044cb5c0 4c8d15394af8ff lea r10,[nt!KiSelectNextThread <PERF> (nt+0x0) (fffff800`04450000)]
fffff800`044cb5c7 e97dfbffff [COLOR=#4b0082]jmp[/COLOR] [COLOR=#4b0082]nt!KiDeferredReadyThread+0x239 (fffff800`044cb149)[/COLOR]
fffff800`044cb5cc 440fb6477b movzx r8d,byte ptr [rdi+7Bh]It looks like nt!KiDeferredReadyThread+0x695 is in a tight loop waiting for a lock to likely release. In fact, save a lot of posting space, all of the cores except #1 are the same. Example:
Code:
2: kd> kv
Child-SP RetAddr : Args to Child : Call Site
fffff880`03794790 fffff800`044d0787 : fffffa80`1502f120 fffffa80`1502f168 fffffa80`1502f168 fffffa80`13848100 :[COLOR=#006400] nt!KiDeferredReadyThread+0x6ab[/COLOR]
fffff880`03794810 fffff800`044d05de : 0000001b`68142479 fffff880`03794e88 00000000`000b8374 fffff880`03768408 : [COLOR=#ff0000]nt!KiProcessExpiredTimerList+0x157[/COLOR]
fffff880`03794e60 fffff800`044d03c7 : fffff880`037651c3 fffffa80`000b8374 fffffa80`150f8bc0 00000000`00000074 : nt!KiTimerExpiration+0x1be
fffff880`03794f00 fffff800`044c8d15 : 00000000`00000000 fffffa80`1513e060 00000000`00000000 fffff800`045e6a90 : nt!KiRetireDpcList+0x277
fffff880`03794fb0 fffff800`044c8b2c : fffff880`03765180 fffff800`044d11b8 00000000`bb046778 00000000`fff35000 : nt!KxRetireDpcList+0x5 (TrapFrame @ fffff880`03794e70)
fffff880`0e5d7aa0 fffff800`04510b53 : fffff800`044d0f4c fffff800`044d0fb8 00000000`0eaf0034 fffff880`0e5d7b60 : nt!KiDispatchInterruptContinue
fffff880`0e5d7ad0 fffff800`044d0fb8 : 00000000`0eaf0034 fffff880`0e5d7b60 00000000`00000018 00000000`00000080 : nt!KiDpcInterruptBypass+0x13
fffff880`0e5d7ae0 00000000`b7543cbb : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : nt!KiSecondaryClockInterrupt+0x1a8 (TrapFrame @ fffff880`0e5d7ae0)
00000000`0e260a64 00000000`00000000 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : 0xb7543cbbOn processor #2, we can see we have a timer expiration and then hang on nt!KiDeferredReadyThread+0x6ab.
From here we can go ahead and run !timer to view all of the DPCs that were pending:
Code:
Total Timers: 572We had 572 DPCs pending, very bad!
Code:
145 fffff80004676140 P 6f7dad9c 0000001b [ 7/ 6/2014 18:04:46.682] [COLOR=#0000cd]nt!PopCheckForIdleness (DPC @ fffff80004676100)[/COLOR]
146 fffffa800c3008b0 685c29e9 0000001b [ 7/ 6/2014 18:04:34.719] [COLOR=#ff0000]aswNdisFlt (DPC @ fffffa800c3008f0)+b830 [/COLOR]
fffff88004c85cb8 P 685c5ac0 0000001b [ 7/ 6/2014 18:04:34.720] tunnel!TunnelTimeoutRoutine (DPC @ fffff88004c85cf8)
fffff88004b17760 685c5ac0 0000001b [ 7/ 6/2014 18:04:34.720] afd!AfdTimerWheelHandler (DPC @ fffff88004b17720)
149 fffffa800d4b0278 431748ae 00000131 [ 7/ 8/2014 03:13:32.282] [COLOR=#800080]ndis!ndisMWakeUpDpcX (DPC @ fffffa800d4b02b8) [/COLOR]
fffffa800d4b6278 431748ae 00000131 [ 7/ 8/2014 03:13:32.282] [COLOR=#800080]ndis!ndisMWakeUpDpcX (DPC @ fffffa800d4b62b8) [/COLOR]
fffffa800d4bc278 431748ae 00000131 [ 7/ 8/2014 03:13:32.282] [COLOR=#800080]ndis!ndisMWakeUpDpcX (DPC @ fffffa800d4bc2b8) [/COLOR]
fffffa800d4c7278 431748ae 00000131 [ 7/ 8/2014 03:13:32.282] [COLOR=#800080]ndis!ndisMWakeUpDpcX (DPC @ fffffa800d4c72b8) [/COLOR]
152 fffffa8013c061f0 686b1d20 0000001b [ 7/ 6/2014 18:04:34.817] afd!AfdTimeoutPoll (DPC @ fffffa8013c061b0)
154 fffffa80154aac10 686f8a03 0000001b [ 7/ 6/2014 18:04:34.846] thread fffffa80154aab50
fffffa800e340120 85004f01 0000001b [ 7/ 6/2014 18:05:22.771] thread fffffa800e340060
155 fffffa800bb515c0 P 6872d5f8 0000001b [ 7/ 6/2014 18:04:34.867] [COLOR=#800080]ndis!ndisMTimerObjectDpc (DPC @ fffffa800bb51600) [/COLOR]
157 fffffa800b6b37f0 6f9b4e54 0000001b [ 7/ 6/2014 18:04:46.877] thread fffffa800b6b3730
158 fffff880019d6500 0639e6a3 0000002c [ 7/ 6/2014 20:03:31.522] cng!seedFileDpcRoutine (DPC @ fffff880019d64c0)
161 fffffa8015a75120 68800501 0000001b [ 7/ 6/2014 18:04:34.954] thread fffffa8015a75060
fffffa800d4ca278 6fa44d23 0000001b [ 7/ 6/2014 18:04:46.935] [COLOR=#800080]ndis!ndisMWakeUpDpcX (DPC @ fffffa800d4ca2b8) [/COLOR]
fffffa8015161120 76c7ea60 0000001b [ 7/ 6/2014 18:04:58.913] thread fffffa8015161060
162 fffffa800cf60260 P 6d44c4ba 0000001b [ 7/ 6/2014 18:04:42.954] [COLOR=#ff0000]aswSP (DPC @ fffffa800cf60220)+30280 [/COLOR]nt!PopCheckForIdleness fires in 100 second intervals to check whether or not a DPC has been completed. If a DPC has not been completed in a set amount of time, the bug check is called. This is a particularly difficult 0x101, probably one of the more difficult ones I've analyzed. It requires a lot of knowledge regarding thread context switching, etc. It appears though in this case that avast! had a role in blocking a network packet, therefore not allowing the network subsystem to do its job.
With that said, I dove into the modules list and noted COMODO backup + avast! which is no good. You have two active security softwares running, certainly causing a potential amount of network conflicts.
1. Remove and replace avast! and COMODO with Microsoft Security Essentials for temporary troubleshooting purposes as they are causing conflicts:
avast! removal - avast! Uninstall Utility | Download aswClear for avast! Removal
MSE - Microsoft Security Essentials - Microsoft Windows
2. AppleCharger.sys is listed and loaded which is the GIGABYTE On/Off Charge driver. See here for details - GIGABYTE ON/OFF Charge
Very troublesome software, so please uninstall ASAP!
3.
Code:
0: kd> lmvm UltraMonUtility
start end module name
fffff880`0d4f9000 fffff880`0d502000 UltraMonUtility (deferred)
Image path: \??\C:\Program Files (x86)\Common Files\Realtime Soft\UltraMonMirrorDrv\x64\UltraMonUtility.sys
Image name: UltraMonUtility.sys
Timestamp: Thu Nov 13 20:10:30 [COLOR=#ff0000]2008[/COLOR]UltraMon's multi monitor drivers are very old (2008) and often cause many problems. Please uninstall.
4. wdcsam64.sys is listed and loaded which is the Western Digital SES (SCSI Enclosure Services) driver. Please remove this software ASAP as it's very troublesome and is also not necessary to the functionality of your system.
Regards,
Patrick
Okay, I went into the Windows -> System32 -> driver and found the two (Appcharger.sys & wdscam64.sys) files (or .dll's) and changed them to wdcsam64.sys.old and AppleCharger.sys.old then I was able to delete them. Restarting computer now to see how everything reacts.
To save a ton of time and posting space:
asmtxhci.sys appears to be holding a lock and not releasing it, which is the Asmedia USB 3.0 driver.
Ouch.... dated from 2011. No update via your motherboard website, bummer.
As of this crash dump, all of avast!'s drivers are still listed and loaded. Did you remove and replace it as I advised?
Also, there's also AsrAppCharger.sys which is the AsRock App Charger driver. Rename this as well as you did the other drivers.
Regards,
Patrick
Code:
0: kd> k
Child-SP RetAddr Call Site
fffff800`05f1a748 fffff800`044d4a4a nt!KeBugCheckEx
fffff800`05f1a750 fffff800`044876f7 nt! ?? ::FNODOBFM::`string'+0x4e3e
fffff800`05f1a7e0 fffff800`049f7895 nt!KeUpdateSystemTime+0x377
fffff800`05f1a8e0 fffff800`0447a113 hal!HalpHpetClockInterrupt+0x8d
fffff800`05f1a910 fffff800`04464c60 [COLOR=#006400]nt!KiInterruptDispatchNoLock+0x163[/COLOR]
fffff800`05f1aaa0 fffff800`04486d19 [COLOR=#0000cd]nt!KxWaitForSpinLockAndAcquire+0x20[/COLOR]
fffff800`05f1aad0 fffff880`102e0f73 [COLOR=#4b0082]nt!KeAcquireSpinLockRaiseToDpc+0x89[/COLOR]
fffff800`05f1ab20 fffff880`102dee76 [COLOR=#ff0000]asmtxhci+0x49f73[/COLOR]
fffff800`05f1ab60 fffff800`0448830c [COLOR=#ff0000]asmtxhci+0x47e76[/COLOR]
fffff800`05f1ab90 fffff800`044758ca nt!KiRetireDpcList+0x1bc
fffff800`05f1ac40 00000000`00000000 nt!KiIdleLoop+0x5a
Code:
0: kd> u @rip
nt!KxWaitForSpinLockAndAcquire+0x20:
fffff800`04464c60 488b0f mov rcx,qword ptr [rdi]
fffff800`04464c63 4885c9 test rcx,rcx
fffff800`04464c66 75e8 [COLOR=#ff0000]jne[/COLOR] [COLOR=#ff0000]nt!KxWaitForSpinLockAndAcquire+0x10 (fffff800`04464c50)[/COLOR]
fffff800`04464c68 f0480fba2f00 lock bts qword ptr [rdi],0
fffff800`04464c6e 72e0 [COLOR=#4b0082]jb[/COLOR] [COLOR=#4b0082]nt!KxWaitForSpinLockAndAcquire+0x10 (fffff800`04464c50)[/COLOR]
fffff800`04464c70 8bc3 mov eax,ebx
fffff800`04464c72 488b5c2430 mov rbx,qword ptr [rsp+30h]
fffff800`04464c77 4883c420 add rsp,20hasmtxhci.sys appears to be holding a lock and not releasing it, which is the Asmedia USB 3.0 driver.
Code:
0: kd> lmvm asmtxhci
start end module name
fffff880`10297000 fffff880`102fa000 asmtxhci (no symbols)
Loaded symbol image file: asmtxhci.sys
Image path: \SystemRoot\system32\DRIVERS\asmtxhci.sys
Image name: asmtxhci.sys
Timestamp: Fri Mar 04 02:54:59 [COLOR=#ff0000]2011[/COLOR]Ouch.... dated from 2011. No update via your motherboard website, bummer.
As of this crash dump, all of avast!'s drivers are still listed and loaded. Did you remove and replace it as I advised?
Also, there's also AsrAppCharger.sys which is the AsRock App Charger driver. Rename this as well as you did the other drivers.
Regards,
Patrick
Has Sysnative Forums helped you? Please consider donating to help us support the site!