[SOLVED] Massive kernel timer latency

R

radynj

Member
Joined
Sep 10, 2016
Posts
6
Hi everyone,

First time poster here. I've been having issues with a strange stuttering in games even when I'm getting 60 FPS+ and I can't seem to pin it down. After running LatencyMon on my gaming system I notice really high kernel timer latency and very high pagefaults. Possibly these are to blame? I'm currently on the newest BIOS for my motherboard and have all the newest drivers that I can find. My system isn't overclocked either.

System specs:

Intel i7 4790K
Gigabyte Z97mx-Gaming 5
4x4GB DDR3 2400MHz RAM
Nvidia GTX 1070 8GB
RAID 0 256GB Samsung 840 Pro SSDs + another 250 840 SSD
4TB Seagate hard-drive


I would be super appreciative for any assistance.

Thanks!
latency2.png

latency3.png

Report from LatencyMon:
_________________________________________________________________________________________________________
CONCLUSION
_________________________________________________________________________________________________________
Your system appears to be suitable for handling real-time audio and other tasks without dropouts.
LatencyMon has been analyzing your system for 0:03:19 (h:mm:ss) on all processors.


_________________________________________________________________________________________________________
SYSTEM INFORMATION
_________________________________________________________________________________________________________
Computer name: NOTSOBIGBOX
OS version: Windows 8 , 6.2, build: 9200 (x64)
Hardware: Z97MX-Gaming 5, Gigabyte Technology Co., Ltd.
CPU: GenuineIntel Intel(R) Core(TM) i7-4790K CPU @ 4.00GHz
Logical processors: 8
Processor groups: 1
RAM: 16311 MB total


_________________________________________________________________________________________________________
CPU SPEED
_________________________________________________________________________________________________________
Reported CPU speed: 40 MHz
Measured CPU speed: 1 MHz (approx.)

Note: reported execution times may be calculated based on a fixed reported CPU speed. Disable variable speed settings like Intel Speed Step and AMD Cool N Quiet in the BIOS setup for more accurate results.

WARNING: the CPU speed that was measured is only a fraction of the CPU speed reported. Your CPUs may be throttled back due to variable speed settings and thermal issues. It is suggested that you run a utility which reports your actual CPU frequency and temperature.



_________________________________________________________________________________________________________
MEASURED KERNEL TIMER LATENCIES
_________________________________________________________________________________________________________
This value represents the maximum measured latency of a perodically scheduled kernel timer.

Highest measured kernel timer latency (µs): 12630.265533


_________________________________________________________________________________________________________
REPORTED ISRs
_________________________________________________________________________________________________________
Interrupt service routines are routines installed by the OS and device drivers that execute in response to a hardware interrupt signal.

Highest ISR routine execution time (µs): 122.9970
Driver with highest ISR routine execution time: HDAudBus.sys - High Definition Audio Bus Driver, Microsoft Corporation

Highest reported total ISR routine time (%): 0.025685
Driver with highest ISR total time: dxgkrnl.sys - DirectX Graphics Kernel, Microsoft Corporation

Total time spent in ISRs (%) 0.032483

ISR count (execution time <250 µs): 56784
ISR count (execution time 250-500 µs): 0
ISR count (execution time 500-999 µs): 0
ISR count (execution time 1000-1999 µs): 0
ISR count (execution time 2000-3999 µs): 0
ISR count (execution time >=4000 µs): 0


_________________________________________________________________________________________________________
REPORTED DPCs
_________________________________________________________________________________________________________
DPC routines are part of the interrupt servicing dispatch mechanism and disable the possibility for a process to utilize the CPU while it is interrupted until the DPC has finished execution.

Highest DPC routine execution time (µs): 507.050750
Driver with highest DPC routine execution time: nvlddmkm.sys - NVIDIA Windows Kernel Mode Driver, Version 372.70 , NVIDIA Corporation

Highest reported total DPC routine time (%): 0.018641
Driver with highest DPC total execution time: nvlddmkm.sys - NVIDIA Windows Kernel Mode Driver, Version 372.70 , NVIDIA Corporation

Total time spent in DPCs (%) 0.040

DPC count (execution time <250 µs): 129290
DPC count (execution time 250-500 µs): 0
DPC count (execution time 500-999 µs): 5
DPC count (execution time 1000-1999 µs): 0
DPC count (execution time 2000-3999 µs): 0
DPC count (execution time >=4000 µs): 0


_________________________________________________________________________________________________________
REPORTED HARD PAGEFAULTS
_________________________________________________________________________________________________________
Hard pagefaults are events that get triggered by making use of virtual memory that is not resident in RAM but backed by a memory mapped file on disk. The process of resolving the hard pagefault requires reading in the memory from disk while the process is interrupted and blocked from execution.

NOTE: some processes were hit by hard pagefaults. If these were programs producing audio, they are likely to interrupt the audio stream resulting in dropouts, clicks and pops. Check the Processes tab to see which programs were hit.

Process with highest pagefault count: firefox.exe

Total number of hard pagefaults 2055
Hard pagefault count of hardest hit process: 1024
Highest hard pagefault resolution time (µs): 15122.71550
Total time spent in hard pagefaults (%): 0.015589
Number of processes hit: 14


_________________________________________________________________________________________________________
PER CPU DATA
_________________________________________________________________________________________________________
CPU 0 Interrupt cycle time (s): 1.124328
CPU 0 ISR highest execution time (µs): 122.9970
CPU 0 ISR total execution time (s): 0.516234
CPU 0 ISR count: 56626
CPU 0 DPC highest execution time (µs): 507.050750
CPU 0 DPC total execution time (s): 0.544324
CPU 0 DPC count: 78103
_________________________________________________________________________________________________________
CPU 1 Interrupt cycle time (s): 0.034021
CPU 1 ISR highest execution time (µs): 55.74550
CPU 1 ISR total execution time (s): 0.000931
CPU 1 ISR count: 135
CPU 1 DPC highest execution time (µs): 61.612250
CPU 1 DPC total execution time (s): 0.009766
CPU 1 DPC count: 4175
_________________________________________________________________________________________________________
CPU 2 Interrupt cycle time (s): 0.101046
CPU 2 ISR highest execution time (µs): 47.060750
CPU 2 ISR total execution time (s): 0.000091
CPU 2 ISR count: 23
CPU 2 DPC highest execution time (µs): 99.8440
CPU 2 DPC total execution time (s): 0.044993
CPU 2 DPC count: 26295
_________________________________________________________________________________________________________
CPU 3 Interrupt cycle time (s): 0.012822
CPU 3 ISR highest execution time (µs): 0.0
CPU 3 ISR total execution time (s): 0.0
CPU 3 ISR count: 0
CPU 3 DPC highest execution time (µs): 17.47850
CPU 3 DPC total execution time (s): 0.003448
CPU 3 DPC count: 1737
_________________________________________________________________________________________________________
CPU 4 Interrupt cycle time (s): 0.044481
CPU 4 ISR highest execution time (µs): 0.0
CPU 4 ISR total execution time (s): 0.0
CPU 4 ISR count: 0
CPU 4 DPC highest execution time (µs): 54.842250
CPU 4 DPC total execution time (s): 0.014721
CPU 4 DPC count: 8391
_________________________________________________________________________________________________________
CPU 5 Interrupt cycle time (s): 0.015104
CPU 5 ISR highest execution time (µs): 0.0
CPU 5 ISR total execution time (s): 0.0
CPU 5 ISR count: 0
CPU 5 DPC highest execution time (µs): 22.4830
CPU 5 DPC total execution time (s): 0.003010
CPU 5 DPC count: 1622
_________________________________________________________________________________________________________
CPU 6 Interrupt cycle time (s): 0.036827
CPU 6 ISR highest execution time (µs): 0.0
CPU 6 ISR total execution time (s): 0.0
CPU 6 ISR count: 0
CPU 6 DPC highest execution time (µs): 63.4170
CPU 6 DPC total execution time (s): 0.012659
CPU 6 DPC count: 6738
_________________________________________________________________________________________________________
CPU 7 Interrupt cycle time (s): 0.026339
CPU 7 ISR highest execution time (µs): 0.0
CPU 7 ISR total execution time (s): 0.0
CPU 7 ISR count: 0
CPU 7 DPC highest execution time (µs): 19.436250
CPU 7 DPC total execution time (s): 0.004031
CPU 7 DPC count: 2234
_________________________________________________________________________________________________________
 
Hi radynj ... and welcome to the forums ...


When you notice the stuttering, are both the audio and video acting up? (or just one of the two?) The support website for your board has an interesting note in it's drivers download area for Windows 10 ... for some reason they prefer the Realtek drivers be installed first, and then the Creative X-Fi drivers after. [A bit odd, since the Creative drivers on their site are older than the Realtek... but they must have put that note there for a reason]. If you want to try an experiment, you could try removing both drivers, then manually install the Realtek, and the manually installing the Creative X-Fi drivers. Who knows? [... sprinkle a little magic dust around, too, maybe? ...]

I'll paste in a little list of adjustments that might help, especially since your main issues look to be the pagefaults. Background activity can be a factor, and sometimes at least a little relief can be had by limiting that. Have a look, too, at the mention of the "port blocking' feature in software firewalls - these can cause stuttering issues on some machines. If you have a hardware firewall built-into your router/gateway device, it likely has a port-scan blocking feature, so that turning off that feature in your software firewall doesn't compromise your overall security.

Here's the list:

Some things to try for limiting audio/video glitches due to high latency issues and high hard pagefault issues


SSDs
Since SSDs are dropping in price and growing in size, I've lately been recommending that the worst-offending audio-pop suffering programs/games be installed onto the SSD. Windows doesn't take up all that much room, and neither do a few games (you can still keep "saved games" and other non-demanding software installed to a larger "spinning" hard drive, along with your photo, music, and video collections).


SATA ports
Some users who haven't seen any improvement by means of updating drivers have had success by altering their SATA port usage. A recent thread here concerns a system where the audio stuttering was helped by unplugging a DVD SATA drive (...and our tech Cluberti was nice enough to help explain why) ... The last few posts are where you will see mention of the SATA ports (and the nature of USB polling & the effect on latencies) .... Here's a link to that thread:
https://www.sysnative.com/forums/wi...efaults-fresh-install-usbport-sys-others.html


Hard Pagefaults
For the high hard pagefaults, it can help to:
1) Limit startup processes that you don't need
2) Limit Windows-Updates file-sharing (in Settings > Updates and security > Windows Update > Advanced Options > Choose how updates are delivered > turn off "updates from more than one place" - that will limit unnecessary background network activity)
3) Limit the amount of "telemetry" (data sent to Microsoft ... choose the "Basic" option in Settings > Privacy > Feedback & Diagnostics ... and you can try "Never" as your "Feedback Frequency" while testing your latency issues)
4) Limit any Notifications that you don't need or want (Settings > System > Notifications & actions ...[especially turn off Windows "tips"]
5) Turn off "Live" tiles on your Start Menu (to limit unnecessary background updating of any information you don't want or need)
6) Uninstall any Windows "Universal" apps that you don't want or need (Settings > System > Apps & features -- click the app you wish to uninstall) ... and for those that you don't want, but can't uninstall (some of the built-in apps) - you can still keep them from starting unwanted services (in Control Panel > Administrative Tools > Services) or startup modules (in Task Manager > Startup tab)


Drivers
Hunt Windows 10 compatible drivers (this can be an ongoing process). Try generics if the manufacturer supplied drivers cause issues. Try older drivers when the new cause issues. Try and try again. Drivers are sometimes the culprits for high latencies and interrupts.


Nvidia drivers - seem to be having issues on Z170 boards, especially when paired with Realtek onboard audio. When updating to the latest Nvidia driver, try installing it using the "custom" option, and the "clean install" option. You might also see if your want or need all of the background Nvidia services that come with the latest drivers, many as part of the "Nvdia GeForce Experience" - which you can choose to install or not when you choose the "custom" install method. Here's a guide that has some details on the background services installed along with the Nvidia drivers, it might help you decide if you can do without some of them:
Disable NVIDIA Streamer Service and other NVIDIA processes - gHacks Tech News


ASMedia USB 3.1 drivers .... have proved to be an issue for some users, especially for ASRock/ASUS Z170 motherboard owners. Hunt newer versions, try old versions, try generics. Still dicey as of July 2016....


USB device drivers ... USB host controllers, USB gaming periferals (mice, keyboards, headsets...), musical instrument interfaces, wireless dongles - all these can prove to be sources of audio issues.


Antivirus
Some users find that certain modules of their antivirus program can have a rough affect on their audio quality. For some, it is especially noticeable when using the third-party firewall. You can try reverting to using just the Windows firewall, or to turn off a feature in the third-party firewall that blocks port scans (since most routers/modems/gateways have built-in port scanning blocking, it doesn't necessarily lower your defenses, since your hardware provides the service). This issue has shown up for users of BitDefender, Norton, and Avast ... it likely can occur for others as well. The issue also doesn't seem to be "universal" - it only seems to affect a certain blend of hardware/software. (I know users of Avast, BitDefender, & Norton who don't experience any issues at all).


Virtual Memory & a large set range
Since some computers experience the audio/video glitching during gaming, and Latency Monitor graphs often point to high hard pagefaults as a contributing factor (as in the pagefault tips above), I like to recommend testing things with the Virtual Memory settings for your computer manually set to a generous range (generally, with the lowest value the amount of your total installed physical system memory, and the highest value twice that number). This is an overly large range = but it can prove useful for testing. [after testing, the upper range could be lowered to one and a half times the physical memory total]... The reason for the large set range is that some games are infamous for memory leaks, and the large range might give you a better chance of surviving the memory trouble caused by the game with the memory leak.
_________________________

We shouldn't have to work this hard to get good performance from computers with top-of-the-line components, so let's hope these tweaks and adjustments aren't going to be needed for long. I hope that the underlying disfunctional relationships between drivers / operating system components / software packages that are causing the problems now will eventually be discovered and fixed over the next few months or so.

Let us know how it goes.
 
P.S. .... since firefox was the largest source of pagefaults in your last test: can we guess that you are gaming online? [Or does the stutter happen when watching something simple, like a YouTube video?] ...
 
This just when gaming. Mostly noticeable in GTAV and Rainbow Six Siege.

Still working away on trying different drivers. So far with newest IRST drivers I've managed to make everything worse so I'm currently rolling back most of my chipset drivers.

latency5.png

_________________________________________________________________________________________________________
CONCLUSION
_________________________________________________________________________________________________________
Your system appears to be suitable for handling real-time audio and other tasks without dropouts.
LatencyMon has been analyzing your system for 0:44:01 (h:mm:ss) on all processors.


_________________________________________________________________________________________________________
SYSTEM INFORMATION
_________________________________________________________________________________________________________
Computer name: NOTSOBIGBOX
OS version: Windows 8 , 6.2, build: 9200 (x64)
Hardware: Z97MX-Gaming 5, Gigabyte Technology Co., Ltd.
CPU: GenuineIntel Intel(R) Core(TM) i7-4790K CPU @ 4.00GHz
Logical processors: 8
Processor groups: 1
RAM: 16311 MB total


_________________________________________________________________________________________________________
CPU SPEED
_________________________________________________________________________________________________________
Reported CPU speed: 40 MHz
Measured CPU speed: 1 MHz (approx.)

Note: reported execution times may be calculated based on a fixed reported CPU speed. Disable variable speed settings like Intel Speed Step and AMD Cool N Quiet in the BIOS setup for more accurate results.

WARNING: the CPU speed that was measured is only a fraction of the CPU speed reported. Your CPUs may be throttled back due to variable speed settings and thermal issues. It is suggested that you run a utility which reports your actual CPU frequency and temperature.



_________________________________________________________________________________________________________
MEASURED KERNEL TIMER LATENCIES
_________________________________________________________________________________________________________
This value represents the maximum measured latency of a perodically scheduled kernel timer.

Highest measured kernel timer latency (µs): 26651.910823


_________________________________________________________________________________________________________
REPORTED ISRs
_________________________________________________________________________________________________________
Interrupt service routines are routines installed by the OS and device drivers that execute in response to a hardware interrupt signal.

Highest ISR routine execution time (µs): 192.7140
Driver with highest ISR routine execution time: dxgkrnl.sys - DirectX Graphics Kernel, Microsoft Corporation

Highest reported total ISR routine time (%): 0.039078
Driver with highest ISR total time: dxgkrnl.sys - DirectX Graphics Kernel, Microsoft Corporation

Total time spent in ISRs (%) 0.050858

ISR count (execution time <250 µs): 1615136
ISR count (execution time 250-500 µs): 0
ISR count (execution time 500-999 µs): 0
ISR count (execution time 1000-1999 µs): 0
ISR count (execution time 2000-3999 µs): 0
ISR count (execution time >=4000 µs): 0


_________________________________________________________________________________________________________
REPORTED DPCs
_________________________________________________________________________________________________________
DPC routines are part of the interrupt servicing dispatch mechanism and disable the possibility for a process to utilize the CPU while it is interrupted until the DPC has finished execution.

Highest DPC routine execution time (µs): 510.953750
Driver with highest DPC routine execution time: nvlddmkm.sys - NVIDIA Windows Kernel Mode Driver, Version 372.70 , NVIDIA Corporation

Highest reported total DPC routine time (%): 0.028024
Driver with highest DPC total execution time: nvlddmkm.sys - NVIDIA Windows Kernel Mode Driver, Version 372.70 , NVIDIA Corporation

Total time spent in DPCs (%) 0.101580

DPC count (execution time <250 µs): 4704033
DPC count (execution time 250-500 µs): 0
DPC count (execution time 500-999 µs): 10
DPC count (execution time 1000-1999 µs): 0
DPC count (execution time 2000-3999 µs): 0
DPC count (execution time >=4000 µs): 0


_________________________________________________________________________________________________________
REPORTED HARD PAGEFAULTS
_________________________________________________________________________________________________________
Hard pagefaults are events that get triggered by making use of virtual memory that is not resident in RAM but backed by a memory mapped file on disk. The process of resolving the hard pagefault requires reading in the memory from disk while the process is interrupted and blocked from execution.

NOTE: some processes were hit by hard pagefaults. If these were programs producing audio, they are likely to interrupt the audio stream resulting in dropouts, clicks and pops. Check the Processes tab to see which programs were hit.

Process with highest pagefault count: explorer.exe

Total number of hard pagefaults 96218
Hard pagefault count of hardest hit process: 41420
Highest hard pagefault resolution time (µs): 648951.92150
Total time spent in hard pagefaults (%): 0.656770
Number of processes hit: 47


_________________________________________________________________________________________________________
PER CPU DATA
_________________________________________________________________________________________________________
CPU 0 Interrupt cycle time (s): 33.033701
CPU 0 ISR highest execution time (µs): 192.7140
CPU 0 ISR total execution time (s): 10.540499
CPU 0 ISR count: 1586101
CPU 0 DPC highest execution time (µs): 510.953750
CPU 0 DPC total execution time (s): 18.295508
CPU 0 DPC count: 3254961
_________________________________________________________________________________________________________
CPU 1 Interrupt cycle time (s): 1.765454
CPU 1 ISR highest execution time (µs): 139.684750
CPU 1 ISR total execution time (s): 0.194162
CPU 1 ISR count: 26897
CPU 1 DPC highest execution time (µs): 283.143750
CPU 1 DPC total execution time (s): 0.526687
CPU 1 DPC count: 167415
_________________________________________________________________________________________________________
CPU 2 Interrupt cycle time (s): 2.201339
CPU 2 ISR highest execution time (µs): 120.450
CPU 2 ISR total execution time (s): 0.010965
CPU 2 ISR count: 2077
CPU 2 DPC highest execution time (µs): 167.1740
CPU 2 DPC total execution time (s): 0.682565
CPU 2 DPC count: 324715
_________________________________________________________________________________________________________
CPU 3 Interrupt cycle time (s): 0.934127
CPU 3 ISR highest execution time (µs): 6.40450
CPU 3 ISR total execution time (s): 0.000194
CPU 3 ISR count: 61
CPU 3 DPC highest execution time (µs): 104.0630
CPU 3 DPC total execution time (s): 0.312465
CPU 3 DPC count: 149654
_________________________________________________________________________________________________________
CPU 4 Interrupt cycle time (s): 2.57830
CPU 4 ISR highest execution time (µs): 0.0
CPU 4 ISR total execution time (s): 0.0
CPU 4 ISR count: 0
CPU 4 DPC highest execution time (µs): 121.17850
CPU 4 DPC total execution time (s): 0.726474
CPU 4 DPC count: 397148
_________________________________________________________________________________________________________
CPU 5 Interrupt cycle time (s): 0.596551
CPU 5 ISR highest execution time (µs): 0.0
CPU 5 ISR total execution time (s): 0.0
CPU 5 ISR count: 0
CPU 5 DPC highest execution time (µs): 84.060750
CPU 5 DPC total execution time (s): 0.153484
CPU 5 DPC count: 68178
_________________________________________________________________________________________________________
CPU 6 Interrupt cycle time (s): 1.864465
CPU 6 ISR highest execution time (µs): 0.0
CPU 6 ISR total execution time (s): 0.0
CPU 6 ISR count: 0
CPU 6 DPC highest execution time (µs): 114.32150
CPU 6 DPC total execution time (s): 0.586527
CPU 6 DPC count: 263550
_________________________________________________________________________________________________________
CPU 7 Interrupt cycle time (s): 0.748451
CPU 7 ISR highest execution time (µs): 0.0
CPU 7 ISR total execution time (s): 0.0
CPU 7 ISR count: 0
CPU 7 DPC highest execution time (µs): 107.73850
CPU 7 DPC total execution time (s): 0.179274
CPU 7 DPC count: 78422
_________________________________________________________________________________________________________
 
So far the best improvement I've had for kernel timer latency is by reducing the audio output bitrate down to 96,000Hz from 192,000. That's gotten rid of the random huge latency spikes. I still have massive pagefault resolution time and that seems to stem from IRST drivers. I'm still playing with that.

latency6.png
 
The nvidia&direct X kernel issues are most certainly a pascal driver issue. I have replicated it across 2 motherboards and 3 OS's. It of course disappears completely with iGFX.
 
Warning

Before you proceed with the following, answer these questions:
Are you still under warranty?
Does your warranty allow you to open up the machine to check hardware?
Do you know HOW to open up your machine and its components (without doing anything risky for you and for your home/PC/pc components)?
If you are unsure of the answers to these questions, contact your system manufacturer (or the component manufacturer).
WARNING: The steps that follow can void your warranty!!!

Remove the dust from inside your pc, especially from fans and air slits.
Check if the fans are working.
Apply a new layer of thermal past on the cpu.
Clean the slot and the gpu card connector.
Try to improve the fan settings and/or their position inside the case (or get better fans, if they don't suffice).
 
Hi again all -


Hope you are all having a good afternoon as we approach the cool breezes of autumn (impatiently waiting for them here in Southern California!)

The Intel Rapid Storage Technology drivers can be uninstalled. From what I've seen earlier (though I'm relying on my not-as-reliable-as-before memory) some of the settings remain even when the drivers are uninstalled... which usually means that your drives still benefit from those helpful settings, while not suffering the odd lags that the drivers can cause on some systems. If you create a Restore Point before removing them, that allows you to "undo" things if all hell breaks loose - or if it simply doesn't help. [You could also do a driver "roll back" in Device Manager].

Your system is quite interesting ... almost no trouble at all for the DPC counts and ISR interrupts.... it's nearly all high pagefaults. If you haven't turned off silly background noise from Windows "notifications", live tiles, nonsensical background activity for everything from Weather, to Sports, to cookie recipes ... try that too [Visit each section in Start > Settings .... background nonsense is hidden three or four levels deep in some of those sections. Turn off all the unbelievably nonsensical stuff].

Hmm ... might want to check in on your "Indexing Options" in Control Panel, so that you aren't needlessly indexing SSDs that won't benefit much from that.

Seems like you should have a good chance of fixing this one. [High DPC and ISR counts have proven very hard to solve... glad you don't have those]
 
After formatting my system and going back to Windows 7, this is what my system is doing. Very puzzling.
latency1.png
 
After performing a billion windows updates (god windows 7 is old) my latency issues seem to be resolved. Seems to just be a bad case of Windows 10 drivers not being ready.

latency2.png

Thanks for all your help everyone!
 
Glad things are better - good work!

Windows 7 will continue to get Security Updates (and any other "Critical" updates) until mid-January 2020, so you should be OK running 7 till then. And by then Windows 10 should behave better (let's hope!), and you can upgrade to Windows 10 then at no extra cost (you already have "digital entitlement" to Windows 10 on that hardware because you already upgraded to 10 on it (your entitlement is stored indefinitely on the Windows Activation servers)..

Congrats again.
 

Has Sysnative Forums helped you? Please consider donating to help us support the site!

Back
Top