[SOLVED] Yet another high DPC latency thread (Wdf0100.sys, tcpip.sys). After some hours the network slows down. FIX: uninstall malwarebytes (until they fix it)

[ctp9]

As the title suggests, things aren't going so well at the moment. The most recent Cumulative Windows Updates (dating back to Feb 12, 2020 and Feb 29, 2020) have introduced audible crackle/pop/tears into my audio streams. I'm using the Surface Pro 6 tablet for Pro Audio (music production/DAW, low latency/ASIO, etc.)

Things get better once the device is rebooted, but quickly degrade after the machine has been idling for some time (I wonder why?).

I have uploaded my latency trace, msinfo, dxdiag, and system specs:

>> Surface Pro Specs and Trace <<


LatencyMon shows the following:

System Manufacturer?
Microsoft Surface Pro 6

Laptop or Desktop?
Laptop/Tablet

Exact model number
Surface_Pro_6_1796_Consumer

OS?
Windows 10 Version 1909 (OS Build 18363.693)

x86 (32-bit) or x64 (64-bit)?
64-bit

Service Pack?
Feature update to Windows 10, version 1909

What was original installed OS on system?
Windows 10 1803

Is the OS an OEM version or full retail version?
OEM

Age of system? (hardware)
7 months

Age of OS installation?
Latest Windows Cumulative Update

Have you re-installed the OS?
No

CPU
Intel(R) Core(TM) i7-8650U CPU @ 1.90GHz

RAM
16 GB LPDDR3 (2 x 8 GB, Channel A and Channel B)

Video Card
Intel(R) UHD Graphics 620

Motherboard
N/A

Power Supply - brand & wattage
Surface Dock 90W

Is driver verifier enabled or disabled?
Disabled (not 100% sure though)

What security software are you using?
Windows Defender (AV) with Malwarebytes Premium (antimalware)

Are you using proxy, vpn, ipfilters or similar software?
dnscrypt-proxy

Are you using Disk Image tools?
No

Are you currently under/overclocking?
Undervolting (slightly, to reduce heat)

Are there overclocking software installed on your system?
ThrottleStop 8.70.6 (latest)


I should add that all unnecessary devices in Device Manager are disabled (Wi-Fi Controller, ACPI-Compliant Control Method Battery, HID Sensor Collection, Card Reader, etc.)


Thank you for your support!

 

[x BlueRobot]

Have you tried removing the under-clocking to see if it makes a difference?

 

[ctp9]

Have you tried removing the under-clocking to see if it makes a difference?

To rule this out as a possible culprit - I have just tried your suggestion by removing undervolting altogether. This resulted in an instanteneous improvement, but still not as good as prior to update KB4535996.

Mind you, the system was running perfectly fine prior to the most recent cumulative update, KB4535996. I never had drop outs with my CPU and GPU being undervolted.

I have read that this particular update is actually responsible for a host of problems that people are experiencing now... including audio, video, and frame rate drop outs. As some people have pointed out:

With that said - I'm just wondering if anything is apparent from the xperf trace that I uploaded? If worse comes to worse, I'll just uninstall this particular update and will defer updates for a while.

 

[ctp9]

Just want to give an update and confirm that Driver Verifier is disabled. The issue still persists.

 

[xilolee]

Try to update the intel graphics driver: Downloads for Intel® UHD Graphics 620

 

[ctp9]

I was already on the latest graphics driver version when the latency issues started occurring.

 

[xilolee]

Create a new system restore point and make a system image with your preferred tool.
Then try this: Basic clean-up, check-up and repair of Windows (W10, W8/8.1, W7) - skip points 4,5 and 7.
If the problem still persists, perform points 4, 5 and 7 (reporting all their results here, obviusly...).
If all results will be clean, get a new trace (when something bad happens...).

 

[ctp9]

I apologize for late reply - I was waiting it out to see if the latest Windows Update would fix it. I have also upgraded (clean install) to the latest Intel graphics driver that came out a few days ago. Unfortunately, the issue still persists.

I have followed all of the following steps as suggested:

Create a new system restore point and make a system image with your preferred tool.
Then try this: Basic clean-up, check-up and repair of Windows (W10, W8/8.1, W7) - skip points 4,5 and 7.
If the problem still persists, perform points 4, 5 and 7 (reporting all their results here, obviusly...).
If all results will be clean, get a new trace (when something bad happens...).

DISM and SFC both reported a few minor errors but fixed them.

I have also uploaded a new trace (updated link in first post).

Other than that... I don't know what else can I do. DPC latencies are still caused mainly by Wdf0100.sys, tcpip.sys and NDIS.sys

Thanks for your help, xilo.

 

[ctp9]

Another thing I should mention - probably the most important observation - is that latency issues go away after a reboot.

HOWEVER, once the PC has been left idling overnight, the audio crackle/pop/tears come back with a vengeance. Do you know why it degrades overnight after idling for 7+ hours? And what is it about a reboot that temporarily eliminates all latency issues?

 

[xilolee]

Let's try to make some changes in device manager, network adapters, double-click surface ethernet adapter.
If it's present, switch to the Power management tab and disable (untick/uncheck) Allow the computer to turn off this device to save power.
Then switch to the advanced tab and disable entries like "green ethernet", "ethernet energy saver", and/or similar entries.
Change "speed & duplex" from "auto-negotiation" to "1.0 Gbps Full Duplex".
Then click apply/ok.

If it won't work, try "100 Mbps Full Duplex".

 

[ctp9]

Thank you for the suggestions. I typically optimize my Ethernet adapters and already had all "green technology" / "energy savings" disabled. I also do not allow Windows to put my Ethernet adapter or USB hubs to sleep.

For your sake, I'll post the entire list of settings on the adapter I'm using:

Adaptive Link Speed ...................> DISABLED
ARP Offload ..............................> DISABLED
Battery Mode Link Speed ............> NOT SPEED DOWN
Energy-Efficient Ethernet ............> DISABLED
Flow Control .............................> DISABLED
Idle Power Saving ......................> DISABLED
IPv4 Checksum Offload ...............> RX & TX ENABLED
Jumbo Frame ............................> 9014 Bytes
Large Send Offload v2 (IPv4) .......> DISABLED
Large Send Offload v2 (IPv6) .......> DISABLED
Modern Standby WoL Magic Pack ..> DISABLED
Network Address .......................> NOT PRESENT
NS Offload ...............................> DISABLED
Priority & VLAN .........................> PRIORITY & VLAN DISABLED
Selective suspend .....................> DISABLED
Speed & Duplex ........................> 1.0 Gbps Full Duplex
SS idle timeout .........................> 5
SS idle timeout (Screen off) ........> 3
TCP Checksum Offload (IPv4) ......> RX & TX ENABLED
TCP Checksum Offload (IPv6) ......> RX & TX ENABLED
UDP Checksum Offload (IPv4) ......> RX & TX ENABLED
UDP Checksum Offload (IPv6) ......> RX & TX ENABLED
VLAN ID ..................................> NOT PRESENT
Wake on link change ..................> DISABLED
Wake on Magic Packet ...............> DISABLED
Wake on Pattern match ..............> DISABLED
WOL & Shutdown Link Speed .......> NOT SPEED DOWN

 

[ctp9]

If it's of any more assistance to you, I'd like to share my undervolting settings, along with a few other tweaks that are currently running on my system. This configuration and tweaks have never given me problems before, but as I have mentioned in my first post - I'm almost fully certain that these issues started occurring after a recent Windows Update that was pushed in February.

Latest version of ThrottleStop (8.74). Slight undervolt to reduce heat accumulation and to prolong the lifespan of hardware:



CPU is set to -87.9 mV:



iGPU is set to -50.8 mV:



Turbo Power Limits:



C-States:



As well, I'm using Intelligent Standby List Cleaner (ISLC) to purge memory standby list occasionally, and automatically... past a certain threshold. This tool allows me to also change Windows timer resolution to 0.5ms:



Power Plan:
Ultimate Performance (no throttling or selective suspend enabled anywhere)

BCDEDIT Tweaks:
useplatformtick.........Yes (to lock timer resolution precisely to 0.5ms as defined by ISLC)
disabledynamictick.....Yes (prevents Windows from putting system timer into power savings mode during idle)


If you see any issues with my settings that you think might be contributing to performance degradation, memory leaks, etc. - please let me know.

 

[xilolee]

I found some changes you may want to try: someone says they works (even if msmq isn't enabled, for the tcpnodelay parameter, for example).

Read More:

From an elevated command prompt:

netsh interface tcp set global autotuninglevel=experimental
(default is normal)

Output:
C:\>netsh interface tcp set global autotuninglevel=experimental
Ok.

netsh interface tcp set global ecncapability=enable
(default is disabled)

C:\>netsh interface tcp set global ecncapability=enable
Ok.

netsh interface tcp set supplemental template=internet congestionprovider=ctcp
(default is cubic)

Output:
C:\>netsh interface tcp set supplemental template=internet congestionprovider=ctcp
Ok.

reg add hklm\SOFTWARE\Microsoft\MSMQ\Parameters /t reg_dword /v TCPNoDelay /d 1
(default: the Parameter key isn't present, hence TCPNoDelay isn't present too)

Output:
C:\>reg add hklm\SOFTWARE\Microsoft\MSMQ\Parameters /t reg_dword /v TCPNoDelay /d 1
The operation completed successfully.

reg query hklm\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters\Interfaces /s /f subnetmask
(The aim of this command is to find the current interface code)

Output:
C:\>reg query hklm\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters\Interfaces /s /f subnetmask

HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters\Interfaces\{7ec1e70d-73d8-4e1f-8cc5-726cf3a220c6}
    SubnetMask    REG_MULTI_SZ    255.255.255.0

End of search: 1 match(es) found.

I found {7ec1e70d-73d8-4e1f-8cc5-726cf3a220c6} and I'll use it in the next command.
That value will be different for you, obviously.

reg add hklm\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters\Interfaces\{7ec1e70d-73d8-4e1f-8cc5-726cf3a220c6} /t reg_dword /v TcpAckFrequency /d 1
(default: the TCPNoDelay data/entry isn't present)

Output:
C:\>reg add hklm\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters\Interfaces\{7ec1e70d-73d8-4e1f-8cc5-726cf3a220c6} /t reg_dword /v TcpAckFrequency /d 1
The operation completed successfully.

 

[ctp9]

I have applied the settings you suggested. (I used a popular tool for modifying the parameters. I left all other values on default).





I will give you an update after the PC has been left idling overnight (because that's when the problems typically arise).

Also - may I ask what your line of reasoning is for trying these settings? I have also suspected that it's the Ethernet adapter causing the spikes, but didn't think there was anything we could do about it. My suspicion was that Windows Update broke something for me (specifically how tcpip.sys and NDIS.sys interfaces with my Ethernet adapter).

 

[ctp9]

Update: After leaving the PC idle overnight, latency issues creep up again the next day, but not as severe. There is some noticeable improvement after applying the above network tweaks and the Ethernet adapter is no longer spiking as much as it used to. Do you need a new trace to determine the next steps?

 

[ctp9]

Recorded new LatencyMon after a reboot, and also after idling overnight.

After a fresh restart:




After PC has been left idling overnight - latency becomes unbearable and audio starts skipping and glitching:

 

[xilolee]

Try to disable also the offload options in your nic settings.

 

[ctp9]

Still happening with all offload settings disabled, but slightly less severe. Do you need a new trace?

 

[ctp9]

LatencyMon looks almost worse, however.

 

[xilolee]

Did you download and install the latest drivers?
See Surface Pro 6 Drivers and Firmware (click)