I've pinpointed the problem to Windows Defender.
When disabling Windows Defender (not only the realtime protection, but disabling by putting the DisableAntiSpyware = 1 registery value in "HKLM/Software/Microsoft/Windows Defender" the problem doesn't happen anymore.
I am using Hyper-V, for Windows Phone development, can that be the cause? Is it possible to disable the hypervisor and check what will happen?