Jared
Sysnative Staff, BSOD Kernel Dump Expert
- Feb 3, 2014
- 1,591
I got back off holiday yesterday and I feel it's time for a blog post, this might be a long one as I have spent some time reading Windows Internals and looking at various subjects of interrupts DPCs and APCs.
DPCs and dispatch interrupts
So a DPC or Deferred Procedure Call is a way for the Kernel to defer execution until a later date, this is a good way for Windows to stop threads that take too long executing or are running for long periods of time at a high IRQL from holding up the system.
There's more to it than that so lets read on.
Well when the Kernel needs to synchronise access to shared data structures the IRQL is raised to DPC/dispatch level or above, therefore thread dispatching is disabled along with system service calling and page faults aren't permitted. When thread dispatching needs to take place the Kernel requests a software interrupt but masks it until the the processor finishes the work it's currently executing, afterwards the processor lowers the IRQL to APC or PASSIVE_LEVEL but first checks if there are any pending dispatch interrupts waiting to be executed, if there are then the IRQL drops to DPC/Dispatch level and executes them. Using dispatch interrupts is a way to defer thread dispatching until the right conditions are met, this is useful as the current activity could be more important than the thread wanting to be executed and therefore makes more sense to finish more important work first.
DPCs are mainly used to defer systems tasks for later execution, ones that are less important and can be executed later on with little repercusions than the task being executed. Device drivers also use DPCs to process interrupts, Windows and device drivers try to keep the IRQL below device IRQLs to prevent the system from running into problems by holding up normal execution. DPCs are used to process timer expiration and to release threads that are waiting on a timer which has already expired, if a thread doesn't release a thread waiting on a timer then the thread will stay in the wait state forever which AFAIK causes a deadlock.
To view DPCs queued on a certain processor then you can use the !dpcs extension.
DPCs Advanced
DPCs do not run in the context of a particular thread being executed and therefore are not allowed to call system services, generate page faults and cannot create/wait for dispatcher objects. I will talk about dispatcher objects in another post as they are a long topic but simply, they are synchronisation mechanisms such as mutex objects, event objects, semaphores and timer objects.However, DPCs can access nonpaged memory as it is mapped in the system address space at all times.
DPCs are put in per processor queues meaning each processor has their own queue which are all independent from each other. A DPC is represented by a DPC object which contains code for the thread being executed, the most important information is the location of the thread in memory being executed when the DPC is initiated. Think of it as a shortcut on the desktop, without the shortcut containing the exact memory location of the actual program it can't run it.
I know what you're thinking now, "why can't the DPC just find the thread in memory if it isn't in the code?"
It's possible to do but it would take that long it would hold everything up and would cause more problems if it did so.
For more information on how page faults work then Patrick made a good post here:
https://www.sysnative.com/forums/bs...-information/10551-page-faults-explained.html
DPCs have priorities (low, medium, medium-high and high), these priorities determine when the DPC is executed and how it is added to the queue, by default device drivers set the DPC at medium but it can be overwritten. When the DPC priority is high it is placed at the front of the DPC queue for that particular processor, if the priority is medium-high or lower it places the DPC at the back of the queue. DPCs also use targeting which determines which CPU will execute the DPC, unfortunately most device drivers don't specify a processor which results in processor 0 being overwhelmed with DPCs. DPCs are then pulled out of the queue and executed one by one until the queue is empty, this is known as draining, once the queue is drained the IRQL can drop below DPC/Dispatch level.
DPC draining takes place when the DPC priority is higher than low and is targeted at the current processor executing, if the priority is low then DPC draining takes place when the amount of DPCs in the queue exceed a certain threshold.
If the targeted CPU is different from the one running the ISR then depending on the priority depends on how the DPC is handled. If the priority is high or medium-high then the current processor sends out an IPI (Inter Processor Interrupt) to the targeted processor to drain the DPC queue, this happens only when the targeted CPU is idle. If the priority is low or medium then the targeted processor's DPC queue must exceed a threshold for the draining to be initiated or the system is idle.
To view the Inter Processor Interrupts pending then you can use the !ipi extension.
The Kernel uses DPCs paired with clock interrupts, when a clock interrupt occurs (at every system clock tick count) the IRQL is raised to Clock level where the clock interrupt handler updates the system time, it then decrements a counter to track how long the current thread as run and when the counter reaches 0 the thread's quantum has expired and the kernel might need to reschedule the processor which is a low priority task and should be done later on which is when it gets implemented as a DPC. When the clock interrupt handler finishes its work the Kernel lowers the IRQL and executes the DPC.
The main problem with DPCs is that they don't run in the context of a particular thread and therefore don't care what thread is currently executing meaning even the highest priority threads can be interrupted which is why you can experience slowness and sound lagging. This can be worked around by using Threaded DPCs which run at Passive level IRQL and run as a real time thread (level 31), this allows the DPC to preempt most user mode threads as they don't run at real time thread priority but allows nonthreaded DPCs, APCs and higher priority threads to preempt the routine, preventing this problem.
For more information on interrupts then take a look here:
Jared's BSOD debugging blog: Interrupt dispatching and handling
For information on thread priority then visit Harry's blog here:
BSODTutorials: Thread Scheduling and Priority Levels
Asynchronous Procedure Calls
Asynchronous Procedure Calls or APCs provide a way for user mode and system code to execute in context to a particular thread. Due to the fact that APCs run in context of a particular thread and therefore in a process address space they don't have the same restrictions to a DPC in that they can acquire objects, wait for object handles, incur page faults and call system services.
APCs are described by a Kernel object called an APC object and APCs waiting to run are placed in an APC queue which unlike DPC queues (which are per processor) are in the context of a thread rather than per processor meaning every thread has its own APC queue. The Kernel requests a software interrupt at APC level and when the thread is ran the APC is executed.
There are two kinds of APCs, Kernel APCs and User APCs which differ in that User APCs need permission from the thread it is trying to run on where as Kernel APCs run without permission.
There are then two types of Kernel mode APCs, normal and special; Special APCs run at APC IRQL and allow the modification of the normal APC, the normal APC runs and uses the altered parameters from the special APC or the original ones if there was no special APC.
Special APCs can be disabled by calling the KeEnterGuardedRegion which sets the SpecialApcDelivery field in the KTHREAD data structure shown here.
You can also see the KernelApcDisable and CombinedApcDisable entries, these are self-explanatory.
You can view the KTHREAD data structure by typing in dt nt!_KTHREAD followed by the thread's address in Windbg.
Special Kernel APCs are used, for example to show results after an asynchronous I/O operation has completed from and is then saved in the thread's address space.
Kernel APCs are also used mainly for thread suspension and termination as well as to query the context of a particular thread, this can be ran from arbitrary threads so device drivers often block APCs when holding a lock otherwise the lcok might never be released resulting in a system deadlock.
There are lots of different APIs that are user mode APCs such as ReadFileEx and WriteFileEx which allow the caller to initiate certain routines when an I/O operation completes. User mode threads however can't recieve APCs unless they're in a wait state by waiting for an object handle or volunatry entering a wait state by calling WaitForMultipleObjectsEx, this allows the kernel to alert the thread which can then let the thread recieve the APC and allows it to be executed. After the APC has completed the thread can resume execution as normal.
Unfortunately when APCs are initiated and the thread is in a wait state, when the APC completes and the thread checks whether the wait is resolved, if it isn't then it is put at the back of the wait queue for the objects it's waiting for. In other words when APCs are initiated the wait on a thread is removed until the APC completes, then it's put at the back of the thread wait list for the objects it's trying to access.
The APCs have different priorities like DPCs:
-Special Kernel APCs which are inserted at the tail of the Kernel mode APC list, theyare ran at APC level and when the thread isn't in a critical region, it's then given pointers to the parameters of the Kernel APC associated with it.
- Kernel APCs are implemented after the last Special Kernel APC (ahead of the rest of the Kernel APCS) and ran at Passive level , it recieves the arguements given from the Special Kernel APC associated with it (or none if there wasn't any alterations).
- User APCs are inserted at the tail of the user mode APC list, they are ran at passive level and when the thread isn't in a guarded region and the thread is in an alerted state, it is then given the arguments from the Special APC.
0x1 bugchecks (APC_INDEX_MISMATCH) can be caused by APCs not entering and leaving guarded or critical states correctly, there are mainly two reasons:
-The thread didn't leave the guarded/critical region which is needed as the I/O manager and Object manager use APCs to perform a lot of operations.
-The thread entered and left guarded/critical regions too many times.
The best option for finding the thread at fault is to enable Driver Verifier.
I believe the option to monitor APCs is Automatic Checks but I'm not too sure.
For more information on Driver Verifier see here:
Driver Verifier Options (Windows Drivers)
I was going to talk a lot about timers but I've already written a lot here so I think I'll save that for a seperate post, a lot of this is from Windows Internals but I've tried to explain it a bit differently as it can be overwhelming.
References:
Windows Internals
DPCs and dispatch interrupts
So a DPC or Deferred Procedure Call is a way for the Kernel to defer execution until a later date, this is a good way for Windows to stop threads that take too long executing or are running for long periods of time at a high IRQL from holding up the system.
There's more to it than that so lets read on.
Well when the Kernel needs to synchronise access to shared data structures the IRQL is raised to DPC/dispatch level or above, therefore thread dispatching is disabled along with system service calling and page faults aren't permitted. When thread dispatching needs to take place the Kernel requests a software interrupt but masks it until the the processor finishes the work it's currently executing, afterwards the processor lowers the IRQL to APC or PASSIVE_LEVEL but first checks if there are any pending dispatch interrupts waiting to be executed, if there are then the IRQL drops to DPC/Dispatch level and executes them. Using dispatch interrupts is a way to defer thread dispatching until the right conditions are met, this is useful as the current activity could be more important than the thread wanting to be executed and therefore makes more sense to finish more important work first.
DPCs are mainly used to defer systems tasks for later execution, ones that are less important and can be executed later on with little repercusions than the task being executed. Device drivers also use DPCs to process interrupts, Windows and device drivers try to keep the IRQL below device IRQLs to prevent the system from running into problems by holding up normal execution. DPCs are used to process timer expiration and to release threads that are waiting on a timer which has already expired, if a thread doesn't release a thread waiting on a timer then the thread will stay in the wait state forever which AFAIK causes a deadlock.
To view DPCs queued on a certain processor then you can use the !dpcs extension.
DPCs Advanced
DPCs do not run in the context of a particular thread being executed and therefore are not allowed to call system services, generate page faults and cannot create/wait for dispatcher objects. I will talk about dispatcher objects in another post as they are a long topic but simply, they are synchronisation mechanisms such as mutex objects, event objects, semaphores and timer objects.However, DPCs can access nonpaged memory as it is mapped in the system address space at all times.
DPCs are put in per processor queues meaning each processor has their own queue which are all independent from each other. A DPC is represented by a DPC object which contains code for the thread being executed, the most important information is the location of the thread in memory being executed when the DPC is initiated. Think of it as a shortcut on the desktop, without the shortcut containing the exact memory location of the actual program it can't run it.
I know what you're thinking now, "why can't the DPC just find the thread in memory if it isn't in the code?"
It's possible to do but it would take that long it would hold everything up and would cause more problems if it did so.
For more information on how page faults work then Patrick made a good post here:
https://www.sysnative.com/forums/bs...-information/10551-page-faults-explained.html
DPCs have priorities (low, medium, medium-high and high), these priorities determine when the DPC is executed and how it is added to the queue, by default device drivers set the DPC at medium but it can be overwritten. When the DPC priority is high it is placed at the front of the DPC queue for that particular processor, if the priority is medium-high or lower it places the DPC at the back of the queue. DPCs also use targeting which determines which CPU will execute the DPC, unfortunately most device drivers don't specify a processor which results in processor 0 being overwhelmed with DPCs. DPCs are then pulled out of the queue and executed one by one until the queue is empty, this is known as draining, once the queue is drained the IRQL can drop below DPC/Dispatch level.
DPC draining takes place when the DPC priority is higher than low and is targeted at the current processor executing, if the priority is low then DPC draining takes place when the amount of DPCs in the queue exceed a certain threshold.
If the targeted CPU is different from the one running the ISR then depending on the priority depends on how the DPC is handled. If the priority is high or medium-high then the current processor sends out an IPI (Inter Processor Interrupt) to the targeted processor to drain the DPC queue, this happens only when the targeted CPU is idle. If the priority is low or medium then the targeted processor's DPC queue must exceed a threshold for the draining to be initiated or the system is idle.
To view the Inter Processor Interrupts pending then you can use the !ipi extension.
The Kernel uses DPCs paired with clock interrupts, when a clock interrupt occurs (at every system clock tick count) the IRQL is raised to Clock level where the clock interrupt handler updates the system time, it then decrements a counter to track how long the current thread as run and when the counter reaches 0 the thread's quantum has expired and the kernel might need to reschedule the processor which is a low priority task and should be done later on which is when it gets implemented as a DPC. When the clock interrupt handler finishes its work the Kernel lowers the IRQL and executes the DPC.
The main problem with DPCs is that they don't run in the context of a particular thread and therefore don't care what thread is currently executing meaning even the highest priority threads can be interrupted which is why you can experience slowness and sound lagging. This can be worked around by using Threaded DPCs which run at Passive level IRQL and run as a real time thread (level 31), this allows the DPC to preempt most user mode threads as they don't run at real time thread priority but allows nonthreaded DPCs, APCs and higher priority threads to preempt the routine, preventing this problem.
For more information on interrupts then take a look here:
Jared's BSOD debugging blog: Interrupt dispatching and handling
For information on thread priority then visit Harry's blog here:
BSODTutorials: Thread Scheduling and Priority Levels
Asynchronous Procedure Calls
Asynchronous Procedure Calls or APCs provide a way for user mode and system code to execute in context to a particular thread. Due to the fact that APCs run in context of a particular thread and therefore in a process address space they don't have the same restrictions to a DPC in that they can acquire objects, wait for object handles, incur page faults and call system services.
APCs are described by a Kernel object called an APC object and APCs waiting to run are placed in an APC queue which unlike DPC queues (which are per processor) are in the context of a thread rather than per processor meaning every thread has its own APC queue. The Kernel requests a software interrupt at APC level and when the thread is ran the APC is executed.
There are two kinds of APCs, Kernel APCs and User APCs which differ in that User APCs need permission from the thread it is trying to run on where as Kernel APCs run without permission.
There are then two types of Kernel mode APCs, normal and special; Special APCs run at APC IRQL and allow the modification of the normal APC, the normal APC runs and uses the altered parameters from the special APC or the original ones if there was no special APC.
Special APCs can be disabled by calling the KeEnterGuardedRegion which sets the SpecialApcDelivery field in the KTHREAD data structure shown here.
You can also see the KernelApcDisable and CombinedApcDisable entries, these are self-explanatory.
You can view the KTHREAD data structure by typing in dt nt!_KTHREAD followed by the thread's address in Windbg.
Special Kernel APCs are used, for example to show results after an asynchronous I/O operation has completed from and is then saved in the thread's address space.
Kernel APCs are also used mainly for thread suspension and termination as well as to query the context of a particular thread, this can be ran from arbitrary threads so device drivers often block APCs when holding a lock otherwise the lcok might never be released resulting in a system deadlock.
There are lots of different APIs that are user mode APCs such as ReadFileEx and WriteFileEx which allow the caller to initiate certain routines when an I/O operation completes. User mode threads however can't recieve APCs unless they're in a wait state by waiting for an object handle or volunatry entering a wait state by calling WaitForMultipleObjectsEx, this allows the kernel to alert the thread which can then let the thread recieve the APC and allows it to be executed. After the APC has completed the thread can resume execution as normal.
Unfortunately when APCs are initiated and the thread is in a wait state, when the APC completes and the thread checks whether the wait is resolved, if it isn't then it is put at the back of the wait queue for the objects it's waiting for. In other words when APCs are initiated the wait on a thread is removed until the APC completes, then it's put at the back of the thread wait list for the objects it's trying to access.
The APCs have different priorities like DPCs:
-Special Kernel APCs which are inserted at the tail of the Kernel mode APC list, theyare ran at APC level and when the thread isn't in a critical region, it's then given pointers to the parameters of the Kernel APC associated with it.
- Kernel APCs are implemented after the last Special Kernel APC (ahead of the rest of the Kernel APCS) and ran at Passive level , it recieves the arguements given from the Special Kernel APC associated with it (or none if there wasn't any alterations).
- User APCs are inserted at the tail of the user mode APC list, they are ran at passive level and when the thread isn't in a guarded region and the thread is in an alerted state, it is then given the arguments from the Special APC.
0x1 bugchecks (APC_INDEX_MISMATCH) can be caused by APCs not entering and leaving guarded or critical states correctly, there are mainly two reasons:
-The thread didn't leave the guarded/critical region which is needed as the I/O manager and Object manager use APCs to perform a lot of operations.
-The thread entered and left guarded/critical regions too many times.
The best option for finding the thread at fault is to enable Driver Verifier.
I believe the option to monitor APCs is Automatic Checks but I'm not too sure.
For more information on Driver Verifier see here:
Driver Verifier Options (Windows Drivers)
I was going to talk a lot about timers but I've already written a lot here so I think I'll save that for a seperate post, a lot of this is from Windows Internals but I've tried to explain it a bit differently as it can be overwhelming.
References:
Windows Internals
Last edited: