[QUESTION] !ipi extension

x BlueRobot · Aug 5, 2013

I understand, what a IPI is and why it may be used, I'm just unsure about the !ipi extension, I best add that the following example is from a Stop 0x101, and the hung processor was number 2.

Code:

2: kd> [COLOR=#008000]!ipi[/COLOR]
IPI State for Processor 0

    As a sender, awaiting IPI completion from processor(s) 2, 3, 4, 5, 6, 7.

    TargetCount          6  PacketBarrier        1  IpiFrozen     0 [COLOR=#ff0000][Running][/COLOR]


IPI State for Processor 1
    TargetCount          0  PacketBarrier        0  IpiFrozen     2 [COLOR=#ff0000][Frozen][/COLOR]


IPI State for Processor 2

    As a receiver, unhandled requests are pending from processor(s) 0.

    TargetCount          0  PacketBarrier        0  IpiFrozen     2 [Frozen]

    From processor 0, active request of type: packet ready
        WorkerRoutine fffff80003e76e30 ([COLOR=#ff0000]nt!FsRtlpNopStackOverflowRoutine[/COLOR])
        Parameter[0] 0  Parameter[1] 0  Parameter[2] 0

IPI State for Processor 3

    As a receiver, unhandled requests are pending from processor(s) 0.

    TargetCount          0  PacketBarrier        0  IpiFrozen     5 [Target Freeze]

    From processor 0, active request of type: packet ready
        WorkerRoutine fffff80003e76e30 (nt!FsRtlpNopStackOverflowRoutine)
        Parameter[0] 0  Parameter[1] 0  Parameter[2] 0

IPI State for Processor 4

    As a receiver, unhandled requests are pending from processor(s) 0.

    TargetCount          0  PacketBarrier        0  IpiFrozen     5 [COLOR=#ff0000][Target Freeze][/COLOR]

    From processor 0, active request of type: packet ready
        WorkerRoutine fffff80003e76e30 (nt!FsRtlpNopStackOverflowRoutine)
        Parameter[0] 0  Parameter[1] 0  Parameter[2] 0

IPI State for Processor 5

    As a receiver, unhandled requests are pending from processor(s) 0.

    TargetCount          0  PacketBarrier        0  IpiFrozen     2 [Frozen]

    From processor 0, active request of type: packet ready
        WorkerRoutine fffff80003e76e30 (nt!FsRtlpNopStackOverflowRoutine)
        Parameter[0] 0  Parameter[1] 0  Parameter[2] 0

IPI State for Processor 6

    As a receiver, unhandled requests are pending from processor(s) 0.

    TargetCount          0  PacketBarrier        0  IpiFrozen     5 [Target Freeze]

    From processor 0, active request of type: packet ready
        WorkerRoutine fffff80003e76e30 (nt!FsRtlpNopStackOverflowRoutine)
        Parameter[0] 0  Parameter[1] 0  Parameter[2] 0

IPI State for Processor 7

    As a receiver, unhandled requests are pending from processor(s) 0.

    TargetCount          0  PacketBarrier        0  IpiFrozen     5 [Target Freeze]

    From processor 0, active request of type: packet ready
        WorkerRoutine fffff80003e76e30 (nt!FsRtlpNopStackOverflowRoutine)
        Parameter[0] 0  Parameter[1] 0  Parameter[2] 0

Code:

0: kd> [COLOR=#008000]!ipi 2[/COLOR]
IPI State for Processor 2

    As a receiver, unhandled requests are pending from processor(s) 0.

    TargetCount          0  PacketBarrier        0  IpiFrozen     2 [Frozen]

    From processor 0, active request of type: packet ready
        WorkerRoutine fffff80003e76e30 (nt!FsRtlpNopStackOverflowRoutine)
        Parameter[0] 0  Parameter[1] 0  Parameter[2] 0
    From processor 1, stale request of type: flush multiple range
        Flush Count 1  Flush List fffff8800a4d1af8  (dp fffff8800a4d1af8 l1)
    From self, stale request of type: none

Code:

2: kd> [COLOR=#008000]~0[/COLOR]
0: kd> [COLOR=#008000]k[/COLOR]
Child-SP          RetAddr           Call Site
fffff880`0cc7e1d8 fffff800`03edca3a nt!KeBugCheckEx
fffff880`0cc7e1e0 fffff800`03e8f6e7 nt! ?? ::FNODOBFM::`string'+0x4e3e
fffff880`0cc7e270 fffff800`04400895 [COLOR=#ff0000]nt!KeUpdateSystemTime+0x377[/COLOR] <-- Doesn't that result in a IPI?
fffff880`0cc7e370 fffff800`03e82153 hal!HalpHpetClockInterrupt+0x8d
fffff880`0cc7e3a0 fffff800`03e5a90f nt!KiInterruptDispatchNoLock+0x163
fffff880`0cc7e530 fffff800`04143e2f nt!KeFlushProcessWriteBuffers+0x6b
fffff880`0cc7e5a0 fffff800`041924f6 nt!ExpGetProcessInformation+0x7f
fffff880`0cc7e6f0 fffff800`04192f4d nt!ExpQuerySystemInformation+0xfb4
fffff880`0cc7eaa0 fffff800`03e84e93 nt!NtQuerySystemInformation+0x4d
fffff880`0cc7eae0 00000000`77a9167a nt!KiSystemServiceCopyEnd+0x13
00000000`0301d9f8 00000000`00000000 0x77a9167a

I've highlighed my main areas of interest in red, and I'm going to add a few guesses of my own:

[Frozen] and [Target Freeze], does frozen simply mean the processor is idle? What is Target Freeze?

nt!FsRtlpNopStackOverflowRoutine the routine being requested to carried out, or the routine which has caused the IPI?

[Running] the processor which sent the request?

Vir Gnarus · Aug 6, 2013

I'm not entirely familiar with IPIs because there's little explained about them that I can find. What I can say is that from previous experience that I've found idle and frozen are similar but not identical states. I believe frozen means a core is in a state where it has to be woken up by an external source like another cpu core. Think of one of the reasons for IPIs and that's synchronization. If you need to synch something between the cores, you can't just call a core with an IPI to synch up then let it run off to do its own thing again, otherwise it may very well end up getting out of synch all over again. Rather, you trigger it to do the work, then have it freeze and wait till its brethren are also synched, then "leader" of cores can finally say, "Ok, we're all synched, we can run again." At least that's how I've gathered how it's done. It's like race horses lining up at the gate before they are sent off.

As for Target Freeze, I'm not sure on that.

As for the routine, I don't know. I would reckon, though, that you can figure that out by doing a !running -it on it and seeing where the routine may appear in each stack. If it shows up in the stack of the thread that triggered the IPI, then I would gather that's what sent the request, and if it shows up in the threads of those targeted by the IPI, then I reckon it's the code being sent to them. Given that it says "worker routine", I have a feeling it's the latter.

The running state just means that: the core is running. It is not conducive of whether or not that core is the one that triggered the IPI. I can see that being the case, but really it just means the core is in a normal state.

Sorry I can't be of much help. For some reason IPIs are not well documented from Microsoft, at least I certainly haven't found anything public that mentions it. I understand it's more of a basic programming/OS design technique, but I can't find anything documenting how Windows performs it.

x BlueRobot · Aug 6, 2013

Thanks for your help as always, it is really difficult to find anything conclusive about the !ipi extension or how it is used by Windows, someone asked a similar question on Stack Overflow, but no one seemed to be able to provide an answer to it.

I'll look into the thread stacks :)

x BlueRobot · Aug 6, 2013

Processor 0 (The Sender):

Code:

fffff880`0cc7e408  fffff800`03ea4843 [COLOR=#ff0000]nt!KiIpiSendRequest+0x2d3[/COLOR]
fffff880`0cc7e410  00000000`00000000
fffff880`0cc7e418  00000000`00000000
fffff880`0cc7e420  00000000`00000000
fffff880`0cc7e428  00000000`00000000
fffff880`0cc7e430  00000000`00000000
fffff880`0cc7e438  00000000`00000000
fffff880`0cc7e440  00000000`00000000
fffff880`0cc7e448  00000000`00000000
fffff880`0cc7e450  00000000`00000000
fffff880`0cc7e458  00000000`00000000
fffff880`0cc7e460  00000000`00000000
fffff880`0cc7e468  00000000`00000000
fffff880`0cc7e470  00000000`00000000
fffff880`0cc7e478  00000000`00000000
fffff880`0cc7e480  00000000`00000007
fffff880`0cc7e488  00000000`00000000
fffff880`0cc7e490  00000000`00000000
fffff880`0cc7e498  fffff880`0cc7e730
fffff880`0cc7e4a0  00000000`00028000
fffff880`0cc7e4a8  00000000`00000000
fffff880`0cc7e4b0  00000000`00000001
fffff880`0cc7e4b8  00000000`00000000
fffff880`0cc7e4c0  00000000`00000000
fffff880`0cc7e4c8  fffff800`03e5ab6f [COLOR=#ff0000]nt!KiIpiSendPacket+0x4f[/COLOR]
fffff880`0cc7e4d0  fffff800`04000e80 nt!KiInitialPCR+0x180
fffff880`0cc7e4d8  00000000`00000001
fffff880`0cc7e4e0  fffff800`04000e80 nt!KiInitialPCR+0x180
fffff880`0cc7e4e8  fffff880`0cc7e500
fffff880`0cc7e4f0  00000000`00000000
fffff880`0cc7e4f8  00000000`00000000
fffff880`0cc7e500  00000000`00000001
fffff880`0cc7e508  fffff800`03e5a90f nt!KeFlushProcessWriteBuffers+0x6b

There's no mention of the nt!FsRtlpNopStackOverflowRoutine.

Processor 7:

Code:

[/COLOR][I][COLOR=#000000][/COLOR][/I]fffff880`035b1798  fffff800`03ea4ae4 nt!KiIpiProcessRequests+0x194
fffff880`035b17a0  00000000`00000032
fffff880`035b17a8  fffff800`03ea4ae4 nt!KiIpiProcessRequests+0x194
fffff880`035b17b0  00000000`00000000
fffff880`035b17b8  00000000`00000000
fffff880`035b17c0  fffff800`03e8fa95 nt!KiIpiInterrupt+0x135

There's a couple of routines involving receiving the interrupt and then processing the request. The other threads seem to be similar.

Shintaro · Aug 7, 2013

Found this on the Internet.
Might be useful.

Code:

/*++

Copyright (c) Microsoft Corporation. All rights reserved. 

You may only use this code if you agree to the terms of the Windows Research Kernel Source Code License agreement (see License.txt).
If you do not agree to the terms, do not use the code.


Module Name:

    stubs.c

Abstract:

    This module implements kernel debugger synchronization routines.

--*/

#include "ki.h"

//
// KiDebugRoutine - This is the address of the kernel debugger. Initially
//      this is filled with the address of a routine that just returns. If
//      the system debugger is present in the system, then it sets this
//      location to the address of the system debugger's routine.
//

PKDEBUG_ROUTINE KiDebugRoutine = &KdpStub;

#if defined(_AMD64_)
#define IDBG    0
#else
#define IDBG    1
#endif


//
// Define flags bits.
//

#define FREEZE_ACTIVE           0x20

//
// Define local storage to save the old IRQL.
//

KIRQL KiOldIrql;                                  

#ifndef NT_UP
PKPRCB KiFreezeOwner;
#endif

BOOLEAN
KeFreezeExecution (
    IN PKTRAP_FRAME TrapFrame,
    IN PKEXCEPTION_FRAME ExceptionFrame
    )

/*++

Routine Description:

    This function freezes the execution of all other processors in the host
    configuration and then returns to the caller.

Arguments:

    TrapFrame - Supplies a pointer to a trap frame that describes the
        trap.

    ExceptionFrame - Supplies a pointer to an exception frame that
        describes the trap.

Return Value:

    Previous interrupt enable.

--*/

{

    BOOLEAN Enable;

#if !defined(NT_UP)

    PKPRCB Prcb;
    KAFFINITY TargetSet;
    ULONG BitNumber;
    KIRQL OldIrql;

#if IDBG

    ULONG Count = 30000;

#endif

#if !defined(_AMD64_)
    BOOLEAN Flag;
#endif

#else

    UNREFERENCED_PARAMETER(TrapFrame);
    UNREFERENCED_PARAMETER(ExceptionFrame);

#endif

    //
    // Disable interrupts.
    //

    Enable = KeDisableInterrupts();
    KiFreezeFlag = FREEZE_FROZEN;

    //
    // Raise IRQL to HIGH_LEVEL.
    //

#if !defined(NT_UP)

    KeRaiseIrql(HIGH_LEVEL, &OldIrql);
    if (FrozenState(KeGetCurrentPrcb()->IpiFrozen) == FREEZE_OWNER) {

        //
        // This processor already owns the freeze lock.
        // Return without trying to re-acquire lock or without
        // trying to IPI the other processors again
        //

        return Enable;
    }

#if defined(_AMD64_)

    UNREFERENCED_PARAMETER(TrapFrame);
    UNREFERENCED_PARAMETER(ExceptionFrame);

    //
    // Acquire the KiFreezeExecutionLock.  No need to check for FREEZE
    // IPIs as these come in as NMIs anyway.
    //

    KeAcquireSpinLockAtDpcLevel(&KiFreezeExecutionLock);

#else

    //
    // Try to acquire the KiFreezeExecutionLock before sending the request.
    // To prevent deadlock from occurring, we need to accept and process
    // incoming FreezeExecution requests while we are waiting to acquire
    // the FreezeExecutionFlag.
    //

    while (KeTryToAcquireSpinLockAtDpcLevel(&KiFreezeExecutionLock) == FALSE) {

        //
        // FreezeExecutionLock is busy.  Another processor may be trying
        // to IPI us - go service any IPI.
        //

        KeEnableInterrupts(Enable);
        Flag = KiIpiServiceRoutine((PVOID)TrapFrame, (PVOID)ExceptionFrame);
        KeDisableInterrupts();

#if IDBG

        if (Flag != FALSE) {
            Count = 30000;
            continue;
        }

        KeStallExecutionProcessor (100);
        if (!Count--) {
            Count = 30000;
            if (KeTryToAcquireSpinLockAtDpcLevel(&KiFreezeLockBackup) == TRUE) {
                KiFreezeFlag |= FREEZE_BACKUP;
                break;
            }
        }

#endif

    }

#endif

    //
    // After acquiring the lock flag, we send Freeze request to each processor
    // in the system (other than us) and wait for it to become frozen.
    //

    Prcb = KeGetCurrentPrcb();  // Do this after spinlock is acquired.
    TargetSet = KeActiveProcessors & ~(AFFINITY_MASK(Prcb->Number));
    if (TargetSet) {

#if IDBG
        Count = 4000;
#endif

        KiFreezeOwner = Prcb;
        Prcb->IpiFrozen = FREEZE_OWNER | FREEZE_ACTIVE;
        Prcb->SkipTick  = TRUE;
        KiSendFreeze(&TargetSet, (KeBugCheckActive != 2));
        while (TargetSet != 0) {
            KeFindFirstSetLeftAffinity(TargetSet, &BitNumber);
            ClearMember(BitNumber, TargetSet);
            Prcb = KiProcessorBlock[BitNumber];

#if IDBG

            while (Prcb->IpiFrozen != TARGET_FROZEN) {
                if (Count == 0) {
                    KiFreezeFlag |= FREEZE_SKIPPED_PROCESSOR;
                    break;
                }

                KeStallExecutionProcessor (10000);
                Count--;
            }

#else

            while (Prcb->IpiFrozen != TARGET_FROZEN) {
                KeYieldProcessor();
            }
#endif

        }
    }

    //
    // Save the old IRQL.
    //

    KiOldIrql = OldIrql;

#else

    //
    // Save the current IRQL.
    //

    KiOldIrql = KeGetCurrentIrql();

#endif      // !defined(NT_UP)

    //
    // Return whether interrupts were previous enabled.
    //

    return Enable;
}

VOID
KiFreezeTargetExecution (
    IN PKTRAP_FRAME TrapFrame,
    IN PKEXCEPTION_FRAME ExceptionFrame
    )

/*++

Routine Description:

    This function freezes the execution of the current running processor.
    If a trapframe is supplied to current state is saved into the prcb
    for the debugger.

Arguments:

    TrapFrame - Supplies a pointer to the trap frame that describes the
        trap.

    ExceptionFrame - Supplies a pointer to the exception frame that
        describes the trap.

Return Value:

    None.

--*/

{

#if !defined(NT_UP)

    KIRQL OldIrql;
    PKPRCB Prcb;
    BOOLEAN Enable;
    KCONTINUE_STATUS Status;
    EXCEPTION_RECORD ExceptionRecord;

    //
    // If the freeze lock is not held, then a debugger freeze was missed.
    //

    if ((KiFreezeExecutionLock == 0) && 
        (KiFreezeLockBackup == 0) &&
        (KeBugCheckActive == FALSE)) {
#if DBG

        DbgBreakPoint();

#endif

        return;
    }

    Enable = KeDisableInterrupts();
    KeRaiseIrql(HIGH_LEVEL, &OldIrql);
    Prcb = KeGetCurrentPrcb();
    Prcb->IpiFrozen = TARGET_FROZEN;
    Prcb->SkipTick  = TRUE;
    if (TrapFrame != NULL) {
        KiSaveProcessorState(TrapFrame, ExceptionFrame);
    }

    //
    // Sweep the data cache in case this is a system crash and the bugcheck
    // code is attempting to write a crash dump file.
    //

    KeSweepCurrentDcache();

    //
    //  Wait for person requesting us to freeze to
    //  clear our frozen flag
    //

    while (FrozenState(Prcb->IpiFrozen) == TARGET_FROZEN) {
        if (Prcb->IpiFrozen & FREEZE_ACTIVE) {

            //
            // This processor has been made the active processor
            //
            if (TrapFrame) {
                RtlZeroMemory (&ExceptionRecord, sizeof ExceptionRecord);
                ExceptionRecord.ExceptionCode = STATUS_WAKE_SYSTEM_DEBUGGER;
                ExceptionRecord.ExceptionRecord  = &ExceptionRecord;
                ExceptionRecord.ExceptionAddress =
                    (PVOID)CONTEXT_TO_PROGRAM_COUNTER (&Prcb->ProcessorState.ContextFrame);

                Status = (KiDebugSwitchRoutine) (
                            &ExceptionRecord,
                            &Prcb->ProcessorState.ContextFrame,
                            FALSE
                            );

            } else {
                Status = ContinueError;
            }

            //
            // If status is anything other then, continue with next
            // processor then reselect master
            //

            if (Status != ContinueNextProcessor) {
                Prcb->IpiFrozen &= ~FREEZE_ACTIVE;
                KiFreezeOwner->IpiFrozen |= FREEZE_ACTIVE;
            }
        }
        KeYieldProcessor();
    }

    if (TrapFrame != NULL) {
        KiRestoreProcessorState(TrapFrame, ExceptionFrame);
    }

    Prcb->IpiFrozen = RUNNING;
    KeFlushCurrentTb();
    KeSweepCurrentIcache();
    KeLowerIrql(OldIrql);
    KeEnableInterrupts(Enable);

#else

    UNREFERENCED_PARAMETER(TrapFrame);
    UNREFERENCED_PARAMETER(ExceptionFrame);

#endif      // !define(NT_UP)

    return;
}

KCONTINUE_STATUS
KeSwitchFrozenProcessor (
    IN ULONG ProcessorNumber
    )
{

#if !defined(NT_UP)

    PKPRCB TargetPrcb, CurrentPrcb;

    //
    // If Processor number is out of range, reselect current processor
    //

    if (ProcessorNumber >= (ULONG) KeNumberProcessors) {
        return ContinueProcessorReselected;
    }

    TargetPrcb = KiProcessorBlock[ProcessorNumber];
    CurrentPrcb = KeGetCurrentPrcb();

    //
    // Move active flag to correct processor.
    //

    CurrentPrcb->IpiFrozen &= ~FREEZE_ACTIVE;
    TargetPrcb->IpiFrozen  |= FREEZE_ACTIVE;

    //
    // If this processor is frozen in KiFreezeTargetExecution, return to it
    //

    if (FrozenState(CurrentPrcb->IpiFrozen) == TARGET_FROZEN) {
        return ContinueNextProcessor;
    }

    //
    // This processor must be FREEZE_OWNER, wait to be reselected as the
    // active processor
    //

    if (FrozenState(CurrentPrcb->IpiFrozen) != FREEZE_OWNER) {
        return ContinueError;
    }

    while (!(CurrentPrcb->IpiFrozen & FREEZE_ACTIVE)) {
        KeYieldProcessor();
    }

#else

    UNREFERENCED_PARAMETER(ProcessorNumber);

#endif  // !defined(NT_UP)

    //
    // Reselect this processor
    //

    return ContinueProcessorReselected;
}

VOID
KeThawExecution (
    IN BOOLEAN Enable
    )

/*++

Routine Description:

    This function thaws the execution of all other processors in the host
    configuration and then returns to the caller. It is intended for use by
    the kernel debugger.

Arguments:

    Enable - Supplies the previous interrupt enable that is to be restored
        after having thawed the execution of all other processors.

Return Value:

    None.

--*/

{

#if !defined(NT_UP)

    KIRQL OldIrql;

#if IDBG
    ULONG Flag;
#endif

    KiSendThawExecution (TRUE);

    //
    // Capture the previous IRQL before releasing the freeze lock.
    //

    OldIrql = KiOldIrql;

#if IDBG

    Flag = KiFreezeFlag;
    KiFreezeFlag = 0;

    if ((Flag & FREEZE_BACKUP) != 0) {
        KiReleaseSpinLock(&KiFreezeLockBackup);
    } else {
        KiReleaseSpinLock(&KiFreezeExecutionLock);
    }

#else

    KiFreezeFlag = 0;
    KiReleaseSpinLock(&KiFreezeExecutionLock);

#endif
#endif  // !defined (NT_UP)


    //
    // Flush the current TB, instruction cache, and data cache.
    //

    KeFlushCurrentTb();
    KeSweepCurrentIcache();
    KeSweepCurrentDcache();

    //
    // Lower IRQL and restore interrupt enable
    //

#if !defined(NT_UP)
    KeLowerIrql(OldIrql);
#endif
    KeEnableInterrupts(Enable);
    return;
}

VOID
KiPollFreezeExecution(
    VOID
    )

/*++

Routine Description:

    This routine is called from code that is spinning with interrupts
    disabled, waiting for something to happen, when there is some
    (possibly extremely small) chance that that thing will not happen
    because a system freeze has been initiated.

    N.B. Interrupts are disabled.

Arguments:

    None.

Return Value:

    None.

--*/

{

#if defined(_AMD64_)

    KeYieldProcessor();

#else

    //
    // Check to see if a freeze is pending for this processor.
    //

    PKPRCB Prcb = KeGetCurrentPrcb();

    if ((Prcb->RequestSummary & IPI_FREEZE) != 0) {

        //
        // Clear the freeze request and freeze this processor.
        //

        InterlockedExchangeAdd((PLONG)&Prcb->RequestSummary, -(IPI_FREEZE));
        KiFreezeTargetExecution(NULL, NULL);

    } else {

        //
        // No freeze pending, assume this processor is spinning.
        //

        KeYieldProcessor();
    }

#endif

}

x BlueRobot · Aug 7, 2013

Thanks that is great, could you post the link?

Vir Gnarus · Aug 7, 2013

Here. It's part of the WRK. Understand that this is related to the WinXP/2003 SP1 x64 kernel, so obviously things would've been changed since then. Also notice that's just one implementation of IPI usage.

cluberti · Aug 7, 2013

Is this a virtual machine? I've seen this happen on VMware hosts - VMware 4.x has HPET issues for sure, and 5.x is so far unscathed as far as I can tell, so it maybe fixed running Windows hosts in 5.x. It could be that the hardware itself has HPET issues too, if it's not a VM, but that looks an awful lot like bad hardware not responding to HPET timer interrupt requests (Vista/2008 and higher use HPET for this, rather than RTC). This might be useful as well:
Debugging an Interrupt Storm (Windows Debuggers)

If this is hardware, and you go into an interrupt storm or hang when you're in HPET code doing timer interrupts, you have bad hardware. If it's in a VM, you need to either disable HPET on the host or change Windows to use RTC to avoid it.

x BlueRobot · Aug 8, 2013

It's on a real computer, and thanks for the debugging link :)

[QUESTION] !ipi extension

x BlueRobot

Administrator

Vir Gnarus

BSOD Kernel Dump Expert

x BlueRobot

Administrator

x BlueRobot

Administrator

Shintaro

Well-known member

x BlueRobot

Administrator

Vir Gnarus

BSOD Kernel Dump Expert

cluberti

Senior Member

x BlueRobot

Administrator