SysnativeBSODApps - additional check 'drivers found in stack'

[Vir Gnarus]

A critical data structure like KTHREAD would most certainly change between OS environments. So yeah, I guess that may need to be taken into account. If you can have the extension search by symbol to find the "StackBase" symbol that'd be indeed far more reliable, as otherwise you'd have to compensate the change for every OS environment, which is kind of an ugly venture.

 

[Vir Gnarus]

I decided to test and found that !thread also uses symbols to generate its contents, rather than relying on static offsets. I figured as much since it's rather daft not to use symbols since if nt module is not present there's something dreadfully wrong, and if you have missing symbols for it then you most likely don't have symbols setup for Windows, period. I say, when all else fails, use symbols. They're there to help you.

2: kd> .reload /u nt
Unloaded nt
2: kd> !thread
fffffa80273ed040: Unable to get thread contents
2: kd> kv [COLOR=#006400]< [/COLOR][COLOR=#006400]stack trace still works[/COLOR]
Child-SP          RetAddr           : Args to Child                                                           : Call Site
fffffa60`089b4a58 fffff800`01e63661 : 00000000`00000050 fffffa60`08c4d718 00000000`00000008 fffffa60`089b4b50 : 0xfffff800`01e54690
fffffa60`089b4a60 00000000`00000050 : fffffa60`08c4d718 00000000`00000008 fffffa60`089b4b50 00000000`00000000 : 0xfffff800`01e63661
fffffa60`089b4a68 fffffa60`08c4d718 : 00000000`00000008 fffffa60`089b4b50 00000000`00000000 00000000`00000000 : 0x50
fffffa60`089b4a70 00000000`00000008 : fffffa60`089b4b50 00000000`00000000 00000000`00000000 00000000`00000000 : <Unloaded_TmPreFlt.sys>+0x4718
fffffa60`089b4a78 fffffa60`089b4b50 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : 0x8
fffffa60`089b4a80 00000000`00000000 : 00000000`00000000 00000000`00000000 00000000`00000000 fffffa60`089b4b50 : 0xfffffa60`089b4b50
2: kd> .reload nt
2: kd> !thread
THREAD fffffa80273ed040  Cid 0004.0a64  Teb: 0000000000000000 Win32Thread: 0000000000000000 RUNNING on processor 2
Not impersonating
DeviceMap                 fffff88000006150
Owning Process            fffffa8024c1a040       Image:         System
Attached Process          N/A            Image:         N/A
Wait Start TickCount      5438           Ticks: 405495019 (73:05:09:22.838)
Context Switch Count      2              IdealProcessor: 2             
UserTime                  00:00:00.000
KernelTime                00:00:00.000
Win32 Start Address rpcxdr!RxWorkThread (0xfffffa6008394f88)
Stack Init fffffa60089b4db0 Current fffffa60089b4a10
Base fffffa60089b5000 Limit fffffa60089af000 Call 0
Priority 12 BasePriority 12 PriorityDecrement 0 IoPriority 2 PagePriority 5
Child-SP          RetAddr           : Args to Child                                                           : Call Site
fffffa60`089b4a58 fffff800`01e63661 : 00000000`00000050 fffffa60`08c4d718 00000000`00000008 fffffa60`089b4b50 : nt!KeBugCheckEx
fffffa60`089b4a60 fffff800`01e53219 : 00000000`00000008 fffffa60`019d8180 fffffa80`273ed000 fffffa80`2b379010 : nt!MmAccessFault+0x1371
fffffa60`089b4b50 fffffa60`08c4d718 : fffffa60`0839537b fffffa80`273ed040 00000000`00000080 fffffa60`0839e350 : nt!KiPageFault+0x119 (TrapFrame @ fffffa60`089b4b50)
fffffa60`089b4ce8 fffffa60`0839537b : fffffa80`273ed040 00000000`00000080 fffffa60`0839e350 fffffa80`2b379010 : <Unloaded_TmPreFlt.sys>+0x4718
fffffa60`089b4cf0 fffff800`020788b3 : 00000000`00000001 00000000`0000000f 00000000`00000000 fffffa80`2b379010 : rpcxdr!RxWorkThread+0x3f3
fffffa60`089b4d50 fffff800`01e8e7f6 : fffffa60`01966180 fffffa80`273ed040 fffffa60`0196fd40 00000000`00000001 : nt!PspSystemThreadStartup+0x57
fffffa60`089b4d80 00000000`00000000 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : nt!KxStartSystemThread+0x16

 

[niemiro]

Thanks a lot for your help. Microsoft provide some very neat functions for accessing structures with symbols. I will post an update soon.

Thanks again for your help.

 

[niemiro]

@usasma: Your crash report is still a work in progress and consequently there is no fix for it in this update. Sorry.

---

However, Vir Gnarus, I think that your problem should now be fixed.

Changelog:

[B]v1.0.0.0
[/B]Initial Release

[B]v1.1.0.0
FIXED:[/B] Extension no longer reports range errors on 32bit targets.
[B]FIXED:[/B] Extension no longer hangs or reports range errors when the entire raw stack lies in unavailable memory.
[B]FIXED:[/B] Extension no longer sometimes loses the first or last available memory address in the raw stack on 32bit targets.
[B]FIXED:[/B] File details are now correct.

[B]v1.2.0.0[/B]
[B]FIXED: [/B]Extension now calculates offset from start of nt!_KTHREAD structure using symbol information to correct incorrect offset calculation on certain platforms.

Latest source code also included.

 

[Vir Gnarus]

Works like a charm. Thanks!

 

[niemiro]

No problem. Glad to help. Couldn't have done any of this without you.

 

[Vir Gnarus]

I certainly wouldn't have anything to be thankful for without yer programming skills. This definitely will help expedite things a lot for me. I'll have to think up other options in the future. Again, thanks a mil.

 

[niemiro]

I certainly wouldn't have anything to be thankful for without yer programming skills. This definitely will help expedite things a lot for me. I'll have to think up other options in the future. Again, thanks a mil.

Definitely. If you have any more ideas for extensions, please just throw them at me (and that goes for anybody with ideas). I will probably create a separate thread of this once I have discussed a little more with writhziden first.

I may also be able to do some more complicated stuff now also. I am really started to get used to the API and how extensions are written, so more complicated stuff is no longer completely out of the question.

 

[writhziden]

Running Windows 8 Release Preview (64 bit)
Had an app crash :0(
using v1.0.0.0
Will try 1.1.0.0 next

Copied x64 dll to x64 directory
Copied x86 dll to x86 directory
Ran using Mike's scripts (used all 3 of your sample dumps at one time)

Got this error:

Application: _v2120_SysnativeBSODApps.exe
Framework Version: v4.0.30319
Description: The process was terminated due to an unhandled exception.
Exception Info: System.AccessViolationException
Stack:
at <Module>.std.basic_string<char,std::char_traits<char>,std::allocator<char> >.assign(std.basic_string<char,std::char_traits<char>,std::allocator<char> >*, std.basic_string<char,std::char_traits<char>,std::allocator<char> >*, UInt32, UInt32)
at <Module>.std.basic_string<char,std::char_traits<char>,std::allocator<char> >.{ctor}(std.basic_string<char,std::char_traits<char>,std::allocator<char> >*, std.basic_string<char,std::char_traits<char>,std::allocator<char> >*, UInt32, UInt32, std.allocator<char>*)
at <Module>.std.basic_string<char,std::char_traits<char>,std::allocator<char> >.substr(std.basic_string<char,std::char_traits<char>,std::allocator<char> >*, std.basic_string<char,std::char_traits<char>,std::allocator<char> >*, UInt32, UInt32)
at <Module>.InputData.currentSpeedString(InputData*, Int32)
at <Module>.InputData.getImportantLines(InputData*, Int32)
at <Module>.InputData.getImportantInfo(InputData*)
at <Module>.OutputDmps.getInputData(OutputDmps*)
at <Module>.OutputDmps.outputAnalysisFiles(OutputDmps*)
at <Module>.main(System.String[])

and this one:

Faulting application name: _v2120_SysnativeBSODApps.exe, version: 2.1.2.0, time stamp: 0x505b4084
Faulting module name: unknown, version: 0.0.0.0, time stamp: 0x00000000
Exception code: 0xc0000005
Fault offset: 0x05b1a55a
Faulting process id: 0x1a8c
Faulting application start time: 0x01cd9a88253792bc
Faulting application path: C:\Users\John\_jcgriff2_\dbug\__Kernel__\_v2120_SysnativeBSODApps.exe
Faulting module path: unknown
Report Id: d8db0890-067b-11e2-9b9b-485b3901d3fe
Faulting package full name:
Faulting package-relative application ID:

Would it be possible to get the outputDmps and the tmp directories generated in this error?

If I could see those, it would really help narrow down why there might be a problem in the apps. It appears that there may be a current speed string showing up in the output, but it may not be in the format expected by the apps. Hard to say without the output, though.

 

[jcgriff2]

No idea if this can be of any use here or not, but thought I'd post it & you can decide.

You can create a text file with commands & variables, then run from Windbg kd>

.cmdtree <file>

Content of <file> = jcgriff2_cmdtree_01-03-2010.txt -

Read More:

 windbg ANSI Command Tree 1.0

title {"Common Commands"}

body

{"Common Commands"}

 {"Information"}

  {"Time of dump"} {".time"}

  {"Process being debugged"} {"|"}

  {"Dump Location"} {"||"}

  {"Create server on port 9999"} {".server tcp:port=9999"}

  {"Show remote connections"} {".clients"}

  {"Process Environment Block"} {"!peb"}

 {"Logging"}

  {"Open Log"} {".logopen /t /u /d"}

  {"Close Log"} {".logclose"}

 {"Modules"}

  {"All Modules"} {"lm D sm"}

  {"Loaded Modules"} {"lmo D sm"}

  {"Loaded Modules (verbose)"} {"lmvo D sm"}

  {"Modules w/o symbols"} {"lme D sm"}

 {"Stacks"}

  {"Set frame length to 2000"} {".kframes 2000"}

  {"Dump current stack w/ DML"} {"kpM 1000"}

  {"Dump stacks without private info"} {"knL 1000"}

  {"Dump stacks with all parameters"} {"kPn 1000"}

  {"Dump stacks (distance from last frame)"} {"kf 1000"}

  {"Dump stacks with Frame Pointer Omission"} {"kvn 1000"}

  {"Dump all stack"} {"~*kbn 1000"}

  {"Dump unique stacks"} {"!uniqstack -pn"}

  {"Thread environment block"} {"!teb"}

  {"Move to next frame"} {".f+"}

  {"Move to previous frame"} {".f-"}

 {"Memory"}

  {"Dump heaps"} {"!heap -a"}

 {"Automated Task"}

  {"!analyze"} {"!analyze -v"}

  {"Locks"} {"!ntsdexts.locks"}

  {"CPU time for User and Kernel Mode"} {"!runaway 7"}

 {"Managed"}

  {"Load sos"} {".loadby sos mscorwks"}

  {"clrstack"} {"!clrstack"}

  {"Threads"} {"!threads"}

  {"Stack Objects"} {"!dso"}

  {"Exceptions"} {"!dae"}

 {"jcgriff2"}
  
  {"FOR 1 Index"} {"!for_each_module .echo @#ModuleIndex : @#Base @#End @#ModuleName @#ImageName  @#LoadedImageName"}

  {"FOR 2 Versions"} {"!for_each_module .echo @#ModuleName fver = @#FileVersion pver = @#ProductVersion"}

Carrona gave this to me years ago. I made some changes, including adding the last few lines -

 {"jcgriff2"}
  
  {"FOR 1 Index"} {"!for_each_module .echo @#ModuleIndex : @#Base @#End @#ModuleName @#ImageName  @#LoadedImageName"}

  {"FOR 2 Versions"} {"!for_each_module .echo @#ModuleName fver = @#FileVersion pver = @#ProductVersion"}

Running kd> .cmdtree <file> -- screenshot:

​

Click on a line in the .cmdtree screen & it will run the associated command in the text file.

cmdtree text file - View attachment jcgriff2_cmdtree_01-03-2010.txt

Google search .cmdtree - https://www.google.com/search?num=1...3.1403.0j6j2.8.0.les;..0.0...1c.1.HxY5QQBn6yI


To use ENV variables in Windbg, start with as command to assign the variable -

3: kd>[B] as /e [COLOR="#800080"]up [/COLOR][COLOR="#0000FF"]userprofile[/COLOR][/B]

3: kd>[B] .echo [COLOR="#800080"]${up}[/COLOR] [/B]
C:\Users\PalmDesert

3: kd> [B].echo [COLOR="#800080"]${up}[/COLOR]\SysnativeBSODApps\jcgriff2_cmdtree_01-03-2010.txt[/B]
C:\Users\PalmDesert\SysnativeBSODApps\jcgriff2_cmdtree_01-03-2010.txt

However, I can't get cmdtree to work with ${up} variable I just assigned (it works with ECHO command) -

.cmdtree [B]${up}[/B]\SysnativeBSODApps\jcgriff2_cmdtree_01-03-2010.txt

This works OK:

.cmdtree c:\users\PalmDesert\SysnativeBSODApps\jcgriff2_cmdtree_01-03-2010.txt

I thought there may be something useful here... maybe with .cmdtree itself or perhaps the use of variables & to show you the syntax (if you didn't already know!)

John

 

[mgrzeg]

Try ${$up}

m.g.

 

[jcgriff2]

Error: The file ${$up}\SysnativeBSODApps\jcgriff2_cmdtree_01-03-2010.txt cannot be opened.

Same error using ${up}\..... - with .cmdtree

EDIT:

3: kd> .echo ${$up}
${$up}

3: kd> .echo ${up}
C:\Users\PalmDesert

 

[mgrzeg]

OK, maybe this one:
{1} as /e up USERPROFILE; .block {.echo ${up}\SysnativeBSODApps\jcgriff2_cmdtree_01-03-2010.txt}
{2} as /e up USERPROFILE; .block {.cmdtree ${up}\SysnativeBSODApps\jcgriff2_cmdtree_01-03-2010.txt}

EDIT:
nope, doesn't work either. ( I think .cmdtree doesn't work with aliasses

EDIT2:
BTW - for managed, I suggest using psscor2 (psscor4) instead of sos. Add to this sosex and you've got almost everything you need :)
Then use !analysis (available in psscor2(4))

m.g.

 

[jcgriff2]

I think you're right -- alias.

I haven't used the .block command yet -- THANKS!

Read More:

The .block token performs no action; it is used solely to introduce a block of statements.

Syntax
Commands ; .block{ Commands } ; Commands


Comments
Blocks of commands are surrounded by braces. When each block is entered, all aliases within the block are evaluated. If you alter the value of an alias at some point within a command block, commands subsequent to that point will not use the new alias value unless they are within a subordinate block.

Each block must begin with a control flow token. If you wish to create a block for the sole purpose of evaluating aliases, you should prefix it with the .block token, since this token has no effect other than to allow a block to be introduced.

Additional Information

For information about using a new block to evaluate an alias, see Using Aliases and as, aS (SetAlias).

For information about other control flow tokens and their use in debugger command programs, see Using Debugger Commands.

 

[usasma]

Here's the Output dumps from the 2 tests (attached)
Could not locate any temp file from yesterday.

 

[usasma]

Interesting! I ran the dumps again (so I could see if I got a temp folder) - and they completed without errors!
Here's the output dump folder for it (attached) Wasn't able to find a temp folder

 

[writhziden]

Strange that it would complete fine now... :confused2:

If it fails again, please provide the tmp folder. The tmp folder is deleted if the apps finish, but if the apps crash, the tmp directory has a lot of useful info within for determining why the apps may have crashed.


EDIT: Found the problem. Solved in the latest release: BETA: Sysnative BSOD Apps 2.1.2.1

Your latest upload was a big help. Thanks John!

 

[usasma]

I've changed nothing since installing the app - haven't even rebooted!
I'll save the tmp folder if it crashes again.
Thanks!

BTW - the analysis of the STOP 0x124 doesn't appear right (missing symbols I think) - it's the 3rd text file
Could you take a look at that when you have time?

 

[usasma]

I've just upgraded to v2121 of the app and got a hang on the 5/28/12 memory dump
Attached is the tmp folder and the dump folder.....

Just noted this from the KD box in the app:

{1} !niemiro.rawstack
{2} !niemiro.rawstack -dc
{3} {124} !niemiro.auto_errrec

I had tried to rename the {3} to {124}
It seems that the app added the {3} back in

I'll try it without the renaming....
************************************************
Note to self - don't screw around with stuff that you know nothing about!
The app ran fine when I fixed the curly brackets thing.

Still having symbol issues in the user3.txt file tho'

 

[niemiro]

BTW - the analysis of the STOP 0x124 doesn't appear right (missing symbols I think) - it's the 3rd text file
Could you take a look at that when you have time?

Curiously, it works fine for me.

Since this is the BSOD app, and you do not have it set up to connect to the Microsoft Symbol Server, can you please check that the following files exist in C:\Symbols for you?

\PSHED.dll\4A5BE02714000\PSHED.dll
\pshed.pdb\F1B5874289B044199CE35519D256D3E32\pshed.pdb

This will absolutely rule out an incompletely cache, as these are the versions the dump you ran used. I admit that this is unlikely though, you run so many dumps for so many users. I will continue to think of other explanations. Does it work in WinDBG for you? Thank you :)