[SOLVED] BSODs on 25-30 Computers

Hi turtlej0e, I hope you had a good long weekend.

I'm not sure the DV special pool settings were sufficient. It does seem to be another instance of a graphics related object being double freed. The function call in which it crashed is ExFreePoolWithTag and the pool tag was Gpft. Gpft is described as "Gdi font table" so likely another Windows driver. However, the bug check code 0x139 suggests a list structure got corrupted. I suppose the freed memory could have been given to the list after having been previously freed and then the next free corrupted it. I'm not sure, though. Perhaps someone with more experience investigating these problems will have better advice or can glean more information from the dump file but in my mind I'd probably try DV again with special pool on for all drivers, including those from Microsoft. That way no driver can reuse freed memory while special pool is enabled.

edit: It does look like a driver is allocating pool memory with no tag and did so 26 times. I can't tell which driver, though, which is why you're supposed to use a pool tag. It's not necessarily causing the problem, though.
 
cwsink said:
Hi turtlej0e, I hope you had a good long weekend.

I'm not sure the DV special pool settings were sufficient. It does seem to be another instance of a graphics related object being double freed. The function call in which it crashed is ExFreePoolWithTag and the pool tag was Gpft. Gpft is described as "Gdi font table" so likely another Windows driver. However, the bug check code 0x139 suggests a list structure got corrupted. I suppose the freed memory could have been given to the list after having been previously freed and then the next free corrupted it. I'm not sure, though. Perhaps someone with more experience investigating these problems will have better advice or can glean more information from the dump file but in my mind I'd probably try DV again with special pool on for all drivers, including those from Microsoft. That way no driver can reuse freed memory while special pool is enabled.

edit: It does look like a driver is allocating pool memory with no tag and did so 26 times. I can't tell which driver, though, which is why you're supposed to use a pool tag. It's not necessarily causing the problem, though.
Click to expand...

Okay! One thing I did forget to mention is that right before the BSOD, it displayed the interactive logon banner for a split second, and then crashed. I know you asked if we had a custom log off/log on process, and the only thing would be that login banner, but it is not "custom" per say. It's included in Group Policy by MS.

I have DV running with Special Pool enabled for all drivers. I'll post another dump next time it crashes. Thanks for your help!
 
cwsink said:
Hi turtlej0e, I hope you had a good long weekend.
Perhaps someone with more experience investigating these problems will have better advice or can glean more information from the dump file but in my mind I'd probably try DV again with special pool on for all drivers, including those from Microsoft.
Click to expand...

Alright, so I received another crash, but this time it appears DV caught something since the bugcheck code is different then usual (0x000000D5). Here it is: https://goo.gl/UqYJSy

It happened when I was signing out while testing the removal of the login banner GP. Hopefully that dump will be more helpful!
 
I was hoping DV would make it obvious but it doesn't seem to have done so. The meat of it for this particular dump seems to be an attempt to read special pool memory after it has already been freed. Surprisingly, I don't see anything involving 3rd party drivers but I'm still investigating. It does appear that there are font related calls being made shortly before the crash but I'm having trouble finding documentation for the win32kbase functions.

It looks like it's happening in the csrss.exe process during shutdown but either it's happening long after the initial allocation is being made (therefore the allocation has aged out from the log), the initial allocation isn't being logged at all, or I'm doing something wrong. Using the memory location referenced from the 1st parameter of the bug check:

2: kd> !verifier 80 fffffb810037c220

Log of recent kernel pool Allocate and Free operations:

There are up to 0x10000 entries in the log.

Parsing 0x0000000000010000 log entries, searching for address 0xfffffb810037c220.


======================================================================
Pool block fffffb810037c200, Size 0000000000000e00, Thread ffffe58d99cbc080
fffff802765b963f nt!VfFreePoolNotification+0x5b
fffff80276087afd nt!ExpFreePoolChecks+0x81
fffff802760b7023 nt!ExFreePoolWithTag+0x1283
fffff802765a84ea nt!VerifierExFreePoolWithTag+0x4a
fffffbbf06ec56ac win32kfull!Win32FreePoolImpl+0x4c
fffffbbf071dc9ac win32kbase!Win32FreePool+0x1c
fffffbbf07210dbc win32kbase!PDEV::Free+0x30
fffffbbf0720f787 win32kbase!vUnreferencePdevWorker+0x2d7
fffffbbf071e0d42 win32kbase!PDEVOBJ::vUnreferencePdev+0xe2
fffffbbf071ecb19 win32kbase!vDeleteDCInternalWorker+0x549
fffffbbf071e2e18 win32kbase!bDeleteDCOBJ+0x2ac
fffffbbf071e2b2d win32kbase!bDeleteDCInternalEx+0x4d
fffffbbf071ada85 win32kbase!DestroyCacheDC+0xd5

Finished parsing all pool tracking information.

It found when it was freed but not when it was allocated. Looking at the assembly prior to the code that attempted to access the freed memory:

2: kd> .trap fffff58f`e0ed6fa0
NOTE: The trap frame does not contain all registers.
Some register values may be zeroed or incorrect.
rax=0000000000000000 rbx=0000000000000000 rcx=ffffe58d99cbc080
rdx=fffffb8100402170 rsi=0000000000000000 rdi=0000000000000000
rip=fffffbbf071e6efe rsp=fffff58fe0ed7130 rbp=fffff58fe0ed71d9
r8=fffff58fe0ed7138 r9=0000000000000001 r10=7ffffffffffffffc
r11=0000000000000000 r12=0000000000000000 r13=0000000000000000
r14=0000000000000000 r15=0000000000000000
iopl=0 nv up ei pl zr na po nc
win32kbase!XDCOBJ::bCleanDC+0x7fe:
fffffbbf`071e6efe 8b5320 mov edx,dword ptr [rbx+20h] ds:00000000`00000020=????????
2: kd> ub
win32kbase!XDCOBJ::bCleanDC+0x7db:
fffffbbf`071e6edb ff158fce1600 call qword ptr [win32kbase!_imp_ExReleasePushLockExclusiveEx (fffffbbf`07353d70)]
fffffbbf`071e6ee1 ff1599ce1600 call qword ptr [win32kbase!_imp_KeLeaveCriticalRegion (fffffbbf`07353d80)]
fffffbbf`071e6ee7 ff1593ce1600 call qword ptr [win32kbase!_imp_KeLeaveCriticalRegion (fffffbbf`07353d80)]
fffffbbf`071e6eed 488b1f mov rbx,qword ptr [rdi]
fffffbbf`071e6ef0 488b5b30 mov rbx,qword ptr [rbx+30h]
fffffbbf`071e6ef4 ff153ea31600 call qword ptr [win32kbase!_imp_IsXDCOBJ_vSetDefaultFontSupported (fffffbbf`07351238)]
fffffbbf`071e6efa 85c0 test eax,eax
fffffbbf`071e6efc 780f js win32kbase!XDCOBJ::bCleanDC+0x80d (fffffbbf`071e6f0d)

The call to a font related function. I'm still looking but wanted to see if anyone has any insight or recommendations.
 
cwsink said:
I was hoping DV would make it obvious but it doesn't seem to have done so. The meat of it for this particular dump seems to be an attempt to read special pool memory after it has already been freed. Surprisingly, I don't see anything involving 3rd party drivers but I'm still investigating. It does appear that there are font related calls being made shortly before the crash but I'm having trouble finding documentation for the win32kbase functions.
Click to expand...

Thanks! We disabled the login banner for a couple days but sadly it did not make a difference. We'll keep digging on our end to see if we can find something graphics/font related. Do you think running PoolMon would be helpful for troubleshooting in this situation?
 
I'm not very knowledgable with WinDBG, or with crash dumps in general, but I sorted the pool tags by the unpaged pool usage and saw the tag "ismc" was using the most - around 135MB. I used findstr to search for that tag in system32\drivers and it was found in iaStor.sys, iaStorAV.sys, and SET74A.tmp. AFAIK the ia*.sys drivers are for the SATA controller, but I'm not sure what SET74A.tmp is. Is this relevant at all, or is it common to see that much memory used under a pool tag?
Code: 
[COLOR=#0000ff]kd> !poolused 2
....
 Sorting by NonPaged Pool Consumed

               NonPaged                  Paged
 Tag     Allocs         Used     Allocs         Used

[/COLOR][COLOR=#ff0000] ismc         3    134098944          0            0	UNKNOWN pooltag 'ismc', please update pooltag.txt[/COLOR][COLOR=#0000ff]
 ConT       689     53092352          0            0	UNKNOWN pooltag 'ConT', please update pooltag.txt
 dump         2     10719312          0            0	UNKNOWN pooltag 'dump', please update pooltag.txt
 EtwB       187     10670112          8       339968	Etw Buffer , Binary: nt!etw
 MmPb         3      8802304          0            0	Paging file bitmaps , Binary: nt!mm
 VfPT         1      8388608          0            0	Verifier Allocate/Free Pool stack traces , Binary: nt!Vf
 File     15220      6456960          0            0	File objects [/COLOR]
 
If you have Process Explorer you can see your pool memory use in real time on the System Information (Ctrl+I) Memory tab dialog. I would imagine your computer has a huge amount of nonpaged pool available. The amount used by that driver tag seems like a lot but I don't get the impression this issue is a problem of running out of pool memory. Unless I've misunderstood, the crash is only happening while shutting down or restarting and not while using the system. If it was running out of pool it would crash during regular use as well. That suggests to me something has stored a reference to an object which thinks that pool allocation is still valid and later, when told to clean up, tries to access that memory (perhaps to free it) which causes the system to crash. That tag isn't showing up in 3 of the 7 dumps you've made available so I don't think it's that but if you have Intel Rapid Storage Technology installed I'd be curious to know if that tag disappears from the list if you uninstall IRST. Totally optional; just curious.

If you execute !thread in the dumps you'll see they are all running in Microsoft processes and all crashing while either freeing pool or cleaning up device contexts (which are probably freeing memory as well.)

One crash happened after less than 50 minutes; some are on systems which have been running for days. But none longer than a week. Are there any scheduled tasks running once a week which have been added through GP?

Windows Internals 6th edition part 2 chapter 13 discusses the shutdown process. It talks about what csrss does during shutdown which is the process running when the crash happens in 4 of the dumps. If you haven't already, it might be worth a read to see if it can shed some light on what might be happening.

If you can get a DV enabled dump using the latest settings we came up with which is more in the 20 to 30 minute uptime range without the computer being used (to minimize pool allocations) we might be able to get more from the DV log.

I haven't used poolmon, myself. Hopefully someone else who has used it can say whether or not it would be usefull. I'm still thinking about it and if I come up with a tool that seems like it will help I'll let you know.​
 
cwsink said:
Unless I've misunderstood, the crash is only happening while shutting down or restarting and not while using the system. If it was running out of pool it would crash during regular use as well.​
Click to expand...

Correct. It doesn't happen during daily use, only at logoff, shutdown, or restart. And that makes sense - it was just something I noticed when I was trying to read the dumps. :)

One crash happened after less than 50 minutes; some are on systems which have been running for days. But none longer than a week. Are there any scheduled tasks running once a week which have been added through GP?
Click to expand...

I believe there is a scheduled task to run the Windows defragger weekly. That would be the only task through GP though.

I skimmed the chapter from Windows Internals, but I will go back and study it a little more later.

If you can get a DV enabled dump using the latest settings we came up with which is more in the 20 to 30 minute uptime range without the computer being used (to minimize pool allocations) we might be able to get more from the DV log.
Click to expand...

I am doing this on another computer right now - hopefully it will crash in that time period!
 
Ok, I just found out that the crash can be triggered after printing a document from Word, Excel, or Foxit Reader. The documents don't need to be large or complex. I just put the word "test" in them and printed them. I then closed the program, and then signed out. It then crashes immediately after signing out. I also tried printing from Notepad, Wordpad, and Chrome, but those don't trigger the crash. I was able to get a crash dump from a laptop running DV w/ Special Pool and no antivirus.

The weekend is almost here, so next week I'll start investigating more.

Thank you for all your help! It feels good to finally know how to trigger it. It's almost embarrassing that it took this long to figure that out!

Dump: WeTransfer
 
That's a good catch and testing, well done. Printing definitely involves working with device contexts. My suspicion would be a Foxit bug since it likely installed Office plug-ins. If there is no update for Foxit I'd probably try uninstalling it and any of it's Office plugins to see if the problem goes away.

Other possibilities would be outdated printer software/drivers but I would expect that to be a problem with Chrome unless printing from Chrome doesn't involve fonts somehow. Does it happen if you print to PDF? Or perhaps it's using different fonts and there's a problem with the fonts themselves. I've seen posts about bug checks where the problem was supposedly 3rd party fonts and/or corrupted fonts.

Whatever it is, it only seems to be a problem during cleanup so likely a destructor being called that is referencing freed memory. The Windows kernel drivers involved probably don't check if the memory has been freed; if at all, probably only checking if it's a null reference.
 
I too was receiving Win32kfull.sys BSOD and I narrowed it down to TeamViewer. Funny because i found this post this morning by searching google for Win32kfull.sys and TeamViewer.

I could have teamviewer open for hours and than experience a BSOD out of no where.

Running 13.0.6447
 
cwsink said:
That's a good catch and testing, well done. Printing definitely involves working with device contexts. My suspicion would be a Foxit bug since it likely installed Office plug-ins. If there is no update for Foxit I'd probably try uninstalling it and any of it's Office plugins to see if the problem goes away.

Other possibilities would be outdated printer software/drivers but I would expect that to be a problem with Chrome unless printing from Chrome doesn't involve fonts somehow. Does it happen if you print to PDF? Or perhaps it's using different fonts and there's a problem with the fonts themselves. I've seen posts about bug checks where the problem was supposedly 3rd party fonts and/or corrupted fonts.

Whatever it is, it only seems to be a problem during cleanup so likely a destructor being called that is referencing freed memory. The Windows kernel drivers involved probably don't check if the memory has been freed; if at all, probably only checking if it's a null reference.
Click to expand...

Tried removing Foxit, but it didn't make a difference. It appears to only happen when printing to our Kyocera printers. We updated the print drivers, and we can't seem to make it blue screen on two different computers. We'll wait a couple days to see how it affects other computers. *crosses fingers* I'll keep you posted. Thanks again!
 
brink668 said:
I too was receiving Win32kfull.sys BSOD and I narrowed it down to TeamViewer. Funny because i found this post this morning by searching google for Win32kfull.sys and TeamViewer.

I could have teamviewer open for hours and than experience a BSOD out of no where.

Running 13.0.6447
Click to expand...

Interesting! Were you able to solve the issue? We have been using TV13 on all our computers and haven't had an issue caused by it.
 
turtlej0e said:
Tried removing Foxit, but it didn't make a difference. It appears to only happen when printing to our Kyocera printers. We updated the print drivers, and we can't seem to make it blue screen on two different computers. We'll wait a couple days to see how it affects other computers. *crosses fingers* I'll keep you posted. Thanks again!
Click to expand...

That sounds promising. Thank you for the update and please do let us know how it goes.
 
Well, it appears to have made a difference on some computers over the past few days, but we still have quite a few computers getting the BSOD. I'm going to look at those closer and see if they grabbed the latest driver or have some residue from the old driver. Have a great weekend!
 
turtlej0e said:
Well, it appears to have made a difference on some computers over the past few days, but we still have quite a few computers getting the BSOD. I'm going to look at those closer and see if they grabbed the latest driver or have some residue from the old driver. Have a great weekend!
Click to expand...

Thank you for the update. If you have any ideas you want to bounce around, feel free. Is there something about the system configurations separating those which are no longer having the problem from those that are? Make and model, for example? Is printing guaranteed to cause problems on the systems still experiencing the issue?

Enjoy your weekend as well!
 
cwsink said:
turtlej0e said:
Well, it appears to have made a difference on some computers over the past few days, but we still have quite a few computers getting the BSOD. I'm going to look at those closer and see if they grabbed the latest driver or have some residue from the old driver. Have a great weekend!
Click to expand...

Thank you for the update. If you have any ideas you want to bounce around, feel free. Is there something about the system configurations separating those which are no longer having the problem from those that are? Make and model, for example? Is printing guaranteed to cause problems on the systems still experiencing the issue?

Enjoy your weekend as well!
Click to expand...

Good morning!

Doesn't seem to be any related hardware, and I'm not seeing any certain software version that is common across the working computers. Printing still appears to cause the computer to crash on the systems still experiencing the issue. We tried disabling options in the printer properties on the printer server (e.g. Job Accounting, bi-directional support) but that didn't make a difference. Currently, what we are trying is just removing the printer and adding it again. It's possible there is some settings that get applied at re-installation that don't get applied when the driver updates.
 
So I did some more testing, and I can trigger the blue screen by printing from the following programs:

  • Microsoft Word, Excel
  • Internet Explorer
  • Foxit Reader
  • Firefox

However I cannot trigger it from:
  • Google Chrome
  • Notepad
  • Wordpad

I discovered if "Render print jobs on client computers" is disabled on the Kyocera printer, I can print without any issues. It must have something to do with how the computer is rendering the document from certain programs. We could disable the option, but it seems more of a workaround then an actual fix - it had worked fine in the past. Unless something changed in the print rendering process in the Fall Creators Update that Kyocera hasn't supported yet.
 
What happens if you try to print a picture opened up in Firefox without any fonts involved? Like a jpeg or bitmap.
 

Has Sysnative Forums helped you? Please consider donating to help us support the site!

Back
Top