BSOD Analysis Help!

[azhang253]

• A brief description of your problem (but you can also include the steps you tried)
  • 6 Machine cluster - one of the nodes seemed to be hung, causing issues with the cluster.
  • One of the node was hung - showed Isolated FCM
  • Login via iDrac - black screen.
  • In order to bring system up, I shut down node from iDrac. Cluster failed over resources automatically and all VM's started on other nodes.
• System Manufacturer?
  • Dell
• Laptop or Desktop?
  • Server
• Exact model number (if laptop, check label on bottom)
  • M640
• OS ? (Windows 11, 10, 8.1, 8, 7, Vista)
  • Server 2022
• x86 (32bit) or x64 (64bit)?
  • x64
• What was original installed OS on system?
  • None
• Is the OS an OEM version (came pre-installed on system) or full retail version (YOU purchased it from retailer)?
  • No
• Age of system? (hardware)
  • Unsure, probably 5-6 year?
• Age of OS installation?
  • 2 years
• Have you re-installed the OS?
  • Yes, clean install in 2022.
• CPU
  • Xeon Gold 5217 x2
• RAM (brand, EXACT model, what slots are you using?)
  • Unsure of exact brand, all 16 slots are used
• Video Card
  • None/Integrated
• MotherBoard - (if NOT a laptop)
  • Dell
• Power Supply - brand & wattage (if laptop, skip this one)
  • Multiple inside Dell M1000E Chassis
• Is driver verifier enabled or disabled?
  • Unsure
• What security software are you using? (Firewall, antivirus, antimalware, antispyware, and so forth)
  • SentinelOne
• Are you using proxy, vpn, ipfilters or similar software?
  • No
• Are you using Disk Image tools? (like daemon tools, alcohol 52% or 120%, virtual CloneDrive, roxio software)
  • No, but our backup software Rubrik uses VSS
• Are you currently under/overclocking? Are there overclocking software installed on your system?
  • No

 

[azhang253]

http://speccy.piriform.com/results/RDlWVIBUEf4hivwnuBDtgQQ

 

[xilolee]

Hi!

You should have a memory dump in C:\Windows\MEMORY.DMP; please, upload it.

Read More:

USER_MODE_HEALTH_MONITOR (9e)
One or more critical user mode components failed to satisfy a health check.
Hardware mechanisms such as watchdog timers can detect that basic kernel services are not executing.
However, resource starvation issues, including memory leaks, lock contention, and scheduling priority misconfiguration, may block critical user mode components without blocking DPCs or draining the nonpaged pool.
Kernel components can extend watchdog timer functionality to user mode by periodically monitoring critical applications.
This BugCheck indicates that a user mode health check failed in a manner such that graceful shutdown is unlikely to succeed.
It restores critical services by rebooting and/or allowing application failover to other servers.
Arguments:
Arg1: ffffb702f6a542c0, Process that failed to satisfy a health check within the
    configured timeout
Arg2: 00000000000004b0, Health monitoring timeout (seconds)
Arg3: 0000000000000065, WatchdogSourceRhsResourceDeadlockPhysicalDisk
    A subclass of WatchdogSourceRhsResourceDeadlock.
    For the resource types that are known to cause RHS termination hangs we isolate them to separate codes so we can quickly triage dumps.
Arg4: 0000000000000000

*************************************************************************
***    Either you specified an unqualified symbol, or your debugger   ***
***    [CUT]                                                          ***
***    Type referenced: netft!NETFT_WATCHDOG_SOURCE                   ***
*************************************************************************
*** WARNING: Check Image - Checksum mismatch - Dump: 0x4cf37, File: 0x4d01d - d:\symbols\netft.sys\6F5C75A848000\netft.sys
*** WARNING: Check Image - Checksum mismatch - Dump: 0x4cf37, File: 0x4d01d - d:\symbols\netft.sys\6F5C75A848000\netft.sys

BUGCHECK_CODE:  9e

BUGCHECK_P1: ffffb702f6a542c0

BUGCHECK_P2: 4b0

BUGCHECK_P3: 65

BUGCHECK_P4: 0

FILE_IN_CAB:  123024-38859-01.dmp

PROCESS_NAME:  rhs.exe

IMAGE_NAME:  rhs.exe

MODULE_NAME: rhs

FAULTING_MODULE: 0000000000000000

STACK_TEXT: 
fffff583`0553f078 fffff804`44cc5294     : 00000000`0000009e ffffb702`f6a542c0 00000000`000004b0 00000000`00000065 : nt!KeBugCheckEx
fffff583`0553f080 fffff804`44cc4a1c     : 00000000`00000000 ffffbf00`0753b610 ffffbf0f`786a4b80 ffffbf0f`786a4c10 : netft!NetftProcessWatchdogEvent+0x368
fffff583`0553f0f0 fffff804`22831fc4     : 00000000`00000002 fffff583`0553f1e9 ffffbf0f`7865e2f0 ffffa501`e6dc0180 : netft!NetftWatchdogTimerDpc+0x4c
fffff583`0553f120 fffff804`22830a64     : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`0708cb7f : nt!KiProcessExpiredTimerList+0x204
fffff583`0553f250 fffff804`22a2751e     : ffffffff`00000000 ffffa501`e6dc0180 ffffa501`e6dcc840 ffffbf0f`78cba080 : nt!KiRetireDpcList+0xa14
fffff583`0553f500 00000000`00000000     : fffff583`05540000 fffff583`05539000 00000000`00000000 00000000`00000000 : nt!KiIdleLoop+0x9e

STACK_COMMAND:  .cxr; .ecxr ; kb
FAILURE_BUCKET_ID:  0x9E_65_IMAGE_rhs.exe
OS_VERSION:  10.0.20348.2849
BUILDLAB_STR:  fe_release_svc_prod1

(I'm trying to guess) It seems like it searched information about the storage (physical disk), but the storage wasn't available.
There are a lot of warnings created by disk, filtermanager and partmgr (several hours and also days before the BSOD).

Read More:

system - disk - event id 153: The IO operation at logical block address 0x11b800 for Disk XX (PDO name: \Device\00000bd3) was retried.
system - partmgr - event id 58: The disk signature of disk XX is equal to the disk signature of disk YY.
system - filtermanager - event id 3: Filter Manager failed to attach to volume '\Device\HarddiskXX\DRYYYY'. This volume will be unavailable for filtering until a reboot. The final status was 0xC03A001C.

One of them is (maybe) more interesting:

Read More:

system - disk - event id 158: Disk ZZ has the same disk identifiers as one or more disks connected to the system. Go to Microsoft's support website and search for KB2983588 to resolve the issue.

Then I searched that KB2983588 but the first result is Event ID 158 for identical disk GUIDs - Windows Client.
While you wait for further opinions, you could try to fix this.

 

[azhang253]

Those disk errors are caused by many HyperV Backup solutions that use VSS (Ie Rubrik, when they create and a snapshot of a disk to take the backup). According to them, they can be ignored. I've seen them before, and I believe they only occur during my backup scheduled time. They are also on other HyperV hosts that have not BSOD'd. Today's BSOD occured when there were no backups happening. That being said, I'm not ruling out Rubrik causing BSOD's... after all Veeam did in the past too, and so did SentinelOne....

How should I upload the memory.dmp file? It's 36GB.

 

[xilolee]

Zipped and sent to google drive as an example.

 

[azhang253]

Zipped and sent to google drive as an example.

will do, zipping and uploading!

 

[azhang253]

Google Drive Link: MEMORY.zip

 

[ubuysa]

Hello, and welcome to the forum.

I'm not experienced at all with server systems and the two bugchecks in those two dumps I've not encountered before, so I'm going to flag @x BlueRobot and @axe0, who are way more experienced that I am in these areas.

What I can tell you is that the two bugchecks are a 0x9E USER_MODE_HEALTH_MONITOR and a 0x23 FAT_FILE_SYSTEM.

The 0x9E bugcheck indicates that a user-mode process failed a health check, in particular it failed a WatchdogSourceRhsResourceDeadlockPhysicalDisk. The failing user-mode process (as shown in that watchdog record) was rhs.exe - the Resource Hosting System which manages cluster resources. The failure would appear to have been a deadlock condition at the drive level perhaps?

The 0x23 bugcheck indicates a problem in a drive using the FAT file system, the drive in question appears to be labelled Device\MPIODisk4 if that helps. These two bugchecks would thus seem to be linked to a drive issue?

I won't attempt to go further because, as I said, server systems are an unknown beast to me. Hopefully @x BlueRobot and/or @axe0 will be along soon. I will continue to monitor this thread, partly in case I can help, but also as a learning experience.

 

[xilolee]

If this helps, you got three errors before the BSOD:

Read More:

• 30/12/2024 20:43:41 system - Microsoft-Windows-FailoverClustering - event id 1230: An invalid operation was attempted on an active network connection.
• 30/12/2024 20:45:02 system - Microsoft-Windows-FailoverClustering - event id 1230: The description for Event ID 1230 from source Microsoft-Windows-FailoverClustering cannot be found. Either the component that raises this event is not installed on your local computer or the installation is corrupted. You can install or repair the component on the local computer.
  If the event originated on another computer, the display information had to be saved with the event.
  The following information was included with the event:
  FS5200_007B_PROS
  Physical Disk
  clusres.dll
  STORAGE_GET_DISK_INFO_EX2
  The message resource is present but the message was not found in the message table
• 30/12/2024 21:05:01 (i.e., 20 minutes later) system - Microsoft-Windows-FailoverClustering - event id 4874: The description for Event ID 4874 from source Microsoft-Windows-FailoverClustering cannot be found. Either the component that raises this event is not installed on your local computer or the installation is corrupted. You can install or repair the component on the local computer.
  If the event originated on another computer, the display information had to be saved with the event.
  The following information was included with the event:
  C:\Windows\Cluster\rhs.exe
  3632
  1200 (seconds = 20 minutes)
  The message resource is present but the message was not found in the message table

(Time has been adjusted automatically by my system, real time is probably six hour less)
You should be able to create a log with Get-ClusterLog (FailoverClusters).