How to solve Components Scanner reporting thousands of missing registry keys

Also... the BSOD 0xC000021a issue has not been solved. I ran WinDbg on the dump and traced it to "the system could not find the file specified" but I didn't find the name of the file. The only way I can get it to boot right now is by disabling driver signing (which I'm doing on a copy of the VM that does not have internet access). I'm hoping that the resolution to the remaining issues will result in SFC /scannow being able to resolve whatever driver is causing this and allow me to turn back on driver signing checks.

I did find that MS had a SSU that caused this in KB5021233 for Win 10 20H2 and later (not directly applicable to WSE 2016). The info from MS was that this was caused by a driver mismatch. Certainly a possibility with all the missing components on this server. So I was considering trying to reapply the WSE2016 versions of the SSUs that occurred around the same time via offline dism if fixing winsxs and sfc /scannow doesn't solve the bsod.

Code: 
SYMBOL_NAME:  nt!NtSetSystemPowerState+b876
STACK_COMMAND:  .cxr; .ecxr ; kb
BUCKET_ID_FUNC_OFFSET:  b876
FAILURE_BUCKET_ID:  0xc000021a_SmpDestroyControlBlock_smss.exe_Terminated_c0000428_nt!NtSetSystemPowerState
OS_VERSION:  10.0.14393.4770
BUILDLAB_STR:  rs1_release
OSPLATFORM_TYPE:  x64
OSNAME:  Windows 10
FAILURE_ID_HASH:  {6ae6fe00-a198-89fa-6794-96f7f849b4bf}
Followup:     MachineOwner

5: kd> !PDE.da ffffbd0fa8faa070
The call to LoadLibrary(PDE) failed, Win32 error 0n2
    "The system cannot find the file specified."
Please check your debugger configuration and/or network access.

Also 'da ffffbd0fa8faa070 ffffbd0fa8fab070' shows something in the memory about 'ACPI/VMbus#0' but I don't know if that's related.
 
Last edited:
mjmeans said:
Is the correct way to resolve the missing member files reported in CBS.log [SR] entries to copy the winsxs package folders from the repairserver and apply them with an SFCFix.zip file using the "{ARCHIVE} %SystemRoot%\winsxs [DIR]" command?
Click to expand...
That sounds right to me, there is an example SFCFix.zip in this tutorial too: Troubleshooting 0x800f081f / 0x800f0982 Error

mjmeans said:
How should I resolve the files reported for missing CanonicalData\Catalogs that all have long hexadecimal codes instead of package names?
Click to expand...
It looks like the files exist but the Catalog keys do not, they should be available on a repair server - provided it has the same updates - under that same registry key. You can repair them using SFCFix or by any means which supports .reg imports or writing directly to registry. There is an option for this with the registry script builder tool.

mjmeans said:
Should I apply the winsxs fixes for the the [SR] entries in CBS.log before trying to resolve f! or Catalogs?
Click to expand...
It doesn't really matter what order you do them in to be honest, although, one thing to note, some of the things which ComponentsScanner reports are just warnings which can sometimes be ignored.
 
mjmeans said:
I ran WinDbg on the dump and traced it to "the system could not find the file specified" but I didn't find the name of the file. The only way I can get it to boot right now is by disabling driver signing (which I'm doing on a copy of the VM that does not have internet access). I'm hoping that the resolution to the remaining issues will result in SFC /scannow being able to resolve whatever driver is causing this and allow me to turn back on driver signing checks.
Click to expand...
It sounds like you might have a corrupted .cat file, if you repair the files mentioned in your CBS log then it might hopefully resolve the BSOD issue as well.

If you run the following command:

Code: 
Get-WindowsDriver -Online -All | ? {$_.BootCritical -eq $True}

It might help to narrow down the issue to a particular driver package. Obviously, you would need to write some form of script or program to compare the hashes of the drivers against the ones on a known good system.
 
I wrote a PowerShell script to read the CBS.log file, find the lines that refer to the files that it was unable to repair because the file not found, parse the line to get a partial package name (without the SHA), search the damaged server for a WinSxS folder name that matches in order to get the full name with the SHA, and if it's not found try to find the partial name on the repair server. If either are found, it copes the folder from the repair server to a local "missing" folder in the same location as the script.

I tried using SFCFix and a ZIP file with the WinSxS packages to be replaced on a copy of the running damaged server (exported the running machine to a different computer). SFCFix.exe forces a reboot. The initial reboot failed with a BSOD and I had to reboot a few times in order to be able to select the option to disable driver signature verification to get it to boot again to complete the fixes where SFCFix reported something about restoring permissions. SFC /scannow on the copy didn't show any improvement. When I rebooted again, the BSOD symptom was not solved.

I'm worried that this forced reboot will result in another round of unrecoverable BSODs and a need to revert back to the 7/7 backup and reapply all the missing DerivedData and other registry fixes all over again.

Is there any reason why I should use SFCFix and a zip file containing the WinSxS folders instead of just taking ownership of the appropriate package folders and copying them myself?
 
Last edited:
mjmeans said:
Is there any reason why I should use SFCFix and a zip file containing the WinSxS folders instead of just taking ownership of the appropriate package folders and copying them myself?
Click to expand...
Because you need to make sure that you can reliably restore the permissions and ownership afterwards. SFCFix takes a backup of the current ownership and permissions, changes them so SFCFix can run and then restores them back using the backup file it created. It then recursively does this for all of the payload files and parent directories.

You could try taking ownership and changing permissions yourself and then restoring them using the following command:

Code: 
icacls "C:\Windows\WinSxS\*" /T /Q /C /RESET

It can take a very long time to complete though.
 
Okay, I went ahead and used SFCFix with a zip containing the missing WinSxS packages like I previously asked about. The server did BSOD again and in order to get it to boot I had to F8 and select the option to disable driver signing. But it does boot. Here is the latest results of SFC and DISM.

Code: 
Windows PowerShell
Copyright (C) 2016 Microsoft Corporation. All rights reserved.

PS C:\Windows\system32> sfc /scannow

Beginning system scan.  This process will take some time.

Beginning verification phase of system scan.
Verification 39% complete.

Windows Resource Protection could not perform the requested operation.
PS C:\Windows\system32> dism /online /cleanup-image /scanhealth

Deployment Image Servicing and Management tool
Version: 10.0.14393.4169

Image Version: 10.0.14393.4169

[==========================100.0%==========================] The component store is repairable.
The operation completed successfully.
PS C:\Windows\system32> dism /online /cleanup-image /restorehealth

Deployment Image Servicing and Management tool
Version: 10.0.14393.4169

Image Version: 10.0.14393.4169

[===========================80.0%==============            ]
Error: 1734

The array bounds are invalid.

The DISM log file can be found at C:\Windows\Logs\DISM\dism.log
PS C:\Windows\system32>

Some excepts of things I think could be relevant.

Code: 
2023-08-21 15:14:29, Info                  CSI    00000087 Changelist shadowed missing its manifest - Microsoft-Windows-ServicingStack-Boot, version 10.0.14393.4650, arch amd64, nonSxS, pkt {l:8 b:31bf3856ad364e35}
2023-08-21 15:14:29, Info                  CSI    00000088 Changelist shadowed missing its manifest - Microsoft-Windows-ServicingStack-Base, version 10.0.14393.4650, arch x86, nonSxS, pkt {l:8 b:31bf3856ad364e35}
2023-08-21 15:14:29, Info                  CSI    00000089 Changelist shadowed missing its manifest - Microsoft-Windows-ServicingStack-Admin, version 10.0.14393.4650, arch amd64, nonSxS, pkt {l:8 b:31bf3856ad364e35}
2023-08-21 15:14:29, Info                  CSI    0000008a Changelist shadowed missing its manifest - Microsoft-Windows-ServicingStack-Boot, version 10.0.14393.4650, arch x86, nonSxS, pkt {l:8 b:31bf3856ad364e35}
2023-08-21 15:14:29, Info                  CSI    0000008b Changelist shadowed missing its manifest - Microsoft-Windows-ServicingStack-Ds, version 10.0.14393.4650, arch amd64, nonSxS, pkt {l:8 b:31bf3856ad364e35}
2023-08-21 15:14:29, Info                  CSI    0000008c Changelist shadowed missing its manifest - Microsoft-Windows-ServicingStack-OneCore, version 10.0.14393.4650, arch amd64, nonSxS, pkt {l:8 b:31bf3856ad364e35}
2023-08-21 15:14:29, Info                  CSI    0000008d Changelist shadowed missing its manifest - Microsoft-Windows-ServicingStack-Inetsrv, version 10.0.14393.4650, arch amd64, nonSxS, pkt {l:8 b:31bf3856ad364e35}
2023-08-21 15:14:29, Info                  CSI    0000008e Changelist shadowed missing its manifest - Microsoft-Windows-ServicingStack-Net, version 10.0.14393.4650, arch amd64, nonSxS, pkt {l:8 b:31bf3856ad364e35}
2023-08-21 15:14:29, Info                  CSI    0000008f Changelist shadowed missing its manifest - Microsoft-Windows-ServicingStack-Admin, version 10.0.14393.4650, arch x86, nonSxS, pkt {l:8 b:31bf3856ad364e35}
2023-08-21 15:14:29, Info                  CSI    00000090 Changelist shadowed missing its manifest - Microsoft-Windows-ServicingStack-Ds, version 10.0.14393.4650, arch x86, nonSxS, pkt {l:8 b:31bf3856ad364e35}
2023-08-21 15:14:29, Info                  CSI    00000091 Changelist shadowed missing its manifest - Microsoft-Windows-ServicingStack-Net, version 10.0.14393.4650, arch x86, nonSxS, pkt {l:8 b:31bf3856ad364e35}
2023-08-21 15:14:29, Info                  CSI    00000092 Changelist shadowed missing its manifest - Microsoft-Windows-ServicingStack-Base, version 10.0.14393.4650, arch amd64, nonSxS, pkt {l:8 b:31bf3856ad364e35}
2023-08-21 15:14:29, Error                 CSI    00000093@2023/8/21:22:14:29.280 (F) onecore\base\wcp\componentstore\csd_winners.cpp(1273): Error STATUS_SXS_TRANSACTION_CLOSURE_INCOMPLETE originated in function OriginateManifestsMissingError expression: (null)
[gle=0x80004005]
2023-08-21 15:14:29, Info                  CBS    Added C:\Windows\Logs\CBS\CBS.log to WER report.
2023-08-21 15:14:29, Info                  CBS    Added C:\Windows\Logs\CBS\CbsPersist_20230820174812.log to WER report.
2023-08-21 15:14:29, Info                  CBS    Added C:\Windows\Logs\CBS\CbsPersist_20230815002457.cab to WER report.
2023-08-21 15:14:29, Info                  CBS    Added C:\Windows\Logs\CBS\CbsPersist_20230813184619.cab to WER report.
2023-08-21 15:14:29, Info                  CBS    Added C:\Windows\Logs\CBS\CbsPersist_20230706211512.cab to WER report.
2023-08-21 15:14:29, Info                  CBS    Added C:\Windows\Logs\CBS\CbsPersist_20230705132443.cab to WER report.
2023-08-21 15:14:29, Info                  CBS    Not able to add pending.xml to Windows Error Report. [HRESULT = 0x80070002 - ERROR_FILE_NOT_FOUND]
2023-08-21 15:14:29, Info                  CBS    Not able to add pending.xml.bad to Windows Error Report. [HRESULT = 0x80070002 - ERROR_FILE_NOT_FOUND]
2023-08-21 15:14:29, Info                  CBS    Not able to add SCM.EVM to Windows Error Report. [HRESULT = 0x80070002 - ERROR_FILE_NOT_FOUND]
2023-08-21 15:14:31, Error                 CSI    00000094 (F) STATUS_SXS_TRANSACTION_CLOSURE_INCOMPLETE #117420# from CCSDirectTransaction::PerformChangeAnalysis(...)[gle=0xd0150024]
2023-08-21 15:14:31, Error                 CSI    00000095 (F) STATUS_SXS_TRANSACTION_CLOSURE_INCOMPLETE #117419# from CCSDirectTransaction::PrepareForCommit(...)[gle=0xd0150024]
2023-08-21 15:14:31, Error                 CSI    00000096 (F) STATUS_SXS_TRANSACTION_CLOSURE_INCOMPLETE #117418# from CCSDirectTransaction::ExamineTransaction(...)[gle=0xd0150024]
2023-08-21 15:14:31, Error                 CSI    00000097 (F) STATUS_SXS_TRANSACTION_CLOSURE_INCOMPLETE #117417# from CCSDirectTransaction_IRtlTransaction::ExamineTransaction(...)[gle=0xd0150024]
2023-08-21 15:14:31, Error                 CSI    00000098 (F) HRESULT_FROM_WIN32(ERROR_SXS_TRANSACTION_CLOSURE_INCOMPLETE) #110665# from Windows::COM::CPendingTransaction::IStorePendingTransaction_Analyze(...)[gle=0x8007371b]
2023-08-21 15:14:31, Error                 CSI    00000099 (F) HRESULT_FROM_WIN32(ERROR_SXS_TRANSACTION_CLOSURE_INCOMPLETE) #100116# from Windows::ServicingAPI::CCSITransaction::ICSITransaction_Commit(Flags = 102, pSink = NULL, disp = 0, coldpatching = FALSE)[gle=0x8007371b]
2023-08-21 15:14:31, Error                 CSI    0000009a (F) HRESULT_FROM_WIN32(ERROR_SXS_TRANSACTION_CLOSURE_INCOMPLETE) #100115# 3737679 us from Windows::ServicingAPI::CCSITransaction_ICSITransaction::Commit(flags = 0x00000066, pSink = NULL, disp = 0)
[gle=0x8007371b]

Code: 
2023-08-21 17:00:26, Error                 CBS    Package "Package_for_KB5012170~31bf3856ad364e35~amd64~~14393.5285.1.4" requires Servicing Stack v10.0.14393.5285 but current Servicing Stack is v10.0.14393.4650. [HRESULT = 0x800f0823 - CBS_E_NEW_SERVICING_STACK_REQUIRED]
2023-08-21 17:00:26, Info                  CBS    Failed to initialize internal package [HRESULT = 0x800f0823 - CBS_E_NEW_SERVICING_STACK_REQUIRED]
2023-08-21 17:00:26, Error                 CBS    Failed to create internal package [HRESULT = 0x800f0823 - CBS_E_NEW_SERVICING_STACK_REQUIRED]
2023-08-21 17:00:26, Info                  CBS    Failed to CreatePackage using worker session [HRESULT = 0x800f0823]
2023-08-21 17:00:26, Info                  CBS    Failed to create internal CBS Package [HRESULT = 0x800f0823]
2023-08-21 17:00:27, Info                  CBS    WU creates the package, AppID:WSEHealthAgent, UpdateID:{E8CDEBCC-E2C3-417F-A2FE-E033BEA24132}, revision: 200
2023-08-21 17:00:27, Info                  CBS    Session: 31052939_2823047400 initialized by client WindowsUpdateAgent, external staging directory: (null), external registry directory: (null
2023-08-21 17:00:27, Error                 CBS    Package "Package_for_KB5012170~31bf3856ad364e35~amd64~~14393.5285.1.4" requires Servicing Stack v10.0.14393.5285 but current Servicing Stack is v10.0.14393.4650. [HRESULT = 0x800f0823 - CBS_E_NEW_SERVICING_STACK_REQUIRED]
2023-08-21 17:00:27, Info                  CBS    Failed to initialize internal package [HRESULT = 0x800f0823 - CBS_E_NEW_SERVICING_STACK_REQUIRED]
2023-08-21 17:00:27, Error                 CBS    Failed to create internal package [HRESULT = 0x800f0823 - CBS_E_NEW_SERVICING_STACK_REQUIRED]
2023-08-21 17:00:27, Info                  CBS    Failed to CreatePackage using worker session [HRESULT = 0x800f0823]
2023-08-21 17:00:27, Info                  CBS    Failed to create internal CBS Package [HRESULT = 0x800f0823]
2023-08-21 17:00:27, Info                  CBS    WU creates the package, AppID:WSEHealthAgent, UpdateID:{E8CDEBCC-E2C3-417F-A2FE-E033BEA24132}, revision: 200
2023-08-21 17:00:27, Info                  CBS    Session: 31052939_2823097435 initialized by client WindowsUpdateAgent, external staging directory: (null), external registry directory: (null
2023-08-21 17:00:27, Error                 CBS    Package "Package_for_KB5012170~31bf3856ad364e35~amd64~~14393.5285.1.4" requires Servicing Stack v10.0.14393.5285 but current Servicing Stack is v10.0.14393.4650. [HRESULT = 0x800f0823 - CBS_E_NEW_SERVICING_STACK_REQUIRED]
2023-08-21 17:00:27, Info                  CBS    Failed to initialize internal package [HRESULT = 0x800f0823 - CBS_E_NEW_SERVICING_STACK_REQUIRED]
2023-08-21 17:00:27, Error                 CBS    Failed to create internal package [HRESULT = 0x800f0823 - CBS_E_NEW_SERVICING_STACK_REQUIRED]

There are still many missing catalog registry entries I'm working through. However, I'm thinking that maybe the reason WU will not install newer servicing stacks is the log entries that the current servicing stack (4650) is missing it's manifests. I tries reinstalling that SSU and it reports that it's already installed. SSU's cannot be uninstalled and there is no 'repair' option presented.

The BSOD also seems to be related to the STATUS_SXS_TRANSACTION_CLOSURE_INCOMPLETE errors. However, the fix info I found here on sysnative was no help. The (default) registry entries were all set to the highest listed version in each key.

I also find it strange that there is a package for KB5012170 that exist on the server that requires a newer SSU suggesting that a newer SSU was actually installed at some point. Or is there another explanation for this apparent inconsistency?

I will continue to work on the catalog registry entry fixes. I don't know how to track or resolve the 'Changelist shadowed missing its manifest" issues.
 
Just to clarify one of my sentences from above that is a little unclear. The KB5012170 is reported in the log that it exists but can't be repaired because a newer SSU is required. Since the KB was installed, it suggests that a newer SSU was actually installed at some point and that these errors have somehow effectively uninstalled later SSUs and the system reverted to the earlier SSU. Is that how you interpret the apparent implication that a KB is installed that could not have been installed in teh first place without a newer SSU? Or could there be another reason how this log entry could even be possible?
 
There is multiple ways to fix that issue and I wouldn't necessarily follow fixes for other machines since they're specific to that one machine. It's why you'll often see different solutions for seemingly the same problem here.

Could you please provide your COMPONENTS hive and SideBySide subkey as a hive file using the following instructions?

Retrieve Components Hive
1. Navigate to C:\Windows\System32\Config and locate the COMPONENTS file.
2. Please copy this file to your desktop.
Note: If you receive an error that this file is in-use, simply reboot your computer and try again.
3. Right-click on this file on your desktop and select Send To...Compressed (zipped) folder. This will create a file named COMPONENTS.ZIP on your desktop.
4. The file will likely be too large to upload here so please upload to a file sharing service. Examples of services to upload to are Dropbox or OneDrive or WeTransfer and then just provide the link in your reply.

Export SideBySide
  1. Click on the Start button and in the search box, type regedit
  2. When you see regedit on the list, right-click on it and select Run as administrator
  3. When regedit opens, using the left pane, navigate to the following registry key and select it by clicking on it once.

    HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\CurrentVersion\SideBySide
  4. Once selected, click File > Export....
  5. Change the Save as type: to Registry Hive Files (.)
  6. Name this file SxS (with no file extension) and save it to your Desktop.


Once done please upload this file via Dropbox or OneDrive or SendSpace.

I wouldn't particularly worry about the SSU update error at the moment, it can sometimes be a false positive and safe to ignore.
 
I have an observation that is troubling relating to creating repair VMs. In my case I created a repair VM of Windows Server Essentials 2016 and did the initial configuration to closely match the damaged VM. I did not activate the VM with a key since it's a throw away VM. Over the weeks, I've added KBs used the results to resolve missing components, reverted to the original configuration, ran other KBs, etc.

Most recently I automated the installation of all the LCUs for 2016. Including all the ones that MS no longer publishes in the catalog, but where the windowsupdate.com site still has the files if you know the right link. Then reboot to complete the pending actions, then reboot again just to make sure, then saved the COMPONENTS hive to another computer so I can easily reuse it later, then ran disk cleanup advanced including system and windows update files and rebooted. Then I repeated that process with the 2017 LCUs, etc. This last round after installation of 2019 LCUs and saving its COMPONENTS hive, disk cleanup was taking days and the progress bar wasn't moving at all. In previous iterations it was taking several hours. For 2019 it would proress through the first analysis of disk cleanup, and present the list, but when selecting the button to include system files, it's at that point that the analysis would hang. It never gave me the list to be able to place the check box to clean up windows updates.

I then tried dism startcomponentcleanup which hung at around 9%. After checking the system logs I discovered several messages about WSE 2016 not being licensed and that several features were disabled, without giving a list of such features. Really? Did it mean that maybe they disabled advanced disk and component cleanup? As a check, I used slmgr to rearm, and reboot. Presto! Disk cleanup of system files progresses normally! If it's not just a coincidence of some other thing happening in the background and MS does in fact disable disk cleanup of WinSxS folder after the registration timeout, then it's a truly petty way to get people to buy a license. Essentially making the OS crippleware. It's like you get 30 days and after that it might still run, but it won't run for long because we are going to eat up all your disk space if you manually install KBs. And it doesn't even have the decency to actually tell you that.

I truly hope this is not the case and that there actually was some other reason why a single years LCUs were somehow hung during disk cleanup of system files and that it wasn't a licensing issue. But the coincidence of running fine right after running slmgr is very suspicious.
 
It looks like the Component subkeys were missing, the attached script should add them back for you and should hopefully at least resolve that error in the CBS log.

That is very odd, I've run several source VMs past their initial evaluation period and I don't think I've encountered any messages about features being disabled? I usually never run disk cleanup on my source VMs because I used to use them to repair several different "iterations" of WS2016. The VMs can get really big though if you install every LCU available, I think my Windows 7 VM was about 90GB almost? And that was with pretty much all the available updates installed.
 

Attachments

The failing server is still unsuccessful in installing newer servicing stacks and fails on SFC and DISM restore health. Though, strangely it succeeds on DISM scanhealth. And it still will not boot without manually selecting the troubleshooting boot option of disabling driver signing.

In the most recent ComponentsScanner.txt there were only CanonicalData/Catalogs entries and mismatched f! entries.

I'll post the latest DISM.log, CBS.log and ComponentsScanner.txt results in a few hours.

For my last round of building a repair server (which was completed this morning) to use with SfcFixScriptBuilder I had specifically disabled the tasks for Disk Cleanup and Servicing because I wanted to get clean copies of the COMPONENTS hive with each year's LCUs to keep for reference and to be able to reapply fixes should they be needed again. I then manually used disk cleanup and component cleanup before going on to the next year's LCUs. I just finished getting the last 2022 LCUs installed, and I was concerned that I wasn't seeing any new CanonicalData\Catalogs being resolved. The disk space also seemed lower that it should have been. It turns out that somewhere along the line, one of the LCUs must have turned back on disk cleanup and servicing tasks. I have no idea which LCU caused it or even which year. So I guess I have to go back and do it all over again after finding a reliable way to prevent those tasks from being re-enabled by an LCU. Or find out which LCU is doing it and save a COMPONENTS hive just before installing that one.
 
Here are the latest logs and some other files extracted from them. The Missing_Catalog_Keys_20230905_1610.txt file is simply an extract from the ComponentsScanner.txt file to try to feed into SfcFixScriptBuilder with the --catalogs switch. The other logs files are two logs from my powershell scripts that may be of interest.
 

Attachments

How do I find out which KB or MSU is responsible for ComponentsScanner.txt entries such as below?
C:\Windows\winsxs\Catalogs\00006083976b624ebaa7f1c37436d11aa4a22d1a720dbf426e9ecb43b6475d19.cat exists but no corresponding CanonicalData\Catalogs key found
 
mjmeans said:
How do I find out which KB or MSU is responsible for ComponentsScanner.txt entries such as below?
Click to expand...
You could try using fsutil from command prompt like so:

Code: 
fsutil hardlink list %systemroot%\WinSxS\Catalogs\00006083976b624ebaa7f1c37436d11aa4a22d1a720dbf426e9ecb43b6475d19.cat

If the named package exists, then it will be displayed as a hardlink.

mjmeans said:
So I guess I have to go back and do it all over again after finding a reliable way to prevent those tasks from being re-enabled by an LCU.
Click to expand...
There is a scheduled task which is set to run every 30 days, effectively it just does a disk cleanup and then deletes some orphaned entries from your registry. It should be called something like StartComponentCleanup. You may want to disable that but I've personally never had any issues with leaving it on.

mjmeans said:
Though, strangely it succeeds on DISM scanhealth. And it still will not boot without manually selecting the troubleshooting boot option of disabling driver signing.
Click to expand...
/RestoreHealth is slightly more involved than /ScanHealth; it actually attempts to carry out repairs so it could possibly explain why one works and the other doesn't.

Code: 
2023-08-12 12:36:47, Info                  DISM   DISM Package Manager: PID=7520 TID=14472  Error in operation: (null) (CBS HRESULT=0x800700c1) - CCbsConUIHandler::Error
2023-08-12 12:36:47, Error                 DISM   DISM Package Manager: PID=7520 TID=5020 Failed finalizing changes. - CDISMPackageManager::Internal_Finalize(hr:0x800700c1)
2023-08-12 12:36:47, Error                 DISM   DISM Package Manager: PID=7520 TID=5020 Failed processing package changes - CDISMPackageManager::StartComponentCleanupEx(hr:0x800700c1)
2023-08-12 12:36:47, Error                 DISM   DISM Package Manager: PID=7520 TID=5020 Failed to start component cleanup. - CPackageManagerCLIHandler::ProcessCmdLine_CleanupImage(hr:0x800700c1)
2023-08-12 12:36:47, Error                 DISM   DISM Package Manager: PID=7520 TID=5020 Failed while processing command cleanup-image. - CPackageManagerCLIHandler::ExecuteCmdLine(hr:0x800700c1)
2023-08-12 12:36:47, Info                  DISM   DISM Package Manager: PID=7520 TID=5020 Further logs for online package and feature related operations can be found at %WINDIR%\logs\CBS\cbs.log - CPackageManagerCLIHandler::ExecuteCmdLine
2023-08-12 12:36:47, Error                 DISM   DISM.EXE: DISM Package Manager processed the command line but failed. HRESULT=800700C1

Code: 
2023-08-12 12:46:32, Info                  CBS    Failed to load sfp DLL from path: C:\Windows\winsxs\amd64_microsoft-windows-servicingstack_31bf3856ad364e35_10.0.14393.4650_none_7eeb34dc2202872c\\wrpint.dll [HRESULT = 0x800700c1 - ERROR_BAD_EXE_FORMAT]
2023-08-12 12:46:32, Info                  CBS    Failed to initialize WRP integrity check and repair. [HRESULT = 0x800700c1 - ERROR_BAD_EXE_FORMAT]

The attached SFCFix should hopefully get DISM running again for you.
 

Attachments

x BlueRobot said:
There is a scheduled task which is set to run every 30 days, effectively it just does a disk cleanup and then deletes some orphaned entries from your registry. It should be called something like StartComponentCleanup. You may want to disable that but I've personally never had any issues with leaving it on.
Click to expand...

I disabled both the scheduled tasks before I started applying the LCUs. Somehow, they got ren-enabled. I suspect and LCU did it, but I have way to know which.

After applying the fix, DISM restorehealth fails at 80%.
Code: 
PS C:\Temp\Tools> dism /online /cleanup-image /restorehealth

Deployment Image Servicing and Management tool
Version: 10.0.14393.4169

Image Version: 10.0.14393.4169

[===========================80.0%==============            ]
Error: 14

Not enough storage is available to complete this operation.

The DISM log file can be found at C:\Windows\Logs\DISM\dism.log

It's strange because there is 30 GB left on a 127 GB C: partition. The partition is clearly not full.
 
Okay, so I checked using fsutil initially and then decided to write a powershell script to do it and also to look up all the KB that could be identified. Most of the catalog entries had no have any hard links, but some did.

Code: 
Id      Title                                                                                              
--      -----                                                                                              
3186568 Microsoft .NET Framework 4.7 for Windows 10 Version 1607 and Windows Server 2016 for x64 (KB3186568)
4013418 Update for Windows Server 2016 for x64-based Systems (KB4013418)                                  
4023834 2017-06 Update for Windows Server 2016 for x64-based Systems (KB4023834)                          
4035631 2017-08 Update for Windows Server 2016 for x64-based Systems (KB4035631)                          
4041691 2017-10 Delta Update for Windows Server 2016 for x64-based Systems (KB4041691)                    
4054590 Microsoft .NET Framework 4.7.2 for Windows 10 Version 1607 and Server 2016 for x64 (KB4054590)    
4465659 2018-11 Update for Windows Server 2016 for x64-based Systems (KB4465659)                          
4485447 2019-02 Servicing Stack Update for Windows Server 2016 for x64-based Systems (KB4485447)          
4498947 2019-05 Servicing Stack Update for Windows Server 2016 for x64-based Systems (KB4498947)          
4503537 2019-06 Servicing Stack Update for Windows Server 2016 for x64-based Systems (KB4503537)          
4509091 2019-07 Servicing Stack Update for Windows Server 2016 for x64-based Systems (KB4509091)          
4512574 2019-09 Servicing Stack Update for Windows Server 2016 for x64-based Systems (KB4512574)          
4520724 2019-11 Servicing Stack Update for Windows Server 2016 for x64-based Systems (KB4520724)          
4521858 2019-10 Servicing Stack Update for Windows Server 2016 for x64-based Systems (KB4521858)          
4530689 2019-12 Cumulative Update for Windows Server 2016 for x64-based Systems (KB4530689)                
4534271 2020-01 Cumulative Update for Windows Server 2016 for x64-based Systems (KB4534271)                
4540723 2020-03 Servicing Stack Update for Windows Server 2016 for x64-based Systems (KB4540723)          
4550994 2020-04 Servicing Stack Update for Windows Server 2016 for x64-based Systems (KB4550994)          
4562561 2020-06 Servicing Stack Update for Windows Server 2016 for x64-based Systems (KB4562561)          
4565912 2020-07 Servicing Stack Update for Windows Server 2016 for x64-based Systems (KB4565912)          
4576750 2020-09 Servicing Stack Update for Windows Server 2016 for x64-based Systems (KB4576750)          
4589210 2021-01 Update for Windows Server 2016 for x64-based Systems (KB4589210)                          
5001402 2021-04 Servicing Stack Update for Windows Server 2016 for x64-based Systems (KB5001402)          
5005698 2021-09 Servicing Stack Update for Windows Server 2016 for x64-based Systems (KB5005698)

I'm surprised on the large number of SSUs here.

FYI here is the PowerShell script I wrote that generates the above output. It requires the module KbUpdate from NuGet is installed first.

Code: 
function Get-HardLinks([string]$path)
{
    (dir $path -force | Where-Object -Property LinkType -eq HardLink | Select-Object -Property Target).Target
}

$file = 'Missing_Catalog_Keys.txt'
If (!(Test-Path $file -PathType leaf))
{
    throw("FILE_NOT_FOUND: $file'")
}

$kbs = [System.Collections.ArrayList]::new()

$lines = Get-Content $file
$i=0
foreach ($line in $lines)
{
    $file = "c:\windows\WinSxS\Catalogs\$($line).cat"
    $i = $i + 1
    $p = ($i * 100 / $lines.Count)
    $a = "Checking: $line"
    Write-Progress -Activity $a -PercentComplete $p
    If (Test-Path $file -PathType leaf)
    {
        $links = Get-hardLinks($file)
        if ($links)
        {
            Write-Progress -Activity $a -PercentComplete $p -CurrentOperation "Checking: $file"
            foreach ($link in $links)
            {
                Write-Progress -Activity $a -PercentComplete $p -CurrentOperation "Checking: $link"
                if ($link -match 'for_(KB.+?)~')
                {
                    $kb = $matches[1]
                    if (-not $kbs.Contains($kb))
                    {
                        [void]$kbs.Add($kb)
                    }
                }
            }
        }
    }  
}
foreach ($kb in ($kbs | Sort-Object))
{
    Get-kbupdate $kb -Architecture x64 -OperatingSystem 'Windows Server 2016' | Select-Object -Property id, Title
}
 
Summary of the current state of this fix 9/8/2023 ...

  1. The server will not boot without disabling driver signature verification and will not install the latest SSU.
  2. Many missing packages and COMPONENTS registry entries have been repaired, as well as a few missing WinSxS packages that could be identified.
  3. ComponentsScanner now only has a few mismatched f! entries showing more f! marks than expected, and thousands of CanonicalData/Catalogs missing registry entries.
  4. In an attempt to install all SSUs from RTM to current on the server, some older SSUs were never installed in the first place, or might have been removed during an older cleanup and the installation succeeded for those. Others were reported as already installed. KB5005698 was the last SSU to succeed with HRESULTS was 0x00240006 - already installed. The very next SSU in sequence after that is KB5011570 which results in success HRESULT 0x0000371b which is an unknown or undocumented result. Looking into the logs, 0x0000371b reported by WUSA can be traced to traced to 0x8007037b (transaction closure incomplete). It seems that WUSA reports the error success instead of a failure.
  5. This suggests to me that the KB5005698 SSU is the update that is damaged somehow. WUSA reports it as already installed because there is no repair install option for SSUs. And since SSUs cannot be uninstalled this leaves the system in a state where no new SSUs can be installed.
  6. LCUs have been installed that rely on SSUs that are newer than KB5005698! That suggests that newer SSUs than KB5005698 had been installed at some point and are damaged. And I wonder if that is possibly the cause of all the missing catalogs.
  7. Malwarebytes has been run several times to verify that there is no known malware that is blocking repairs.

Is it wrong to conclude that since WinSxS has LCUs in it that rely on SSUs that are apparently not installed, and that there are thousands of *.cat files with missing registry entries, that perhaps the COMPONENTS hive was truncated at some point? And perhaps the task that runs component cleanup actually did further damage by removing registry entries it could not find any reference to?

Anyway, I think my next two fix attempts will be to create a repair server and bring it up to exactly KB5005698 and see what happens with using that as a repair source in DISM.

If that doesn't work, I suppose my next step could be to use Get-KbInstalledUpdates (which does it's darndest to get a list of every KB installed on a machine, even ones that don't show) and then install the exact same list of KB on a repair server as a repair source.

I'm really just guessing. Is there a better way to proceed?
 
mjmeans said:
I disabled both the scheduled tasks before I started applying the LCUs. Somehow, they got ren-enabled. I suspect and LCU did it, but I have way to know which.

After applying the fix, DISM restorehealth fails at 80%.
Code: 
PS C:\Temp\Tools> dism /online /cleanup-image /restorehealth

Deployment Image Servicing and Management tool
Version: 10.0.14393.4169

Image Version: 10.0.14393.4169

[===========================80.0%==============            ]
Error: 14

Not enough storage is available to complete this operation.

The DISM log file can be found at C:\Windows\Logs\DISM\dism.log

It's strange because there is 30 GB left on a 127 GB C: partition. The partition is clearly not full.
Click to expand...

The error in the log is Failed to get bundled updates [HRESULT = 0x8007000e - E_OUTOFMEMORY]. But it hasn't run out of memory either. Typically runs between 6-8 GB of RAM in a 16 GB RAM hyper-v host. I re-ran it and the same thing happened, but it didn't go over 75% RAM usage.
 
I would create some space on your disk, 30GB really isn't much considering most of those files may need to be uncompressed in order to be loaded and repaired.

mjmeans said:
Is it wrong to conclude that since WinSxS has LCUs in it that rely on SSUs that are apparently not installed, and that there are thousands of *.cat files with missing registry entries, that perhaps the COMPONENTS hive was truncated at some point? And perhaps the task that runs component cleanup actually did further damage by removing registry entries it could not find any reference to?
Click to expand...
I wouldn't honestly worry about most of those missing .cat files at the moment but instead would focus on resolving the immediate errors in your CBS log(s). That error about being out of memory is probably a red herring, what is the size of your COMPONENTS, SOFTWARE, SYSTEM and DRIVERS hive? They should be available under %systemroot%\System32\config.
 
You must log in or register to reply here.
Back
Top