[SOLVED] Integrity violations that cannot be repaired - Windows Server 2019 Standard [AECSS19]

Do you think I should kill TrustedInstaller first? I ran the /Add-Package command, but it's just hanging with no output. Been like this for about 15 mins.
 
I certainly wouldn't do that, but DISM does not show the progress line? Can you please provide a screenshot for me to see at which point it stops doing anything.

Please check if there is a reboot pending as well from the Windows Update window, this because the CBS log reports that a shutdown is in progress.

Note: Before installing KB5016623 (LCU), the SSU KB5005112 must be installed first.
 
Sure, this probably isn't very helpful, but here's the screenshots. As you can see, it's just a black screen, so I can't check for pending updates other than via CMD.

EDIT: I confirmed that there is NOT a pending restart at the moment: From the command line, how can I tell if a reboot is pending? - Microsoft Q&A

Code: 
ComputerName                     : AECSS19
ComponentBasedServicing          : False
PendingComputerRenameDomainJoin  : False
PendingFileRenameOperations      : False
PendingFileRenameOperationsValue :
SystemCenterConfigManager        :
WindowsUpdateAutoUpdate          : False
IsRebootPending                  : False

EDIT #2: According to our patching monitor KB5005112 was installed on 08/14/202. See attached screenshot.
 

Attachments

  • Untitled.png
    Untitled.png
    124.2 KB · Views: 6
  • Untitled2.png
    Untitled2.png
    78.4 KB · Views: 6
  • KB.png
    KB.png
    41.3 KB · Views: 4
Last edited:
Rich (BB code): 
2022-08-17 06:33:22, Info                  CBS    Failed to enumerate all packages for index: Microsoft-Windows-EnterpriseSNEvalEdition~31bf3856ad364e35~amd64~~0.0.0.0 [HRESULT = 0x80070013 - ERROR_WRITE_PROTECT]
2022-08-17 06:33:22, Info                  CBS    Failed to remove package dependencies for package: Package_2104_for_KB5004947~31bf3856ad364e35~amd64~~10.0.1.0 [HRESULT = 0x80070013 - ERROR_WRITE_PROTECT]
2022-08-17 06:33:22, Info                  CBS    Failed to remove from store package: Package_2104_for_KB5004947~31bf3856ad364e35~amd64~~10.0.1.0 [HRESULT = 0x80070013 - ERROR_WRITE_PROTECT]
2022-08-17 06:33:22, Info                  CBS    Startup: Package: Package_2104_for_KB5004947~31bf3856ad364e35~amd64~~10.0.1.0 failed to complete startup processing, new state: Absent, original: Resolved, targeted: Absent.  hr = 0x0 [HRESULT = 0x80070013 - ERROR_WRITE_PROTECT]
2022-08-17 06:33:22, Info                  CBS    Failed to persist changes for package: Package_2104_for_KB5004947~31bf3856ad364e35~amd64~~10.0.1.0 [HRESULT = 0x80070013 - ERROR_WRITE_PROTECT]
2022-08-17 06:33:22, Info                  CBS    Failed to open key . [HRESULT = 0x80070013 - ERROR_WRITE_PROTECT]
2022-08-17 06:33:22, Info                  CBS    Failed to completely delete store object. [HRESULT = 0x80070013 - ERROR_WRITE_PROTECT]
2022-08-17 06:33:22, Info                  CBS    Failed to delete pending package: Package_2104_for_KB5004947~31bf3856ad364e35~amd64~~10.0.1.0 [HRESULT = 0x80070013 - ERROR_WRITE_PROTECT]
2022-08-17 06:33:22, Info                  CBS    Failed to process pending package object. [HRESULT = 0x80070013 - ERROR_WRITE_PROTECT]
2022-08-17 06:33:22, Error                 CBS    Startup: Failed to process all pending packages, ignoring failure and continuing. [HRESULT = 0x80070013 - ERROR_WRITE_PROTECT]
2022-08-17 06:33:22, Info                  CBS    Failed to open key: SOFTWARE\Microsoft\Windows\CurrentVersion\Component Based Servicing\Interface [HRESULT = 0x80070013 - ERROR_WRITE_PROTECT]

Hmm, let's take a look at this error first. Please run the following commands to export the CBS hive from the registry. Attach CBS-Hive.zip to your next post.

Code: 
REG Save "HKLM\SOFTWARE\Microsoft\Windows\CurrentVersion\Component Based Servicing" "%systemdrive%\WUTemp\CBS.hiv"
PowerShell Compress-Archive -Path "%systemdrive%\WuTemp\*.hiv" -DestinationPath "%systemdrive%\WUTemp\CBS-Hive.zip"
 
Here you go!

Also, based on my math, I'd bet the server will reboot itself again around 12:30 PM today. Just after looking at the timestamps in CBS.log. I've already prepped the users for this.
 

Attachments

Question... It seems like the Windows Update process is actually making progress, UNTIL this part:

Code: 
2022-08-17 06:32:57, Info                  CBS    Startup: Timed out waiting for startup processing to complete
2022-08-17 06:32:57, Info                  CBS    Current global progress. Current: 1, Limit: 1, ExecuteState: CbsExecuteStateComplete
2022-08-17 06:32:57, Info                  CBS    Previous global progress. Current: 1, Limit: 1, ExecuteState: CbsExecuteStateComplete
2022-08-17 06:32:57, Info                  CBS    Setting HangDetect value to 1
2022-08-17 06:32:57, Info                  CBS    Startup: will attempt a restart to recover.
2022-08-17 06:32:57, Info                  CBS    A restart has been initiated
2022-08-17 06:32:57, Info                  CBS    Setting reboot in progress flag
2022-08-17 06:32:57, Info                  CBS    Startup: A system shutdown was initiated while waiting for startup to complete

Do you think we should figure out how to adjust the timeout value to something higher? That way it won't time out and enter a boot loop, but rather [perhaps] complete the updates?
 
Thanks, I think that too, due to the presence of the "ServicingInProgress" value. This is an indicator that theTrustedInstaller Service is still doing some things. When it has the value data 0x0 it is in 'idle' mode, but from the provided CBS hive I'll see 0x2, so it is still active in expectation of an reboot I think? But you stopped the TrustedInstaller service earlier, so I would try to perform a new reboot first and the take a new look at the latest CBS log.

I've used this command to check the state of the "ServicingInProgress" value.
Rich (BB code): 
REG Query "HKLM\SOFTWARE\Microsoft\Windows\CurrentVersion\Component Based Servicing\Interface"

Rich (BB code): 
2022-08-17 06:33:22, Error                 CBS    Startup: Failed to notify session manager on startup finish. [HRESULT = 0x80070013 - ERROR_WRITE_PROTECT]
2022-08-17 06:33:22, Info                  CBS    Startup: Retrying failed packages.
2022-08-17 06:33:22, Info                  CBS    Exec: Resuming session: 30978468_1342637377, last committed phase: 1, Current phase: 1, Total phase: 1
2022-08-17 06:33:22, Info                  CBS    Failed to open online package store: SessionsPending. [HRESULT = 0x80070013 - ERROR_WRITE_PROTECT]
2022-08-17 06:33:22, Info                  CBS    Failed to get key for store object type: 5 [HRESULT = 0x80070013 - ERROR_WRITE_PROTECT]
2022-08-17 06:33:22, Info                  CBS    Failed to get session exclusivity [HRESULT = 0x80070013 - ERROR_WRITE_PROTECT]
2022-08-17 06:33:22, Info                  CBS    Failed to get session exclusivity [HRESULT = 0x80070013 - ERROR_WRITE_PROTECT]
2022-08-17 06:33:22, Info                  CBS    Exec: Session: 30978468_1342637377 resumed. Reboot required: no, Current phase: 1, Total phase: 1 [HRESULT = 0x80070013 - ERROR_WRITE_PROTECT]
2022-08-17 06:33:22, Info                  CBS    Startup: Failed to resume session: 30978468_1342637377, status = 0x80070013 [HRESULT = 0x80070013 - ERROR_WRITE_PROTECT]
2022-08-17 06:33:22, Info                  CBS    Startup: Failed to resume sessions: hr: 0x80070013 [HRESULT = 0x80070013 - ERROR_WRITE_PROTECT]
2022-08-17 06:33:22, Info                  CBS    Failed to get key for store object type: 1 [HRESULT = 0x80070013 - ERROR_WRITE_PROTECT]
2022-08-17 06:33:22, Info                  CBS    Startup: Processing complete. [HRESULT = 0x00000000 - S_OK]
2022-08-17 06:33:22, Info                  CBS    Failed to get key for store object type: 1 [HRESULT = 0x80070013 - ERROR_WRITE_PROTECT]
2022-08-17 06:33:22, Info                  CBS    Failed to get root store object [HRESULT = 0x80070013 - ERROR_WRITE_PROTECT]
2022-08-17 06:33:22, Info                  CBS    Failed to get key for store object type: 12 [HRESULT = 0x80070013 - ERROR_WRITE_PROTECT]
2022-08-17 06:33:22, Info                  CBS    Failed to get root applicability cache store object [HRESULT = 0x80070013 - ERROR_WRITE_PROTECT]
2022-08-17 06:33:22, Info                  CBS    Enabling LKG boot option
2022-08-17 06:33:22, Info                  CBS    Failed enabling LKG [HRESULT = 0xd0000189 - NTSTATUS Error]
2022-08-17 06:33:22, Info                  CBS    Startup: Failed enabling LKG [HRESULT = 0xd0000189 - NTSTATUS Error]
2022-08-17 06:33:22, Info                  CBS    Failed to get key for store object type: 17 [HRESULT = 0x80070013 - ERROR_WRITE_PROTECT]
2022-08-17 06:33:22, Info                  CBS    Failed getting root key [HRESULT = 0x80070013 - ERROR_WRITE_PROTECT]
2022-08-17 06:33:22, Info                  CBS    Failed to open key: SOFTWARE\Microsoft\Windows\CurrentVersion\Component Based Servicing\Interface [HRESULT = 0x80070013 - ERROR_WRITE_PROTECT]
2022-08-17 06:33:22, Info                  CBS    Failed writing LastTrustedInstallerBoot value. [HRESULT = 0x80070013 - ERROR_WRITE_PROTECT]
2022-08-17 06:33:22, Info                  CBS    Failed to get key for store object type: 5 [HRESULT = 0x80070013 - ERROR_WRITE_PROTECT]
2022-08-17 06:33:22, Info                  CBS    Failed to get online session pending registery object [HRESULT = 0x80070013 - ERROR_WRITE_PROTECT]
2022-08-17 06:33:22, Info                  CBS    Startup: Failed to set total session reboot counts [HRESULT = 0x80070013 - ERROR_WRITE_PROTECT]
 
I didn't actually kill the TrustedInstaller service on this server (yet)... Just the old server named CARSON. Is there any way I can adjust that timeout value in the next 10 mins? I think it'll time out within the next 10 mins...

Whoops, jumped the gun there. Here's the output:

Code: 
PS C:\Windows\system32> REG Query "HKLM\SOFTWARE\Microsoft\Windows\CurrentVersion\Component Based Servicing\Interface"

HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\CurrentVersion\Component Based Servicing\Interface
    ServicingInProgress    REG_DWORD    0x2
    LastTrustedInstallerBoot    REG_QWORD    0x1d8af012fa4e86d
 
I don't know if you can adjust the timeout value. I don't dare say if that is wise either, at this stage I would let everything as it goes and see what happens.
 
I've been trying to locate the registry key that corresponds to that timeout value, but to no avail yet. So far I've seen that BlockTimeIncrement may pertain to something similar in nature: Windows updates consistently failing after 15 minutes

However, by default it was set to 15 minutes, so this can't be it... I guess I'll just hang tight and see what happens. Not sure what to tell end users though. "Your server might reboot at some point today", haha.
 
The link you've posted is not working here, it displays a timeout?? But in this situation, I would do a manual reboot from the command line and then check the "ServicingInProgress" value again.
 
Unfortunately it just restarted. I'll attach CBS.log shortly.

Here's the log:
 

Attachments

Last edited:
Hey! Just had a realization...

Even though this server is rebooting, I still think it's making progress. From a high level perspective of CBS.log, you'll see a huge chunk of lines that look like this:

Code: 
2022-08-17 15:23:29, Info                  CBS    Startup: Package: Package_3068_for_KB5014692~31bf3856ad364e35~amd64~~10.0.2.15 completed startup processing, new state: Absent, original: Resolved, targeted: Absent.  hr = 0x0
2022-08-17 15:23:29, Info                  CBS    Startup: Package: Package_3068_for_KB5016623~31bf3856ad364e35~amd64~~10.0.1.7 completed startup processing, new state: Installed, original: Staged, targeted: Installed.  hr = 0x0
2022-08-17 15:23:32, Info                  CBS    Startup: Package: Package_3069_for_KB5003171~31bf3856ad364e35~amd64~~10.0.1.4 completed startup processing, new state: Absent, original: Resolved, targeted: Absent.  hr = 0x0

Then, abruptly these will stop with the following line:2022-08-17 12:43:45, Info CBS Startup: Timed out waiting for startup processing to complete

However, once the progress starts up again, I think it's actually resuming progress, not restarting. The Package_xxxx_for_kbxxxxxxx are still increasing after each reboot. For example:

Started @ Package_1000_for_KB4592440, rebooted @ Package_1567_for_KB5004947 after 6 hours and 12 mins
Re-started @ Package_1568_for_KB4592440, rebooted @ Package_2103_for_KB5012647 after 6 hours and 10 mins
Re-started @ Package_2104_for_KB4592440, rebooted @ Package_273_for_KB5001342 after 6 hours and 10 mins (yes - it's not going sequentially, but rather all 1's, then 2's, then 3's, etc.)
Re-started @ Package_273_for_KB5003646, and it's currently @ Package_3069_for_KB5003171

I kind of think this might resolve itself if we're patient. I really hope so. We'll see where we're at tomorrow!
 
I couldn't find anything in CBS.log that indicates how many "packages" need to get processed by this server. But here's some logic that may indicate how long we'll need to wait:

  • We know packages are processed in lexicographical/alphabetical order. In other words, non-numerical order (e.g. 1000 comes before 10).
  • First package to be processed is 1,000. Thus, if there was a 10,000th package, we'd have seen it by now. So, at most there are 9,999 packages.
  • Package 1,000 processed on 08/16/22 @ 18:22:20. Package 1 processed on 08/17/22 @ 05:19:00. It took 10 hours 56 minutes 40 seconds to process the first ~1,000 packages.
  • Package 2,000 processed on 08/17/22 @ 05:19:05. Package 29 (there is no Package 2, oddly) processed on 08/17/22 @ 14:54:56. It took 9 hours 35 minutes 51 seconds to process the second ~1,000 packages.
  • Package 3,000 processed on 08/17/22 @ 14:54:56. We're currently on Package 3,443 (almost halfway through the 3,000's) @ 17:36:29. It took 2 hours 41 minutes 33 seconds to process ~1/2 of the third ~1,000 packages.
  • Doubling that timeframe gives us 5 hours 23 minutes 6 seconds (we'll round up to 6 hours) to finish processing the 3,000's.
  • The avg. processing time of 1,000 packages given these three data points is 8 hours 50 minutes 50 seconds.
  • Worst case scenario of having 9,999 packages lands us at a total processing time of 3 days 16 hours 28 minutes 20 seconds.
  • Considering our current progress (roughly 35% finished of worst case scenario of having 9,999 packages), we'll be finished by 08/20/22 01:42:11 PM
I sure hope these calculations are not accurate, because I have 17 more servers running Server 2019 Standard left to repair. It'll be tough to tell clients to expect rolling blackouts for nearly a week.

Tough situation.
 
Hi,

Thanks for the detailed information, my first thought is that the system is running out of disk space? Other question, is this system protected with drive encryption?
 
I confirmed there's still plenty of disk space (522 GB / 55% remaining). No drive encryption either. Anyways, I had somewhat of an interesting night. To make a long story short, the server finished processing everything; it's currently at the login screen. I confirmed it processed packages up until ~8,000, and it finished around 7 AM this morning. Much sooner than expected!! It began processing packages roughly 100% faster for some reason. It can't be a load thing based on time of day, as it had all of Tuesday night/Wednesday morning to perform the same task, and it was quite slow. Maybe the packages themselves became smaller? Anyone's guess really.

I ran DISM /Online /Cleanup-Image /AnalyzeComponentStore and caught 9 reclaimable packages. I ran DISM /Online /Cleanup-Image /StartComponentCleanup and it's been hanging for a while. I'll keep you posted on its progress.
 
soj4trade said:
I had somewhat of an interesting night.
Click to expand...
I can imagine that, this is rather a special case. I don't think it was load thing either, the only thing I can think of is (third-party) security software that caused slowness?!
 
Well, /StartComponentCleanup took THE ENTIRE DAY to complete (10 AM to 5:27 PM), but it worked. 0 reclaimable packages, and Windows Updates says it's up to date. Marvelous. I seriously appreciate your help!! I'm going to try this again on another Server 2019 OS (since I have access to a ton of donor files). But I'm sure we'll be chatting again, as I have a couple other threads open for other servers. :)

Thank you, profusely!!
 
You must log in or register to reply here.

Has Sysnative Forums helped you? Please consider donating to help us support the site!

Back
Top