No... the app runs mostly Windows EXEs along with some WMI and VBS code.
Server 2012 R2 possible memory leak?
- Thread starter bascotie
- Start date
No... the app runs mostly Windows EXEs along with some WMI and VBS code.
Since things have been working well for almost a couple weeks now, I'm gonna keep an eye on this and go from there. Those app crashes (some) were from the bios update / firmware update programs that would crash when it tried to run them.
Thanks for all your help. The biggest issues seem resolved but I'll report back if they occur again
Were you ever able to update the BIOS?
Nope, just left that as is for now.
So looks like the issue recently came back (after 1.5-2 weeks seemingly without issue). This time, I found I can use the server fine but guys who are using the SQL based program are complaining of hang ups every 20-30 minutes for 30-60 seconds before it resumes. I've contacted the company responsible but here are the new logs just in case
Consider reinstalling server 2012 R2 or a newer version.
After installing the server operating system reset the BIOS then make additional attempts to upgrade the BIOS.
Windows Server Catalog
How to Clear Your Computers CMOS to Reset BIOS Settings:
How to Clear Your Computer’s CMOS to Reset BIOS Settings
3 Ways to Reset Your BIOS - wikiHow:
How to Reset Your BIOS
3 Easy Ways to Clear CMOS (Reset BIOS)
3 Easy Ways to Clear CMOS (Reset BIOS)
After installing the server operating system reset the BIOS then make additional attempts to upgrade the BIOS.
Windows Server Catalog
How to Clear Your Computers CMOS to Reset BIOS Settings:
How to Clear Your Computer’s CMOS to Reset BIOS Settings
3 Ways to Reset Your BIOS - wikiHow:
How to Reset Your BIOS
3 Easy Ways to Clear CMOS (Reset BIOS)
3 Easy Ways to Clear CMOS (Reset BIOS)
- May 7, 2013
- 10,400
It seems like an application-specific issue then. Have you checked the server/applications logs to see what the issue is? Are there any services or tasks which are set to run at specific intervals?
- May 7, 2013
- 10,400
Do you know how difficult that may be with a production server? I don't see the benefit of a clean install.
The fact that he cannot do BIOS updates bothers me.
What else is potentially wrong with the server?
What else is potentially wrong with the server?
It seems to happen throughout the day, especially when a high number of remote desktop users are in (it's all on a local network but they use RDP to login to their own sessions and run the program from there).
I've asked them to start reporting more specifically to us the times that it happened, what they were doing, etc so I can check the logs more accurately.
Unfortunately, we're trying to avoid a reinstall at all costs as setting this back up would be a huge job.
I did check yesterday's logs and am seeing a lot of these events (araavl.exe is part of the program in question
perhaps the software guys just need to install the software from scratch and reload the database
I've asked them to start reporting more specifically to us the times that it happened, what they were doing, etc so I can check the logs more accurately.
Unfortunately, we're trying to avoid a reinstall at all costs as setting this back up would be a huge job.
I did check yesterday's logs and am seeing a lot of these events (araavl.exe is part of the program in question
perhaps the software guys just need to install the software from scratch and reload the database
I bet, I only have 5 users and 6 remotes at my work and a clean or even a re-install on the Point of Sale server would put us down for at least a day. Tons of work to get back up and running.
I noticed that the computer name on all = ENTERPRISE_RENTAL_LOCAL
Is that the name of the server?
I assume the car rental company?
If so, how many locations RDP into the server in question (is there just a single server?) and do all locations have the same computer name or are the above entries (screenshots) just from one location?
Is there a way to tell which entries came from which locations, assuming >1 location?
I used to be a COO for Hertz RAC and know that there cannot be tons of entries occurring (cars going out or being returned) at the same exact time. RAC server traffic is usually rather staggered (unless airport locations involved) and the time it takes to process a single outgoing rental or rental return should be a few seconds at most, I would assume.
Have you ran Sysinternals Process Monitor (ProcMon) which records every I/O, including Registry entries?
Process Monitor - Windows Sysinternals | Microsoft Docs
Stand alone executable; nothing to install. All output written to page file, so virtual memory increases really fast.
If you run it, "Run as Administrator"; make sure capture ON; scrolling ON; then after a few million entries, you can turn capture OFF and save the data as an ARN file.
Zip it; upload to One Drive or other legit 3rd party site as the file will be huge - even after a 30-60 minute execution.
Regards. . .
John
p.s. No idea if this info will tell us anything, but it is worth a try. Also, 30-60 minutes may not be a long enough time period.
Is that the name of the server?
I assume the car rental company?
If so, how many locations RDP into the server in question (is there just a single server?) and do all locations have the same computer name or are the above entries (screenshots) just from one location?
Is there a way to tell which entries came from which locations, assuming >1 location?
I used to be a COO for Hertz RAC and know that there cannot be tons of entries occurring (cars going out or being returned) at the same exact time. RAC server traffic is usually rather staggered (unless airport locations involved) and the time it takes to process a single outgoing rental or rental return should be a few seconds at most, I would assume.
Have you ran Sysinternals Process Monitor (ProcMon) which records every I/O, including Registry entries?
Process Monitor - Windows Sysinternals | Microsoft Docs
Stand alone executable; nothing to install. All output written to page file, so virtual memory increases really fast.
If you run it, "Run as Administrator"; make sure capture ON; scrolling ON; then after a few million entries, you can turn capture OFF and save the data as an ARN file.
Zip it; upload to One Drive or other legit 3rd party site as the file will be huge - even after a 30-60 minute execution.
Regards. . .
John
p.s. No idea if this info will tell us anything, but it is worth a try. Also, 30-60 minutes may not be a long enough time period.
Enterprise is the name of the server, the other is the domain (Enterprise like Star Trek) =P
All RDP is coming from the local network (maybe 2-3 outside via vpn sometimes). Up to 60 people. I'll take a look at the sysinternals doc and gave it a shot, thank you.
How safe is it to run during work hours? I'm guessing I'd have to run it during actual heavy use to get an accurate idea of what's going on.
So, interesting development. I'm in the server and it's 1am. RAM and CPu usage low but resource monitor is showing 100% disk active time . See screenshot. By the way, when users were complaining, active time was only 1-3%, so not sure if it's related.
I was getting some bad stutter and "not respondings" in task manager and other windows. I could click the start menu fine, though.
Seems to be working ok now after a few minutes but disk active time is still 100%. It does look like our Cloudberry backup is still in progress so that may very well be it..
By the way @jcgriff2 . When i run systernals, do I just hit ok and let it run from there?
I was getting some bad stutter and "not respondings" in task manager and other windows. I could click the start menu fine, though.
Seems to be working ok now after a few minutes but disk active time is still 100%. It does look like our Cloudberry backup is still in progress so that may very well be it..
By the way @jcgriff2 . When i run systernals, do I just hit ok and let it run from there?
Basically YES - When running Process Monitor (procmon.exe), be sure to RIGHT-CLICK; "Run as Administrator". The very first time you run procmon, a EULA will appear - agree to it; no big deal.
The screen should fill instantly with line items of disk activity, including Registry activity as the ProcMon viewer appears. Go to the 4th icon at the top from the left (auto-scroll) of the ProcMon screen and click on it. This turns auto-scroll ON and the items on the screen should now start scrolling up (rather fast - not enough time to really read it) - the 4th icon is the icon under "Event" in the following screenshot - (NOTE: hovering mouse over it should confirm its function). Click on it.
To immediately suspend/stop the scrolling, right-click one time on any part of any line item in the screen. To resume scrolling - click on the 4th icon.
Here is a screenshot of the top-left of the ProcMon viewer for a little help/hint - :) -
ProcMon said:
Initially, let ProcMon run for at least an hour or more, but keep an eye on the size of your page file. We do not want ProcMon to use up all of your RAM, then start heavily using virtual memory as it will bring the server to a crawl and eventually cause it to freeze if the virtual memory (page file) gets ~more than a few hundred MB in size.
To check the size of the page file, run #34, 2nd column "TEXT Output" batch file -
https://www.sysnative.com/SysnativeTutorials/wmi/batch/pagefile_t.bat
This is the WMI batch file source code:
Rich (BB code):
@echo off
::
:: © Sysnative Forums
:: 1 January 2015
:: J. C. Griffith, Microsoft MVP
:: jcgriff2
:: www.sysnative.com
::
set y1=list full
set z1=/format:htable
@title = PalmDesert
@color 1e
prompt $s
echo.
echo.
echo Welcome to Sysnative Forums - WMI Batch
echo.
echo.
echo Running selected batch script now for WMI pagefile
echo.
echo This may take a minute. . .
echo.
echo.
wmic pagefile list full > "%temp%\pagefile.txt"
echo This may take a minute. . . D O N E
echo.
echo.
start /max "notepad" "%temp%\pagefile.txt"
:Go Home - EOJ
exitIt is from this comprehensive WMI Tutorial command page covering every WMI command that I wrote ~6 years ago - (1) Windows Management Instrumentation (WMI) - (Windows 10, 8.1, 8, 7, Vista) | Sysnative Forums
Since it is an unknown Batch executable script file to Windows, Windows will do everything it can to prevent you from running this batch script file. You have to get creative to get around Windows Security. Be sure when downloading the batch file to select a folder like Documents or Downloads so you can easily find the file again.
Once more, be sure to RIGHT-CLICK; "Run as Administrator" on procmon.exe
The only thing that this batch file does is run a WMI (Windows Management Instrumentation) command to REPORT info about your page file. It does not write any data to your system except a very small %temp% text file that it then displays on your screen. It makes no other file or registry changes.
If you cannot get the batch file to run because of Windows Security, run it yourself in an elevated CMD prompt/screen.
Bring up an Admin CMD screen - (5) Open an Elevated Administrative Command Prompt (CMD) - Windows 10 | Sysnative Forums
Copy/paste the following into the Elevated/Admin CMD screen -
Rich (BB code):
wmic pagefile list full > "%temp%\pagefile.txt" & start /max "notepad" "%temp%\pagefile.txt"Give it a few seconds and a full-screen Notepad will appear on your screen that looks like this -
Code:
AllocatedBaseSize=12288
[hi]CurrentUsage=353[/hi]
Description=C:\pagefile.sys
InstallDate=20210106185314.658512-300
Name=C:\pagefile.sys
PeakUsage=354
Status=
TempPageFile=FALSEI'm submitting this post now so I don't lose it and will post again later on this. I'll let mine run for hours longer.
When you finally stop the capture and create an ARN file, (or you can do it while it is running) - click on TOOLS at the top and you can obtain many different types of summary information.
*** BE SURE TO STOP THE CAPTURE FIRST - or it will continue even though the scrolling has stopped too, assuming you've turned that off as well.
Regards. . .
John
p.s. I've noticed an overall system slow down. Per Task Manager, I have exhausted my 32 GB RAM and am running on virtual memory now -
Task Manager reports current RAM usage @ 31.8 GB.
`
Last edited:
When I run procmon64.exe as admin, I get this window:
I simply hit ok there but don't see anything. At the bottom it just seems like it's excluding events and the number keeps climbing.
Has Sysnative Forums helped you? Please consider donating to help us support the site!