Health Service restarts even after installing the following hotfixes (monitoringhost.exe) on X64 Systems running Exchange 2007.
950853 A memory leak occurs when you monitor Exchange Server 2007 by using the MOM 2007 agent in System Center Operations Manager 2007
http://support.microsoft.com/default.aspx?scid=kb;EN-US;950853
951979 Problems occur on a management server that is running System Center Operations Manager 2007 Service Pack 1 when certain management packs are installed
http://support.microsoft.com/default.aspx?scid=kb;EN-US;951979
You'll see the following event being logged in the Operations Manager Event log.
Event Type: Warning
Event Source: Health Service Script
Event Category: None
Event ID: 6026
Date: 6/26/2008
Time: 8:59:55 PM
User: N/A
Computer: SGBD012512
Description:
LaunchRestartHealthService.js : Launching Restart Health Service. Monitoring Host exceeded Process\Private Bytes threshhold.
For more information, see Help and Support Center at http://go.microsoft.com/fwlink/events.asp.
<><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><>
From Microsoft SCOM product development:
On Exchange 20007 servers the .Net runtime (CLR) gets loaded into MonitoringHost.exe. The way Windows performance counters work is that to read a performance counter from an application a DLL provided by that application is loaded into your process. Since some of the Exchange services (ex hub transport) are written in managed code the CLR performance counter client is loaded into our process for reading those counters. The .net runtime performance counter client implementation loads the CLR to handle reading the counters from the managed service.
The CLR handles memory management via a garbage collection system rather than traditional native code where you explicitly allocate and de-allocate memory. In a garbage collection system the application doesn't explicitly release memory. Periodically the garbage collector will run to see what objects are no longer being used and then free up the memory for those objects. The CLR garbage collector is designed to monitor system load so that if there is no memory pressure then it doesn't need to run as often. When the garbage collector runs it will internally release objects and make space for new objects. However, it may not trim the amount of memory it has requested from the operating system if it detects there is no memory pressure on the system. If the system starts seeing memory pressure then the CLR may start decreasing the amount of memory it has asked for.
On large systems like these exchange servers our default private byte threshold of 100MB ends up being too small. Since there is limited memory pressure on these systems the CLR will commit more than 100MB of memory and not de-allocate until there is more memory pressure on the system.
Override the Monitoring Host Rule Threshhold of 100Mb, most customers have had good success with a value of 200Mb however some have chosen to triple the threshhold. There are recommendations to even increase to even 600Mb.
To override this for a specific Agent:
1.) Open the Authoring Pane of the Operations Manager Console
2.) Select Management Pack Objects > Rules
3.) Click Scope and choose Agent
4.) In the Look For: space type 'Private bytes' w/out the quotes and click find now. The rule we want to override is:
Monitoring Host Private Bytes Threshold
5.) Right click on the rule and choose Overrides > Override the rule > For a specific object of type
6.) Choose the agent that you would like to override the threshhold for in the Select Object pane
7.) Select Destination Management Pack, if neccessary create a new MP for these overrides. NOTE: It is not recommended to use the Default Management Pack.
8.) Click the Enabled check box for overrides, ensure that it is set to 'True'.
9.) Click the Threshold check box and enter the new value 209715200 (this doubles it).
The second piece to this is that you will also want to override the value in the 'Health Service Private Bytes Threshold' monitor to send an alert or display health state for these agents.
Thanks to Marc Reyhner, Ake Pettersson for this information.
