Notes on System Center Operations Manager : Management Packs

OpsMgr 2007 R2 - Alert Storm Recognition (possible rule misconfiguration) for SDK alert

MSutara — Tue, 29 Sep 2009 21:18:47 GMT

I described the mitigation for alert storm in the one of my previous post. This mitigation is included out of the box with current deployment of OpsMgr 2007 R2 after system management packs were imported.

As I failed to remember this post (which talks about how to insert alerts from SDK), I also failed to realize that this alert suppression feature will not work unless a change to your custom management pack is made.

I was reminded by Markus, that I need to provide an update in order for this feature to work properly for SDK inserted data generating too much alerts within short period of the time.

History of SDK inserted alert requirements:

When we designed this SDK feature, we decided that it is best for MP (and connector) author to make a copy of condition detection module System.Connectors.GenericAlertMapper, publisher write action System.Connectors.PublishAlert and final composite module type System.Connectors.GenerateAlertFromSdkEvent into their own management pack.

That was the reason why we did not include such definition(s) inside of Microsoft management packs (well some things had changed since then so it is possible we will make this available as public module from one of our system MP in the future).

Management pack with sample modules can be also downloaded directly by clicking this.

Unfortunately this decision has a negative impact on “alert storm mitigation” feature which was delivered as a part current OpsMgr release. This is because implementation of this feature included minor change in the configuration of the module converting input data type to alert (System.Connectors.GenericAlertMapper).

By seeing your current implementation still working, it is clear that such change was not breaking backward compatibility.

Unfortunately, if configuration is not changed for your modules generating alert for SDK inserted data as well, runtime will not initialize component responsible for alert storm recognition and suspension.

Customization needed:

It is required to change the version for all referenced system management packs to OpsMgr R2. This ensures that MP will not be imported to previous releases of OpsMgr where feature is not implemented and would cause alert generating workflow unload. (This is because of failure during configuration processing while loading a module responsible for alert generation.)

<Version>6.1.7221.0</Version>

It is also required to add ManagementGroupName tag into configuration of the module(s). Highlighted areas in XML representation of module types should be self descriptive both for alert generating module as well as final composite module type.

<ConditionDetectionModuleType ID="System.Connectors.GenericAlertMapper" Accessibility="Internal" Batching="false" Stateful="false" PassThrough="false">
  <Configuration>
    <IncludeSchemaTypes>
      <SchemaType>Health!System.Health.AlertSchema</SchemaType>
    </IncludeSchemaTypes>
    <xsd:element name="Priority">
      <xsd:simpleType>
        <xsd:restriction base="xsd:integer">
          <xsd:minInclusive value="0" />
          <xsd:maxInclusive value="2" />
        </xsd:restriction>
      </xsd:simpleType>
    </xsd:element>
    <xsd:element name="Severity">
      <xsd:simpleType>
        <xsd:restriction base="xsd:integer">
          <xsd:minInclusive value="0" />
          <xsd:maxInclusive value="2" />
        </xsd:restriction>
      </xsd:simpleType>
    </xsd:element>
    <xsd:element name="ManagedEntityId" type="xsd:string" />
    <xsd:element name="AlertName" type="xsd:string" minOccurs="0" maxOccurs="1" />
    <xsd:element name="AlertDescription" type="xsd:string" minOccurs="0" maxOccurs="1" />
    <xsd:element name="AlertOwner" type="xsd:string" minOccurs="0" maxOccurs="1" />
    <xsd:element name="AlertMessageId" type="xsd:string" minOccurs="0" maxOccurs="1" />
    <xsd:element name="AlertParameters" type="System.Health.AlertParameters" minOccurs="0" maxOccurs="1" />
    <xsd:element name="Suppression" type="System.Health.Suppression" minOccurs="0" maxOccurs="1" />
    <xsd:element name="WorkflowId" type="xsd:string" />
    <xsd:element name="Custom1" type="xsd:string" minOccurs="0" maxOccurs="1" />
    <xsd:element name="Custom2" type="xsd:string" minOccurs="0" maxOccurs="1" />
    <xsd:element name="Custom3" type="xsd:string" minOccurs="0" maxOccurs="1" />
    <xsd:element name="Custom4" type="xsd:string" minOccurs="0" maxOccurs="1" />
    <xsd:element name="Custom5" type="xsd:string" minOccurs="0" maxOccurs="1" />
    <xsd:element name="Custom6" type="xsd:string" minOccurs="0" maxOccurs="1" />
    <xsd:element name="Custom7" type="xsd:string" minOccurs="0" maxOccurs="1" />
    <xsd:element name="Custom8" type="xsd:string" minOccurs="0" maxOccurs="1" />
    <xsd:element name="Custom9" type="xsd:string" minOccurs="0" maxOccurs="1" />
    <xsd:element name="Custom10" type="xsd:string" minOccurs="0" maxOccurs="1" />
    <xsd:element name="ManagementGroupName" type="xsd:string" />
  </Configuration>
  <ModuleImplementation Isolation="Any">
    <Native>
      <ClassID>2325018e-eef4-41a3-8c17-db831b85c93b</ClassID>
    </Native>
  </ModuleImplementation>
  <OutputType>Health!System.Health.AlertUpdateData</OutputType>
  <InputTypes>
    <InputType>System!System.BaseData</InputType>
  </InputTypes>
</ConditionDetectionModuleType>
 
<WriteActionModuleType ID="System.Connectors.GenerateAlertFromSdkEvent" Accessibility="Public" Batching="false">
  <Configuration>
    <IncludeSchemaTypes>
      <SchemaType>Health!System.Health.AlertSchema</SchemaType>
    </IncludeSchemaTypes>
    <xsd:element name="Priority">
      <xsd:simpleType>
        <xsd:restriction base="xsd:integer">
          <xsd:minInclusive value="0" />
          <xsd:maxInclusive value="2" />
        </xsd:restriction>
      </xsd:simpleType>
    </xsd:element>
    <xsd:element name="Severity">
      <xsd:simpleType>
        <xsd:restriction base="xsd:integer">
          <xsd:minInclusive value="0" />
          <xsd:maxInclusive value="2" />
        </xsd:restriction>
      </xsd:simpleType>
    </xsd:element>
    <xsd:element name="AlertName" type="xsd:string" minOccurs="0" maxOccurs="1" />
    <xsd:element name="AlertDescription" type="xsd:string" minOccurs="0" maxOccurs="1" />
    <xsd:element name="AlertOwner" type="xsd:string" minOccurs="0" maxOccurs="1" />
    <xsd:element name="AlertMessageId" type="xsd:string" minOccurs="0" maxOccurs="1" />
    <xsd:element name="AlertParameters" type="System.Health.AlertParameters" minOccurs="0" maxOccurs="1" />
    <xsd:element name="Suppression" type="System.Health.Suppression" minOccurs="0" maxOccurs="1" />
    <xsd:element name="Custom1" type="xsd:string" minOccurs="0" maxOccurs="1" />
    <xsd:element name="Custom2" type="xsd:string" minOccurs="0" maxOccurs="1" />
    <xsd:element name="Custom3" type="xsd:string" minOccurs="0" maxOccurs="1" />
    <xsd:element name="Custom4" type="xsd:string" minOccurs="0" maxOccurs="1" />
    <xsd:element name="Custom5" type="xsd:string" minOccurs="0" maxOccurs="1" />
    <xsd:element name="Custom6" type="xsd:string" minOccurs="0" maxOccurs="1" />
    <xsd:element name="Custom7" type="xsd:string" minOccurs="0" maxOccurs="1" />
    <xsd:element name="Custom8" type="xsd:string" minOccurs="0" maxOccurs="1" />
    <xsd:element name="Custom9" type="xsd:string" minOccurs="0" maxOccurs="1" />
    <xsd:element name="Custom10" type="xsd:string" minOccurs="0" maxOccurs="1" />
  </Configuration>
  <ModuleImplementation Isolation="Any">
    <Composite>
      <MemberModules>
        <ConditionDetection ID="Mapper" TypeID="System.Connectors.GenericAlertMapper">
          <Priority>$Config/Priority$</Priority>
          <Severity>$Config/Severity$</Severity>
          <ManagedEntityId>$Data/ManagedEntityId$</ManagedEntityId>
          <AlertName>$Config/AlertName$</AlertName>
          <AlertDescription>$Config/AlertDescription$</AlertDescription>
          <AlertOwner>$Config/AlertOwner$</AlertOwner>
          <AlertMessageId>$Config/AlertMessageId$</AlertMessageId>
          <AlertParameters>$Config/AlertParameters$</AlertParameters>
          <Suppression>$Config/Suppression$</Suppression>
          <WorkflowId>$MPElement$</WorkflowId>
          <Custom1>$Config/Custom1$</Custom1>
          <Custom2>$Config/Custom2$</Custom2>
          <Custom3>$Config/Custom3$</Custom3>
          <Custom4>$Config/Custom4$</Custom4>
          <Custom5>$Config/Custom5$</Custom5>
          <Custom6>$Config/Custom6$</Custom6>
          <Custom7>$Config/Custom7$</Custom7>
          <Custom8>$Config/Custom8$</Custom8>
          <Custom9>$Config/Custom9$</Custom9>
          <Custom10>$Config/Custom10$</Custom10>
          <ManagementGroupName>$Target/ManagementGroup/Name$</ManagementGroupName>
        </ConditionDetection>
        <WriteAction ID="WA" TypeID="System.Connectors.LibraryPublishAlert" />
      </MemberModules>
      <Composition>
        <Node ID="WA">
          <Node ID="Mapper" />
        </Node>
      </Composition>
    </Composite>
  </ModuleImplementation>
  <InputType>System!System.BaseData</InputType>
</WriteActionModuleType>

Enjoy alert storm mitigation (customization is possible as described in original post) after importing newly customized management pack!

What is new: OpsMgr 2007 R2 - How to reset monitor state with recovery?

MSutara — Mon, 20 Apr 2009 20:28:00 GMT

Cameron had a nice example of using new R2 feature process monitoring in real life, but that raised a question about feature he wanted to use. Following is report of his issue and how we can help him to address his challenges using already existing feature of OpsMgr 2007.

Scenario: Monitoring a system with a process monitor. Define a recovery to reboot the system if it’s not running the process required. Run this recovery automatically on critical state.

Problem: In OpsMgr prior to R2 when a Recovery was created it had an option to “Reset monitor” which would put it to a healthy state. In R2, this now says “Recalculate State Monitor”. This is representing a challenge as described below:

Challenge: Recalculating the state may keep the monitor in a critical state until the system has been rebooted successfully and is in fact running the process. If the process does not start correctly after reboot, it gets stuck in the critical state and the recovery will not run again. With a Reset of this monitor to a Healthy state, this would work properly, but without that option available I am not seeing an effective way to make this work.

Workaround: Recovery is no different than other workflows loaded by OpsMgr and is rather similar to task. It consists of modules that are chained together and should provide some corrective action in order for monitor to fix its state. For that reason, first module could be the module which resets state of the monitor.

Following is module that could be used with recovery directly. It will reset the state of the monitor specified in configuration.

<xsd:element minOccurs="1" name="MonitorId" type="xsd:string" />

</Configuration>

</OverrideableParameters>

<ManagementGroupId>$Target/ManagementGroup/Id$</ManagementGroupId>

<ManagedEntityId>$Target/Id$</ManagedEntityId>

<MonitorId>$Config/MonitorId$</MonitorId>

</WriteAction>

</MemberModules>

</Composition>

</Composite>

</ModuleImplementation>

<OutputType>System!System.BaseData</OutputType>

<InputType>System!System.BaseData</InputType>

</WriteActionModuleType>

Next is another module which can be used as well. It resets the state of the monitor first and then executes command.

<SchemaType>System!System.CommandExecuterSchema</SchemaType>

</IncludeSchemaTypes>

<xsd:element minOccurs="1" name="ApplicationName" type="xsd:string" />

<xsd:element minOccurs="1" name="WorkingDirectory" type="xsd:string" />

<xsd:element minOccurs="1" name="CommandLine" type="xsd:string" />

<xsd:element minOccurs="1" name="TimeoutSeconds" type="xsd:integer" />

<xsd:element minOccurs="1" name="RequireOutput" type="xsd:boolean" />

<xsd:element minOccurs="1" name="MonitorId" type="xsd:string" />

</Configuration>

<ApplicationName>$Config/ApplicationName$</ApplicationName>

<WorkingDirectory>$Config/WorkingDirectory$</WorkingDirectory>

<CommandLine>$Config/CommandLine$</CommandLine>

<TimeoutSeconds>$Config/TimeoutSeconds$</TimeoutSeconds>

<RequireOutput>$Config/RequireOutput$</RequireOutput>

</WriteAction>

<MonitorId>$Config/MonitorId$</MonitorId>

</WriteAction>

</MemberModules>

</Node>

</Composition>

</Composite>

</ModuleImplementation>

<OutputType>System!System.BaseData</OutputType>

<InputType>System!System.BaseData</InputType>

</WriteActionModuleType>

Sealed MP with both modules is attached to this post.

Sample: Attached is also example providing use of modules with simple event based monitor. Monitor targets instance of “Root Management Server” and that is a reason why management pack also defines a view for state of this entity. When you choose to display “Health explorer”, you should be easily able to locate sample monitor.

One of the recoveries present in attached MP runs automatically with WARNING state. Highlighted is MPElement replacement representing monitor you want to reset. (It should be same as value of the attribute Monitor! Also, please observe that using just a reset module causes its output to be displayed in “Context” tab as well as two state changes will appear to have “same” time of change in Health Explorer.

<Category>Maintenance</Category>

<MonitorId>$MPElement[Name="Microsoft.SystemCenter.Community.Monitors.RecoverySample.EventBasedMonitor"]$</MonitorId>

</WriteAction>

</Recovery>

DISCLAIMER:

Please evaluate in your test environment first! As expected, this solution is provided AS-IS, with no warranties and confers no rights. Use is subject to the terms specified at Microsoft.

What is new: New version of cluster MP released !

MSutara — Fri, 10 Apr 2009 17:30:00 GMT

Pleased to announce that a small but important update to the Cluster Management Pack has now been released. This update focuses on improving scalability of the Management Pack.

The following summarizes the changes to the MP:

· Introduce cook-down for workflows that query the state of clustered resources hosted by the node. Positive effect is based on better and more effective WMI use (one query handles all resources compare to as many queries as many resource needed before).

· Changed the frequency for some workflows and documented how to override them.

· Added various other documentation improvements.

Windows Server 2008 Cluster MP released to web !!!

MSutara — Fri, 13 Feb 2009 03:29:00 GMT

In my opinion, Boris did kick a** job driving this puppy out, especially knowing that Windows hotfix is required for its functionality. This one should help monitoring infrastructure of clustered RMS with future releases of OpsMgr 2007 R2 as well (not beta though)! You can download at:

http://www.microsoft.com/downloads/details.aspx?FamilyId=AC7F42F5-33E9-453D-A923-171C8E1E8E55&displaylang=en&displaylang=en

This release contains some fixes for library and discovery so even Win2k3 cluster users should consider downloading. Thanks for being patient with us!

Clustered Virtual Server 2005 R2 and Operations Manager 2007

MSutara — Thu, 08 Jan 2009 02:20:00 GMT

Problem:

Monitoring of clustered virtual machines (guests) is unreliable with Operations Manager 2007. Instances of Virtual Machine are not monitored without any apparent reason.

Scenario:

Having simple wolfpack cluster (implemented thru Microsoft Cluster Services – MSCS) with just a quorum cluster resource group and clustered Virtual Server 2005 R2 (following this guide). Operations Manager 2007 is installed and health service is pushed to every cluster node. Agents are also pushed to every virtual machine (and those computers become agent managed computers). At the end, Server Virtualization Management Pack for Microsoft System Center Operations Manager 2007 is imported after successful installation and deployment of OpsMgr.

Following is the list of issues one can observe with such setup:

1. Instance of virtual machine (guest) or instance representing windows computer is NEVER monitored when virtual machine cluster resource group is active on the same cluster node as quorum resource group

2. Instance of virtual machine (guest) or instance representing windows computer is monitored fine when its adequate cluster resource group is not active on the same cluster node as quorum resource group

3. Everything works as expected when Server Virtualization MP is not present!

Root cause:

Based on investigation, all instances that are to be monitored by health service running on the virtual machine agent (guest) are disabled. It appears that Server Virtualization MP has not been designed to work under cluster scenarios, mainly due to the fact of associating Virtual Machine Host with virtual computer (cluster).

Possible solution:

Let’s quickly review some MP implementation details. Main discovery targets Microsoft.Windows.Server.Computer (and cluster is discovered as instance of Microsoft.Windows.Cluster.VirtualServer where this type extends Windows Server type). This means that discovery executed on cluster nodes is discovering instances of the VMHost while their virtual machines are active on particular node, but it also means that discovery is executed against quorum cluster resource group (virtual computer).

As mentioned in “Root cause” section, this is, I believe, where MP author might make a mistake with health modeling, as in fact you do not need to associate quorum with virtual machine group as they are able to coexist independently. I base such comment on the fact that virtual machine resource group will stay online when quorum moves to different cluster node. It also remains fully functional and accessible thru VS Administration Website with name of the physical computer (cluster node) used.

Translating this to real life example, having VM1 on node1 and VM2 plus quorum on node2, following seems discovered:

· Instance of VM host for VM1 associated with physical computer 1.

· Instance of VM host for VM2 associated with physical computer 2.

· Instance of VM host for VM2 associated with virtual computer (cluster = quorum)

Investigation revealed that avoiding an association of VM host with virtual computer will fix the issue. This can be done by having DiscoveryPropertyOverride to disable VM host discovery in the context of Microsoft.Windows.Cluster.VirtualServer. (Solution verified while investigating issue.)

<Value>false</Value>

</DiscoveryPropertyOverride>

Implementing workaround:

Really should be as simple as importing attached MP. But I had an environment where following steps had to be taken:

1. Open Operations Console

2. Go to “Administration” -> ManagementPack

3. Select “Microsoft Virtualization Reports” -> right click -> delete

4. Select “Microsoft Virtual Server 2005 R2” -> right click -> delete. If another dependency exist, delete as well (please be careful if such dependency is default MP as diff steps needs to be taken first)

5. Right click in “Administration” pane -> Import management packs

6. Add Microsoft.Virtualization.Reports.mp

7. Add Microsoft.Virtualization.VirtualServer.2005R2.mp

8. Add attached MP

When all management packs are imported at the same time, override is properly applied and such allowed my test environment to work and monitor virtual machines

DISCLAIMER:

Please evaluate in your test environment first! As expected, this solution is provided AS-IS, with no warranties and confers no rights. Use is subject to the terms specified at Microsoft.

Cluster and maintenance mode

MSutara — Sat, 06 Sep 2008 04:54:00 GMT

Maintenance mode and cluster became a big issue for most of our customers. Starting MM for physical computer serving as one of the cluster node causes FALSE-positives alerts raised from cluster infrastructure monitoring MP. Please accept our apologizes, let’s hope this post clears things out a bit as well as provides some insight/tools how to perform this maintenance mode process.

Following is class hierarchy and relationships (part of management pack guide, so it should not be completely new). We can see that managed entity Cluster contains all managed entities which create cluster infrastructure (remember that hosting is specialization of containment introducing lifetime dependency)

1. MANUAL PROCESS

Usual steps to maintenance physical computer that serves as one of the nodes for failover cluster should consist of finding cluster which contains this computer. The one must start maintenance mode for this managed entity (Cluster) while also starting MM for all contained entities as well. Inserting entity with all contained objects causes node (which extends (inherits from) Windows computer) to enter MM for every local application hosted by this computer (though in SP1, Health Service which is local app is exempt from this rule). Only thing left to do is to start MM for each health service and health service watcher associated with cluster node computers.

2. Powershell script

To automate above said work, you can use attached powershell script. It locates all the nodes related to cluster specified by name sent into script as first argument. After it locates cluster nodes, it also enters maintenance mode for heath service and health service watcher associated with those physical computers.

$clusterName = $args[0]

$HoursInMaintenance = $args[1]

$Description = $args[2]

& {

$ErrorActionPreference = "silentlycontinue"

Add-PSSnapin "Microsoft.EnterpriseManagement.OperationsManager.Client" -ErrorVariable errSnapin;

}

$pathOpsMgr = $env:ProgramFiles + "\System Center Operations Manager 2007"

cd $pathOpsMgr

.\Microsoft.EnterpriseManagement.OperationsManager.ClientShell.Startup.ps1

$startTime = (Get-Date).ToUniversalTime()

$endTime = $startTime.AddHours($HoursInMaintenance)

$clusterCriteria="Name='"+ $clusterName +"'"

$cluster = get-monitoringClass -name 'Microsoft.Windows.Cluster' | get-monitoringObject -criteria:$clusterCriteria

$clusterNodeClass=get-monitoringclass -name 'Microsoft.Windows.Cluster.Node'

$nodes=$cluster.GetRelatedMonitoringObjects($clusterNodeClass)

"Putting cluster " + $clusterName + " into maintenance mode recursively"

$cluster.ScheduleMaintenanceMode($startTime,$endTime,"PlannedOther",$Description,"Recursive")

foreach ($node in $nodes)

{

$healthServiceWatcherCriteria = "HealthServiceName='" + $node.Name + "'"

$healthServiceWatcher = get-monitoringclass -name 'Microsoft.SystemCenter.HealthServiceWatcher' | get-monitoringobject -criteria:$healthServiceWatcherCriteria

"Putting HSW " + $node.Name + " into maintenance mode"

& {

$ErrorActionPreference = "silentlycontinue"

New-MaintenanceWindow -startTime:$startTime -endTime:$endTime -monitoringObject:$healthServiceWatcher -comment:$Description

}

$computer = Get-Agent | Where-object {$_.PrincipalName –eq $node.Name}

$healthService = $computer.HostedHealthService

"Putting HS " + $node.Name + " into maintenance mode"

& {

$ErrorActionPreference = "silentlycontinue"

New-MaintenanceWindow -startTime:$startTime -endTime:$endTime -monitoringObject:$healthService -comment:$Description

}

3. Recap

These steps with entering MM for whole cluster will cause temporary suppression of all monitoring for said cluster infrastructure (including cluster nodes (physical computers)) because all workflows are unloaded. There will be no alerting on cluster infrastructure or computer and health service while instances are in maintenance mode. I will try to answer question if it is possible to enter just active cluster node into maintenance mode in some of my next posts.

“Net view” as diagnostic used when heartbeat is missing

MSutara — Fri, 25 Jul 2008 01:25:00 GMT

This all started as question about how “Computer Not Reachable” works. It has been asked in newsgroup and I thought I will help and provide some inside. First, I would like to repeat, that in my opinion regarding the fact that computer not reachable detection is based on the fact that heart beat is missing while targeting “Health Service Watcher” is at least questionable and not very fortunate (it is almost like we try to find causality without root causality engine). Due to performance limitations (Main reasons lying in the fact we did not want to end up having as many workflows doing ICM Ping (which is from OpsMgr perspective a blocking operation) as many computers are present within topology, or not having efficient targeting story for such computer not reachable recognition) we now have this “nasty beast” where we execute diagnostic on heartbeat monitor state change which is later supposed to (thru some set of recoveries) make a monitor state change for “Computer Not Reachable”. This all design was not very reliable in RTM and I had to introduce some mitigations and reliability improvements in SP1 (which off course was not too positive because now two alerts are raised instead of one which was raised in RTM, but believe me, you want reliability J).

Regardless of my runt, I decided to show how to add custom diagnostics for Heartbeat monitor and eventually answer a question and provide management pack which will set the state of some custom monitor based on recovery output. This all could be advance talk, please comment and/or ask question thru this blog if something requires further explanation (this I will not discuss on newsgroup, all this provided info is about undocumented functionality which may break in any future release!).

So to adding custom diagnostic is really easy. In this case all we need (monitor and target) is public and accessible outside of management pack where such elements are defined. At the end it is just adding following raw XML into custom management pack (assuming you have all MP references set properly).

<Diagnostic ID="Microsoft.SystemCenter.Community.Diagnostic.NetViewDiagnostic" Comment="In response to heartbeat failure, net view machine" Accessibility="Internal" Enabled="true" Target="SCLibrary!Microsoft.SystemCenter.HealthServiceWatcher" Monitor="SC2007!Microsoft.SystemCenter.HealthService.Heartbeat" ExecuteOnState="Error" Remotable="true" Timeout="300">

<Category>Maintenance</Category>

<ProbeAction ID="Command" TypeID="System!System.CommandExecuterProbe">

<ApplicationName><![CDATA[%windir%\System32\net.exe]]></ApplicationName>

<WorkingDirectory />

<CommandLine>view $Target/Property[Type="SCLibrary!Microsoft.SystemCenter.HealthServiceWatcher"]/HealthServiceName$</CommandLine>

<TimeoutSeconds>30</TimeoutSeconds>

<RequireOutput>true</RequireOutput>

</ProbeAction>

</Diagnostic>

Now acting based on the result is where real fun starts! First I need to mention that “Computer Not Reachable” is defined as internal, so is not accessible and new monitor must be defined rather then disable original thru override included with management pack. I will not provide much information about why it is “aggregate” monitor, the only thing I will say is that in OpsMgr, aggregate monitor is only monitor without a workflow and runtime is magically making all the state changes. In our case we are setting the state thru recovery, so we do not want to “WASTE” a workflow (and there would be as many of it as many computers within enterprise are monitored) if we never expect that workflow to set the state anyway. Also, there are some “magic” modules I WILL NOT DISCUSS (maybe ever, but we will see in next release), where those modules will set the state of the monitor. (There might be some of you willing to do some reverse engineering and you may get an idea how set state critical when result of recovery provides info about command “net view” failing, and how we set state when command succeeded though.) So here is the recap thru screenshots:

1. After import, monitor state is NEVER set until net view command is executed

2. When RMS recognizes that heart beat is missing, we execute “net view” command inside of diagnostic. When net view succeeds, we set state of monitor to “Healthy”, we set to “Critical” otherwise

Attached is management pack that provides this functionality. It may be used AS IS and confers no rights and support. Enjoy!

Cluster resource groups are not monitored! Is there anything I can do?

MSutara — Sun, 04 May 2008 00:19:00 GMT

Recently I saw some questions about why cluster infrastructure is only partially discovered and it is not monitored when all cluster services are healthy. I replied to some of those questions either internally or at the newsgroup, where I was trying to explain why such health model was possible (so this answer is not subject of this post, but I will explain again if needed; just ask thru comment and I will get back to you). Back to the original problem, I was never really able to provide information what was indeed wrong with cluster infrastructure discovery. In fact, this issue turned rather tricky to repro locally in order to investigate. Luckily, thanks to Brian, one of the possible root causes was containing trailing spaces in cluster resource group name and/or containing such trailing spaces in cluster resource name.

PROBLEM

There is a bug in current discovery script, where value for key property is trimmed when creating instance, but trailing spaces are not removed when creating instance of relationship for that same instance. That causes our discovery insertion module failing discovery data validation when such is to be inserted and following event s provided (all GUIDs may/will differ in your topology):

Event Type: Error
Event Source:        Health Service Modules
Event Category:     None
Event ID:     10801
Date:           4/30/2008
Time:          11:16:41 AM
User:           N/A
Computer:   PACER01D

Description:

Discovery data couldn't be inserted to the database. This could have happened because of one of the following reasons:

- Discovery data is stale. The discovery data is generated by an MP recently deleted.

- Database connectivity problems or database running out of space.

- Discovery data received is not valid.

The following details should help to further diagnose:

DiscoveryId: 7d5fdd7b-2e94-406b-8178-8434625a7885

HealthServiceId: 6127d93a-76c8-c381-a593-4ebd7849ca7e

Invalid relationship target specified in the discovery data item.

RelationshipTargetBaseManagedEntityId: 14a220e0-68b4-59e8-a789-16502957eef2

RuleId: 7d5fdd7b-2e94-406b-8178-8434625a7885

Instance:

<?xml version="1.0" encoding="utf-16"?>

<RelationshipInstance TypeId="{121337e9-cd89-393f-2571-bcfaaf54d1eb}" SourceTypeId="{72700ece-9199-408a-9d74-241df8143e2c}" TargetTypeId="{5bb0294e-6846-b398-740c-06dd351f772a}">

<Settings />

<SourceRole>

<Settings>

<Setting>

<Name>{A0F90386-D16E-43AD-C419-77525FD9F62B}</Name>

<Value>smcluster90</Value>

</Setting>

</Settings>

</SourceRole>

<TargetRole>

<Settings>

<Setting>

<Name>{6A12CC8B-89C2-29A3-3D5A-B3A6266481CD}</Name>

<Value>smcluster90</Value>

</Setting>

<Setting>

<Name>{AB1EB97B-7A96-2544-F713-BA27D1D9BAF3}</Name>

<Value>marius trail space </Value>

</Setting>

</Settings>

</TargetRole>

</RelationshipInstance>.

You could quickly verify if discovery experiencing problems is the one responsible for cluster infrastructure discovery by querying DB directly and looking for discovery ID from event 10801which in my case would be:

use OperationsManager

select discoveryname from discovery where discoveryid = '7d5fdd7b-2e94-406b-8178-8434625a7885'

You could also research using powershell:

get-discovery -criteria "Id like '7d5fdd7b-2e94-406b-8178-8434625a7885'"

SOLUTION

This issue will be addressed in next web release of management packs responsible for monitoring of cluster infrastructure. Closest future release is Windows 2008 Failover cluster management pack, Right now I do not have enough information to provide and disclose detail release date, MP will be verified and deployed in MSIT and in TAP customers to get as much as possible verification of its functionality prior releasing to web. The date I’d say if had to guess is midsummer or early fall timeframe.

This also means that something should or may need to be done prior official fix is available to download. Most obvious solution seems to remove trailing spaces from group or resource names. In the case there are multiple cases with names that contain trailing spaces, or when changing name is simply unacceptable, attached management pack provides workaround discovery script.

This workaround MP file contains new class definition which is extending Microsoft.Windows.Cluster.Group with new property called OriginalGroupName. This property is needed to execute tasks related to group, because in order to succeed, cluster WMI provider requires name which contains trailing spaces. Such approach with defining new managed entity class had to be taken as original class is defined in sealed MP and there was not extending of properties possible, something which can be done when fixing issue while creating next version of original management pack. That is also the reason why new tasks appear in cluster group state view. Another important thing to mention is that cluster discovery interval is set to 30 minutes, which means that discovery should succeed and fix the original problem within this time after management pack reached cluster node. This value is override-able, so can be further customized. Please remember that local discovery cache is not sending data thru the distributed workflow to the database unless properties changed, so with fairly stable cluster system (where by "stable" I mean one where cluster groups and resources no longer change too often or at all) value could be increased (after successful discovered data appeared in state views) just to lower CPU usage on agent itself. Because officially released management packs are required, it also means that event describing discovery data issue will remain to be raised in OperationsManager log. At the same time, unlike with official management pack, only pooling discovery is present which also means that deleting cluster resource group from console will succeed (very likely), but group will stay present in the view until next discovery removes its instace from database, which in fact should happen within interval specified for discovery itself (by default within 30 minutes).

DISCLAIMER

Please import attached management pack in your test environment to evaluate if this cluster infrastructure discovery works for you. It is not sealed and can be further customized if you wish to do so. As expected, this workaround is provided AS IS, with no warranties, and confer no rights. Use of this management pack is subject to the terms specified at Microsoft.

Cluster Infrastructure Monitoring and Versions

MSutara — Fri, 04 Apr 2008 21:31:00 GMT

Boris made this nice workflow in Visio. Please use it with your decision making when importing Windows 2003 Cluster Infrastructure management packs!

Remove old management packs after breaking change was introduced in newer version

MSutara — Thu, 03 Apr 2008 23:18:00 GMT

New version of cluster monitoring management pack was released. Hooray! This release requires removing previously imported version due to breaking changes introduced. Breaking change is a big problem, unfortunately was impossible to avoid as it had to be done to allow monitoring of Windows 2008 Failover Cluster at least with respect to effortlessly reuse common components.

In my post (and others), as well as in management pack guide, there is a paragraph stating this requirement to remove old version of cluster monitoring management pack. Unfortunately, all of us failed to remind our customers about backup of any management packs that for some reason reference any of the cluster monitoring MPs (either custom or those where overrides are stored).

I will reiterate what needs to be done when deleting of the management pack is required in order to import newer version:

1. Locate all management packs that reference MP to be deleted. You can use MPViewer

2. Export those management packs (to back up)

3. Delete all management packs, including your custom MP files

4. Import new MP files with more recent version

5. Re-import backed-up custom management packs (referencing older version of MP within your custom MP is not a problem when newer version of that MP exists in the system)

UPDATE: NEW MP for cluster services (MSCS) monitoring was released!

MSutara — Wed, 02 Apr 2008 20:27:00 GMT

In this post, I mentioned that new version of management pack responsible for monitoring of cluster infrastructure will be available shortly. Release management takes its time here, but now I can announce that it is finally available @ catalog or can be downloaded directly from MS downloads.

What is included with this release:

· Bug fixes monitoring of cluster resource group and cluster resource when name contains special character.

· Bug fixes for proper recognition of partially online cluster resource group.

· Bug fixes for noisy level of disk resource monitoring.

· Bug fixes for proper account usage with all discovery workflows (Cluster profile which needs to be populated when default action account doesn’t hold enough privileges to monitor cluster.)

· Preparation for Windows 2008 Failover Cluster management pack (type space and base functionality is shared between Win2k3 and Win2k8 cluster infra monitoring)

· Additional work including some further minor changes …

Following management pack files are available with this release:

Microsoft.Windows.Cluster.Management.Library.mp (6.0.6277.1)

Microsoft.Windows.Cluster.Management.Monitoring.mp (6.0.6277.1)

Microsoft.Windows.2003.Cluster.Management.Library.mp (6.0.6277.1)

Microsoft.Windows.2003.Cluster.Management.Monitoring.mp (6.0.6277.1)

Similarly to previous version, please follow these steps before importing management packs to meet requirements:

If you have installed, remove any and all versions of following management pack files prior to importing version included with this release:

· Microsoft.Windows.2003.Cluster.Management.Monitoring

· Microsoft.Windows.2003.Cluster.Management.Library

· Microsoft.Windows.Cluster.Management.Monitoring

· Microsoft.Windows.Cluster.Management.Library

NOTE: unfortunately with upcoming release of Windows 2008 Failover Cluster Management Packs, breaking change had to be introduced with Cluster Views and minor change in discovery, where such regrettably prohibits importing new version over any prior imported management packs.

Following are steps required to delete said management packs:

1. Navigate to the Administration Space.

2. Select Management Packs.

3. Select Windows 2003 Cluster Management Monitoring, right-click and select Delete, click Yes.

4. Select Windows 2003 Cluster Management Library, right-click and select Delete, click Yes.

5. Select Windows Cluster Management Monitoring, right-click and select Delete, click Yes.

6. Select Windows Cluster Management Library, right-click and select Delete, click Yes.

Before you import the Windows Server 2003 Cluster Management Pack, please do following actions too:

· Import the Windows Server 2003 Base Operating System Management Pack.

· Enable the Agent Proxy on all agents which are installed on servers that are members of a cluster if it was not enabled already.

Based on OpsMgr 2007 SKU, this is quick reminder about dependencies:

· OpsMgr 2007 RTM (version 06.0.5000.0000):

QFE needs to be applied and Microsoft.Windows.Cluster.Library management pack needs to be imported immediately after binaries on cluster nodes changed. This is required due to dependency on cluster user profile in the case action account has no access to cluster administration and profile needs to be manually populated to allow monitoring of MSCS as well as hot fix required for WMI event provider used with cluster infrastructure monitoring. (QFE is also included with released package)

· OpsMgr 2007 SP1 RC (version 06.0.6246.0000):

No special requirements for runtime are required. But it is required to import Microsoft.Windows.Cluster.Library from QFE (again, it is also attached to web released package). This makes sure you can simply follow steps to download files and successfully import management packs.

· OpsMgr 2007 SP1 RTM (version 06.0.6278.0000):

No special requirements are to be met. All referenced management packs are at their latest version. Simply follow steps to download management pack files and import all except Microsoft.Windows.Cluster.Library.

UPDATE: MP for monitoring cluster services (MSCS) finally web-released!

MSutara — Wed, 19 Mar 2008 06:03:00 GMT

I made a mistake in my post announcing web release of management packs for monitoring of Failover Clusters. I posted incorrect requirements to meet prior importing management packs. I will correct that information now, especially for OpsMgr 2007 SP1 RC build.

These requirements must be met based on the version of System Center Operations Manager 2007 used:

· All OpsMgr 2007 releases:

Enable the Agent Proxy on all agents which are installed on cluster nodes if it was not enabled already. Then import the Windows Server 2003 Base Operating System Management Pack

· OpsMgr 2007 SP1 RC (version 06.0.6246.0000):

No special requirements for runtime are required. But it is required to import Microsoft.Windows.Cluster.Library from QFE (also attached to web released package). This makes sure you can simply follow steps to download files and successfully import management packs.

Update on new web release:

Bunch of major issues were reported so far as part of the feedback.

1. Offline resource causes critical state set for resource group, while it is only partially online and should have been set to warning.(link)

2. Unable to discover and monitor resource group or resource when special character is present in the name.(link)

3. Full disk notification is too aggressive and causes a flood of alerts to be raised.(link)

Those were addressed already and new web-release will happen shortly. It will also integrate couple of changes necessary for monitoring of Windows 2008 Failover Cluster (coming to web release later).

MP for monitoring cluster services (MSCS) finally web-released!

MSutara — Tue, 04 Mar 2008 22:19:00 GMT

Cluster Management M-Packs finally hit our MP catalog. These management packs monitor health of the Windows Server 2003 clusters and are designed to improve the availability, performance, and security of your Windows Server 2003 cluster deployment

Getting the Latest Management Pack and Management Pack Documentation

To monitor Windows Server 2003 clusters, you must first download the Windows Server 2003 Cluster Management Pack from the Management Pack Catalog; see "System Center Operations Manager 2007 Catalog" or download by clicking here. There will be a single installer that will extract all management pack files. Released package also contains QFE hotfix (KB article to become publicly available soon), which must be applied and installed when trying to import cluster MP on build prior SP1 RC (6246) or SP1 RTM (6278) – see bellow.

Before You Import the Management Pack

Before you import the Windows Server 2003 Cluster Management Pack, take the following actions:

· Import the Windows Server 2003 Base Operating System Management Pack.

· Enable the Agent Proxy on all agents which are installed on servers that are members of a cluster if it was not enabled already.

Also, following requirements must be met based on the version of System Center Operations Manager 2007 used:

· OpsMgr 2007 RTM (version 06.0.5000.0000)
QFE needs to be applied and Microsoft.Windows.Cluster.Library management pack needs to be imported immediately after binaries on cluster nodes changed. This is required due to dependency on cluster user profile in the case action account has no access to cluster administration and profile needs to be manually populated to allow monitoring of MSCS as well as hotfix required for WMI event provider used with cluster infrastructure monitoring.

· OpsMgr 2007 SP1 RC (version 06.0.6246.0000) and OpsMgr 2007 SP1 RTM (version 06.0.6278.0000)
No special requirements are to be met. All referenced management packs are at their latest version. Simply follow steps to download files and import management packs (all except Microsoft.Windows.Cluster.Library)

Objects the Windows Server 2003 Cluster Management Pack Discovers

The Windows Server 2003 Cluster Management Pack automatically discovers the following object types:

· Windows Clusters

· Windows Cluster

· Windows Server 2003 Cluster Service

· Windows Server Cluster Node

· Windows Server Cluster Network

· Windows Server 2003 Cluster Network Interface

· Windows Server 2003 Resource Group

· Windows Server Hosted Resource Group

· Windows Server Cluster Resource

Classes

The following diagram shows defined classes.

How Health Rolls Up

The following illustration shows how health rolls up in this management pack.

Screenshots

Maintenance Mode And Server Availability (Another Solution)

MSutara — Sun, 06 Jan 2008 07:50:00 GMT

Maintenance mode expired, monitored server is still down, but there is no alert in the alert view of OpsMgr 2007 console indicating server being offline.

In my previous post I tried to walk through one possible implementation of the workaround to recognize server unavailability. That post also contains information about reasons behind scenario described above so I won’t repeat myself here. I will spend short time to discuss impact of that workaround to RMS and compare with solution I will suggest in this post though. One of the biggest “issues” with previous implementation is potential duplication of alert. Dependency monitor rolls up health state of the instance of its contributing health service and it is likely that alert was raised already. Another “issue” is duplication of health service watcher dependency monitor just to get alert for monitored health service unavailability. I suggested disabling original dependency monitor (Local Health Service Availability), but if that was not done, workflow count is affected and increased by count of health services within enterprise.

This post will try to suggest different solution to avoid alert duplication problem. It still uses dependency monitor and its feature to set the state its unavailable contributing entity uses. Difference is that dependency monitor will use always healthy monitor of contributing entity to guarantee that unhealthy state is a result of unavailability only and is never a result of health state of the contributing entity. Here is step by step set up and snapshot.

· Create a unit monitor targeted to the Health Service class that is always going to be green

· Create a dependency monitor targeted to Health Service Watcher class

· Set this monitor to roll up health from the always healthy unit monitor (created in Step 1)

· Set this monitor MemberUnAvailable configuration to Error

· Configure the alert for this dependency monitor

This will ensure that alert is generated when the Health Service is actually not available even after the computer leaves maintenance mode. One more benefit is it will also alert when the Health Service is not started after computer comes back up.

Unfortunately this implementation still affects workflow count on RMS. There will be an addition of that many workflows as many health service watcher instances your enterprise contains. This means that you need to decide if want to live with increased TCO with possible duplicate alert from previous post, or if your RMS is able to withstand heavier workflow load.

To test and use this implementation, please import attached management pack in your test environment to evaluate if this workaround works for you. It is not sealed and can be further customized if you wish to do so. Test entering MM for all entities as described here (you can use script from this post), shutdown server and wait for alert.

Maintenance Mode And Server Availability

MSutara — Fri, 04 Jan 2008 22:11:00 GMT

Maintenance mode expired, monitored server is still down, but there is no alert in the alert view of OpsMgr 2007 console indicating server being offline. This scenario is happening too often with current implementation of availability monitoring. What our customer face and fear is that if server which undergo maintenance mode each night is not rebooted successfully, there is no indication about its unavailability and this could lead to at least bad user experience if not more …

As described in this knowledge base article, one needs to put computer, health service hosted by maintained computer as well as watcher monitoring this health service remotely into maintenance mode to avoid unexpected unavailability alerts. But “unfortunately”, this is also a direct reason why alert is not generated when maintenance mode terminated.

When instance of the managed entity enters MM, all its monitoring is suspended. When it leaves MM, monitoring is “restarted” with clean slate. As there is not a probe detecting the current availability of instance implemented with current version of OpsMgr 2007, there is no indication about its unavailability. Luckily, there is a runtime component which is always aware of current availability of instances it cares about. This component is dependency monitor. It is always aware of availability status of its contributing instances. This gives us hope and chance to implement alert mechanism which notifies about possible instance unavailability.

Back to the subject of this post, workaround for health service availability. Instance of health service indicates the possibility to monitor discovered instances of other entities. Instance of health service watcher monitors availability of health service instance. There is already a relationship between these two entities and that allows for creation of the dependency monitor. If we would inspect monitor topology for health service watcher, we can see that there is such dependency monitor already.

Unfortunately this dependency monitor doesn’t allow for customization of alert, if one was enabled thru “Generate Alert” override, simply because there is no alert configuration with released dependency monitor.

Such “issues” indicate that workaround should implement its own dependency monitor. This new monitor will be “equal” to existing one, only difference is it defines alert configuration. It also means we could disable original dependency monitor (Local Health Service Availability) because new monitor will provide same monitoring for us.

How this new monitor changes state to error? One possible way is that instance of contributing health service experiences problem and contributes with state error. Another possibility is to use health state “error” in the case contributing instance is not available.

Here is a copy of snapshot from the “Knowledge base” I used with monitor implemented in this workaround. It tries to explain how to troubleshoot its error state and how to recognize (to some extent) that health service is not available.

Causes

The error health state for this dependency monitor could be caused by unavailability of watched health service. Alert is generated each time.

Resolutions

The state change event tab and context of the state change of this monitor needs to be investigated. ICMP ping diagnostic should be executed and its result carries information about availability of computer which is hosting watched health service.

Computer is online when there is no diagnostic output available. Contributing monitor hierarchy should be expanded and will point to problem with local state of watched health service.

You can try the following to diagnose and remediate issue when computer is not online: (when diagnostic output exists)

· Perform a trace route to check if there is packet loss to a router or switch to the target Health Service. You can use the tracert.exe utility in Windows. If you have Window XP or Windows Server 2003 and higher, you can use pathping.exe

· Physically check the target computer to ensure it is connected or plugged into the network.

· Check with your network administrators if there are any known issues or outages that may be affecting the target Health Service and its parent Management Server.

Please import attached management pack in your test environment to evaluate if this workaround works for you. It is not sealed and can be further customized if you wish to do so. Test entering MM for all entities as described here (you can use script from this post), shutdown server and wait for alert. Steps from knowledge base article should be self explanatory then.