Notes on System Center Operations Manager : Discovery

Clustered Virtual Server 2005 R2 and Operations Manager 2007

MSutara — Thu, 08 Jan 2009 02:20:00 GMT

Problem:

Monitoring of clustered virtual machines (guests) is unreliable with Operations Manager 2007. Instances of Virtual Machine are not monitored without any apparent reason.

Scenario:

Having simple wolfpack cluster (implemented thru Microsoft Cluster Services – MSCS) with just a quorum cluster resource group and clustered Virtual Server 2005 R2 (following this guide). Operations Manager 2007 is installed and health service is pushed to every cluster node. Agents are also pushed to every virtual machine (and those computers become agent managed computers). At the end, Server Virtualization Management Pack for Microsoft System Center Operations Manager 2007 is imported after successful installation and deployment of OpsMgr.

Following is the list of issues one can observe with such setup:

1. Instance of virtual machine (guest) or instance representing windows computer is NEVER monitored when virtual machine cluster resource group is active on the same cluster node as quorum resource group

2. Instance of virtual machine (guest) or instance representing windows computer is monitored fine when its adequate cluster resource group is not active on the same cluster node as quorum resource group

3. Everything works as expected when Server Virtualization MP is not present!

Root cause:

Based on investigation, all instances that are to be monitored by health service running on the virtual machine agent (guest) are disabled. It appears that Server Virtualization MP has not been designed to work under cluster scenarios, mainly due to the fact of associating Virtual Machine Host with virtual computer (cluster).

Possible solution:

Let’s quickly review some MP implementation details. Main discovery targets Microsoft.Windows.Server.Computer (and cluster is discovered as instance of Microsoft.Windows.Cluster.VirtualServer where this type extends Windows Server type). This means that discovery executed on cluster nodes is discovering instances of the VMHost while their virtual machines are active on particular node, but it also means that discovery is executed against quorum cluster resource group (virtual computer).

As mentioned in “Root cause” section, this is, I believe, where MP author might make a mistake with health modeling, as in fact you do not need to associate quorum with virtual machine group as they are able to coexist independently. I base such comment on the fact that virtual machine resource group will stay online when quorum moves to different cluster node. It also remains fully functional and accessible thru VS Administration Website with name of the physical computer (cluster node) used.

Translating this to real life example, having VM1 on node1 and VM2 plus quorum on node2, following seems discovered:

· Instance of VM host for VM1 associated with physical computer 1.

· Instance of VM host for VM2 associated with physical computer 2.

· Instance of VM host for VM2 associated with virtual computer (cluster = quorum)

Investigation revealed that avoiding an association of VM host with virtual computer will fix the issue. This can be done by having DiscoveryPropertyOverride to disable VM host discovery in the context of Microsoft.Windows.Cluster.VirtualServer. (Solution verified while investigating issue.)

<Value>false</Value>

</DiscoveryPropertyOverride>

Implementing workaround:

Really should be as simple as importing attached MP. But I had an environment where following steps had to be taken:

1. Open Operations Console

2. Go to “Administration” -> ManagementPack

3. Select “Microsoft Virtualization Reports” -> right click -> delete

4. Select “Microsoft Virtual Server 2005 R2” -> right click -> delete. If another dependency exist, delete as well (please be careful if such dependency is default MP as diff steps needs to be taken first)

5. Right click in “Administration” pane -> Import management packs

6. Add Microsoft.Virtualization.Reports.mp

7. Add Microsoft.Virtualization.VirtualServer.2005R2.mp

8. Add attached MP

When all management packs are imported at the same time, override is properly applied and such allowed my test environment to work and monitor virtual machines

DISCLAIMER:

Please evaluate in your test environment first! As expected, this solution is provided AS-IS, with no warranties and confers no rights. Use is subject to the terms specified at Microsoft.

Cluster resource groups are not monitored! Is there anything I can do?

MSutara — Sun, 04 May 2008 00:19:00 GMT

Recently I saw some questions about why cluster infrastructure is only partially discovered and it is not monitored when all cluster services are healthy. I replied to some of those questions either internally or at the newsgroup, where I was trying to explain why such health model was possible (so this answer is not subject of this post, but I will explain again if needed; just ask thru comment and I will get back to you). Back to the original problem, I was never really able to provide information what was indeed wrong with cluster infrastructure discovery. In fact, this issue turned rather tricky to repro locally in order to investigate. Luckily, thanks to Brian, one of the possible root causes was containing trailing spaces in cluster resource group name and/or containing such trailing spaces in cluster resource name.

PROBLEM

There is a bug in current discovery script, where value for key property is trimmed when creating instance, but trailing spaces are not removed when creating instance of relationship for that same instance. That causes our discovery insertion module failing discovery data validation when such is to be inserted and following event s provided (all GUIDs may/will differ in your topology):

Event Type: Error
Event Source:        Health Service Modules
Event Category:     None
Event ID:     10801
Date:           4/30/2008
Time:          11:16:41 AM
User:           N/A
Computer:   PACER01D

Description:

Discovery data couldn't be inserted to the database. This could have happened because of one of the following reasons:

- Discovery data is stale. The discovery data is generated by an MP recently deleted.

- Database connectivity problems or database running out of space.

- Discovery data received is not valid.

The following details should help to further diagnose:

DiscoveryId: 7d5fdd7b-2e94-406b-8178-8434625a7885

HealthServiceId: 6127d93a-76c8-c381-a593-4ebd7849ca7e

Invalid relationship target specified in the discovery data item.

RelationshipTargetBaseManagedEntityId: 14a220e0-68b4-59e8-a789-16502957eef2

RuleId: 7d5fdd7b-2e94-406b-8178-8434625a7885

Instance:

<?xml version="1.0" encoding="utf-16"?>

<RelationshipInstance TypeId="{121337e9-cd89-393f-2571-bcfaaf54d1eb}" SourceTypeId="{72700ece-9199-408a-9d74-241df8143e2c}" TargetTypeId="{5bb0294e-6846-b398-740c-06dd351f772a}">

<Settings />

<SourceRole>

<Settings>

<Setting>

<Name>{A0F90386-D16E-43AD-C419-77525FD9F62B}</Name>

<Value>smcluster90</Value>

</Setting>

</Settings>

</SourceRole>

<TargetRole>

<Settings>

<Setting>

<Name>{6A12CC8B-89C2-29A3-3D5A-B3A6266481CD}</Name>

<Value>smcluster90</Value>

</Setting>

<Setting>

<Name>{AB1EB97B-7A96-2544-F713-BA27D1D9BAF3}</Name>

<Value>marius trail space </Value>

</Setting>

</Settings>

</TargetRole>

</RelationshipInstance>.

You could quickly verify if discovery experiencing problems is the one responsible for cluster infrastructure discovery by querying DB directly and looking for discovery ID from event 10801which in my case would be:

use OperationsManager

select discoveryname from discovery where discoveryid = '7d5fdd7b-2e94-406b-8178-8434625a7885'

You could also research using powershell:

get-discovery -criteria "Id like '7d5fdd7b-2e94-406b-8178-8434625a7885'"

SOLUTION

This issue will be addressed in next web release of management packs responsible for monitoring of cluster infrastructure. Closest future release is Windows 2008 Failover cluster management pack, Right now I do not have enough information to provide and disclose detail release date, MP will be verified and deployed in MSIT and in TAP customers to get as much as possible verification of its functionality prior releasing to web. The date I’d say if had to guess is midsummer or early fall timeframe.

This also means that something should or may need to be done prior official fix is available to download. Most obvious solution seems to remove trailing spaces from group or resource names. In the case there are multiple cases with names that contain trailing spaces, or when changing name is simply unacceptable, attached management pack provides workaround discovery script.

This workaround MP file contains new class definition which is extending Microsoft.Windows.Cluster.Group with new property called OriginalGroupName. This property is needed to execute tasks related to group, because in order to succeed, cluster WMI provider requires name which contains trailing spaces. Such approach with defining new managed entity class had to be taken as original class is defined in sealed MP and there was not extending of properties possible, something which can be done when fixing issue while creating next version of original management pack. That is also the reason why new tasks appear in cluster group state view. Another important thing to mention is that cluster discovery interval is set to 30 minutes, which means that discovery should succeed and fix the original problem within this time after management pack reached cluster node. This value is override-able, so can be further customized. Please remember that local discovery cache is not sending data thru the distributed workflow to the database unless properties changed, so with fairly stable cluster system (where by "stable" I mean one where cluster groups and resources no longer change too often or at all) value could be increased (after successful discovered data appeared in state views) just to lower CPU usage on agent itself. Because officially released management packs are required, it also means that event describing discovery data issue will remain to be raised in OperationsManager log. At the same time, unlike with official management pack, only pooling discovery is present which also means that deleting cluster resource group from console will succeed (very likely), but group will stay present in the view until next discovery removes its instace from database, which in fact should happen within interval specified for discovery itself (by default within 30 minutes).

DISCLAIMER

Please import attached management pack in your test environment to evaluate if this cluster infrastructure discovery works for you. It is not sealed and can be further customized if you wish to do so. As expected, this workaround is provided AS IS, with no warranties, and confer no rights. Use of this management pack is subject to the terms specified at Microsoft.

Backwards Compatibility Mapper fails to start event on OpsMgr 2007 SP1

MSutara — Wed, 12 Mar 2008 09:18:00 GMT

Many of you (especially those trying to monitor Exchange using converted management pack) had noticed that with Operations Manager 2007 SP1, noisy event complaining about transient state of instance discovery appeared inside of agent’s event log rather frequently ultimately causing its flooding.

Sample event:

Event Type:     Warning
Event Source:   Health Service Modules
Event Category: None
Event ID: 10720
Date:      3/11/2008
Time:      9:06:45 PM
User:      N/A
Computer: SMCNODE10

Description:
The Backwards Compatibility mapper module was not able to start because discovery has not fully completed for this Health Service. This is a transient issue, the affected rule(s) will be unloaded and the Health Service will restart them when initialization is complete.

One or more workflows were affected by this.

Workflow name: System.Mom.BackwardCompatibility.ServiceStateMonitoring
Instance name: SMCLUSTER05.smx.net
Instance ID: {7E669FE2-B4F9-00CB-1E1D-99B07E791F91}
Management group: VerifyMP_SP1

Root cause of this issue lies in the fact that backward compatibility feature requires some properties for instance of Microsoft.Windows.Computer being always populated after discovery completes. And as you probably guessed, with cluster involvement, there will be an instance which unfortunately doesn’t fulfill such requirement. It is instance of Microsoft.Windows.Cluster.VirtualServer. Following is class hierarchy for that managed entity type:

VirtualServer really is a special managed entity, originally designed to help the discovery of cluster aware application with optimized (not converted) management pack (In the near future I will try to get to post comments about how it should be used with such discovery). It was never really meant to serve as a target for a real monitoring workflow and as such, it only populates some properties on top of keys (keys are always populated – that is how instance is recognized by OpsMgr). VirtualServer as defined by OpsMgr is a cluster resource group with cluster resource IPAddress and cluster resource NetworkName. Because it is also a Computer, it will also appear in Computer state view and as said before, without any unit monitors targeting this instance, it will be marked as not monitored with all so far released versions of OpsMgr. It is also going to appear inside of “Agenless Managed” due to the fact that such instance never hosts HealthService. Following is a T-SQL that one could use to investigate some of the computer properties (just replace <partial name of virtual server> below with real substring identifying your VirtualServer)

use OperationsManager

select BME.DisplayName, C.PrincipalName, C.NetworkName, C.NetbiosComputerName, C.DNSName

from MTV_Computer C

join BaseManagedEntity BME with(nolock) on BME.BaseManagedEntityId = C.BaseManagedEntityId

join ManagedType MT with(nolock) on MT.ManagedTypeId = BME.BaseManagedTypeId

where BME.IsDeleted = 0

and MT.TypeName = 'Microsoft.Windows.Computer'

and BME.DisplayName like '%<partial name of virtual server>%'

Following is query result for smcluster05 prior applying workaround:

We can see that problem is that there is no value for DNSName property. This issue was already addressed with future release of OpsMgr 2007 (SP2), but there are some workarounds that could be applied before it is available to public. The one that I will introduce in this post consists of a discovery rule, which will populate property DNSName. Such discovery rule will be targeted to the instance of VirtualServer. OpsMgr guarantees that VS instance is monitored by health service which is running on the active cluster node (node where cluster resource group is currently online). With a failover, monitoring of the instance is stopped and is started on new active node once failover finishes. That guarantees that discovery rule populating DNSName property will make sure that its value will remain filled and available for backward compatibility feature to be used.

Following screenshot displays the fact that after MP is delivered to the health service running on the cluster node (event 1201), there could be a set of additional events 10720 before discovery rule is loaded, executed and data delivered to the operational DB. Events will no longer appear after final configuration reload event (1210).

Executing query again shows that property is now populated:

Please import attached management pack in your test environment to evaluate if this workaround works for you. It is not sealed and can be further customized if you wish to do so. As expected, this workaround is provided AS IS, with no warranties, and confer no rights. Use of included samples is subject to the terms specified at Microsoft.

OpsMgr2007 SP1 RC rolling ugrade and cluster aware application monitoring.

MSutara — Mon, 12 Nov 2007 23:25:00 GMT

Rolling upgrade of OpsMgr2007 is performed the way that server bits are upgraded, new management packs are requested and delivered and those are later pushed down to agents. Operations Manager 2007 SP1 Release Candidate contains an issue when monitoring of cluster aware application (like SQL, Exchange …) is present within topology.

This issue is manifesting itself by unloading several rules responsible for discovering instance of Virtual Server as well as monitoring failover of cluster resource group matching that particular instance of Virtual Server. It is caused by changes to configuration of native modules defined inside of Microsoft.Windows.Cluster.Library management pack.

Rules will be unloaded until new binaries are approved, delivered and replace binaries shipped with Microsoft operations Manager 2007 RTM. Impact of this unload is that during the rolling upgrade of OpsMgr2007 SP1 RC, newly created cluster aware applications are not discovered, already present cluster aware applications are not “actively” monitored if they had failed over to another cluster node or when cluster service stopped, paused or crashed.

Rules are re-loaded and are able to process new configuration once binaries present with OpsMgr2007 SP1 RC are replaced and loaded by runtime on agent (health service) running on cluster node.

Sample of event notifying about discovery rule unload:

Event Type:   Error
Event Source:          HealthService
Event Category:       Health Service
Event ID:       4511
Date:            11/09/2007
Time:            9:26:27 AM
User:            N/A
Computer:     testBox

Description:
Initialization of a module of type "ClusterDiscoveryDS" (CLSID "{97B1EF21-757C-4004-86BB-57939E2C98D8}") failed with error code “Element not found” causing the rule "Microsoft.Windows.Cluster.Classes.Discovery" running for instance "Cluster Service" with id:"{0753905A-5ACE-5C70-1B0A-7980743053FA}" in management group “marius”

Agent Proxying Alert

MSutara — Fri, 09 Nov 2007 21:59:00 GMT

Agent proxying needs to be enabled when health service discovers instance of some managed entity type for which management context doesn’t provide information that this exact health service will also monitor this instance.

For example proxying must be enabled when discovering instances of Virtual Server on cluster nodes (instance of Virtual Server will be managed by health service running on active cluster node where matching cluster resource group resides); it should be enabled for AD topology discovery etc.

Failure to insert discovery data when proxying was required was always present, but with SP1 release of Operations Manager 2007 we added alert that notifies operator. This alert should help to distinguished if it is desired to insert discovery data after inspecting alert context.

In the case alert is resolved without enabling proxying, it will be reprised next time discovery properties for that instance had changed. It will also be reprised when agent’s heath service restarted as this action always causes discovery of all instances to be managed by this health service to be reinserted.

Sample alert:

Agent proxying needs to be enabled for a health service to submit discovery
data about other computers.

Source: contoso.com
Path: contoso.com\contoso.com

Alert Rule:
Microsoft.SystemCenter.DiscoveryWriteActionModule.AgentProxyingNotEnabled.Alert

Alert description: Details:Health service (B261D17B-CF3C-41A7-7D54-90B8A03EE7D3) should not generate data about this managed object ( 889DC254-F9E6-D44F-A1FF-265F525B88E3 ).

Troubleshoot:

Execute following T-SQL against OperationsManager database:

select * from basemanagedentity
where basemanagedentityid = 'B261D17B-CF3C-41A7-7D54-90B8A03EE7D3'

Make sure that proxy is enabled for health service matching Path property.