Notes on System Center Operations Manager : Health space

Tool: OpsMgr 2007 - RuntimeHealthExplorer

MSutara — Thu, 04 Jun 2009 00:12:00 GMT

Did you ever wonder what is the state of the instance as known to the runtime (health service) monitoring it? Did you believe that some state changes are unaccounted for? Did you see discrepancy in Health Explorer?

I believe many of you may answer yes to one of these questions.

Right now, there really is not a good guidance on how to troubleshoot state change problems, but since OpsMgr 2007 SP1 release, there was a way to at least display states of the monitors targeting the instance as recorded by runtime during state calculation. This led me to creation of the tool returning those states from runtime. It also provides visual comparison against “real” Health Explorer (states are returned from Ops DB) while integrated with OpsMgr console thru console task. This task targets instance of “HealthService” managed entity type. Tool uses Health Explorer like view of monitors for each active instance monitored by specific runtime. Following is a snapshot of the tool executed against my Root Management Server. Please observe that I created view listing all health service instances as well as console task associated with this type and accessible thru “Actions” pane.

There still may be a long way for us to recognize all the issues and take corrective actions automatically. That is the reason why this tool provides at least a manual way to synchronize states of the monitors associated with instance into operational DB – right click anywhere in tree control and select “Synchronize to DB”. Unfortunately, such corrective action is unable to synchronize state of the dependency rollup monitor, but I will try to find the way to achieve this although plumbing is not present in current implementation of the runtime yet!

Honestly, I’m not expecting big download count. But it is possible this tool helps someone with investigation of health state issues and that is the main reason why I do this post anyway. Tool works for OpsMgr2007 SP1 and R2!

Based on the feedback (if any) I may try to extend feature set in future versions too (permitting my bandwidth).

DISCLAIMER:

Please evaluate in your test environment first! As expected, this solution is provided AS-IS, with no warranties and confers no rights. Use is subject to the terms specified at Microsoft.

Link to x86 installation package

Link to x64 installation package

HOWTO: “restart” monitoring of my environment?

MSutara — Sat, 10 Jan 2009 01:45:00 GMT

My friend Cameron and I discussed following issue as one of the challenges he occasionally faces with his customers. In order to minimize his TCO and manual interventions, I promised to help even when such design request is not making it in feature set for our next release.

Problem Description:

There may be legitimate situations where a customer needs to reset many health monitors at once. For example after a network outage, there might be a significant number of alerts which may have been generated as well as the health state of various items becomes unhealthy. Another case is incorrect approach to Maintenance Mode may cause similar outcome, especially when manual reset monitors or alerts generated without “auto-resolve” feature are present in instances involved with maintenance.

To address this type of situation, the bulk of alerts from the outage need to be closed (which can be done with a PowerShell script). Also, resetting of the health state for multiple systems is required but not viable and manual intervention is needed.

His proposal was that it should be possible to select multiple servers and force their health back to green. Specifically, the health model for those instances would be walked and each monitor not Healthy is reset. This would “restart” the environment to green so that only real issues would resurface as alerts recurred and the states would be updated.

Analyzing proposal:

It is already possible to use SDK tasks to accomplish this proposal. It is event achievable to “speed” up the recognition of real issues by submitting additional “recalculate” state task for given instance (where this task forces to recalculate what the state of given instance should be (at the time of the execution) by working with on-demand detection (assuming that such detection is defined for monitor types used for monitoring of that same instance)).

My approach to implementing this proposal was little different than stated above. I’m not finding every unhealthy monitor, but crawl relationship tree for selected instance recursively adding each instance contributing to the overall health. While making sure instance is present just once, result of reset request against each of those instances affects the health state of all other instances that depend on its state either directly or indirectly.

Note:
Following post contains video trying to describe the difference between Reset and Recalculate tasks. It also touches bases on what does “on-Demand” detection means etc. Please contact me thru comments if I should try to provide additional/different explanation of those monitor features.

Solution:

Attached, you can find source code for my solution as well as installers for deployment of already built binaries. I provide two types of integration with our operations console.

First is having a task associated with managed entity “Microsoft.SystemCenter.ComputerGroup”. This will become present when installation of “RestartMonitoringSetup” for particular SKU succeeds. Following is screenshot providing self-descriptive use of the task:

Second possible integration is using the fact that console is able to act like a browser. Deployment is performed by RestartMonitoringWebSetup and consists of creating Web application and MP import. Web application allows regular web browser to act as the tool which triggers requested restart action. MP associated with this approach contains following WEB view to allow integration with console:

Choosing option with group allows “restart” of the monitoring for all instances contained within all selected groups. Such operation may become rather consuming, as I hinted above, instance space is crawled and all necessary instances (contributing directly or indirectly) are asked to reset and then recalculate their state.

Option to restart monitoring for which active alert is present is doing similar operation as the one made for group, only difference is that likely-hood of having many instances contributing to overall health state is smaller that it is with group (or multiple groups for that matter).

Note:
Additional warning is that tool is not smart enough to recognize if there is an alert raised by monitor, which means that restarting may have no effect as in fact monitors were healthy and alert has been generated by rule. This may change in future versions.

DISCLAIMER:

Please evaluate in your test environment first! As expected, this solution is provided AS-IS, with no warranties and confers no rights. Use is subject to the terms specified at Microsoft. Future versions of this tool may be created based on time and requests.

VIDEO: Reset And Recalculate Health State

MSutara — Wed, 09 Jan 2008 21:19:00 GMT

Our TechNet web site and especially its video section went finally live with a training video I did some time ago. That video demonstrates functionality and differences between tasks which could change a health state after being spawned from within “Health Explorer”.

Brief description:

Reset State – In OpsMgr 2007 SP1 this task causes unit monitor state change to healthy regardless of its current state. This means that state change “Healthy (green) -> Healthy (green)” is possible as well.

Recalculate State – In OpsMgr 2007 SP1 this task always succeeds, but really works just for monitor implementing monitor type which defines on-demand detection. In my video I tried to show how one could use educated guess and recognize if there is on-demand detection without inspecting XML and particular monitor configuration by simply looking at the initial state change in “Health Explorer”. On-demand detection is present pretty much every time there is “Not Monitored -> X” and X is not healthy (green) or when X is green and the initial state change context is different than “monitor was initialized for the first time …”

Here is direct link, enjoy and post question to this blog if some answers are needed.

Additional Note:

Implementation of Reset State from OpsMgr2007 RTM is equal to implementation of Recalculate State from OpsMgr2007 SP1.

Maintenance Mode And Server Availability (Another Solution)

MSutara — Sun, 06 Jan 2008 07:50:00 GMT

Maintenance mode expired, monitored server is still down, but there is no alert in the alert view of OpsMgr 2007 console indicating server being offline.

In my previous post I tried to walk through one possible implementation of the workaround to recognize server unavailability. That post also contains information about reasons behind scenario described above so I won’t repeat myself here. I will spend short time to discuss impact of that workaround to RMS and compare with solution I will suggest in this post though. One of the biggest “issues” with previous implementation is potential duplication of alert. Dependency monitor rolls up health state of the instance of its contributing health service and it is likely that alert was raised already. Another “issue” is duplication of health service watcher dependency monitor just to get alert for monitored health service unavailability. I suggested disabling original dependency monitor (Local Health Service Availability), but if that was not done, workflow count is affected and increased by count of health services within enterprise.

This post will try to suggest different solution to avoid alert duplication problem. It still uses dependency monitor and its feature to set the state its unavailable contributing entity uses. Difference is that dependency monitor will use always healthy monitor of contributing entity to guarantee that unhealthy state is a result of unavailability only and is never a result of health state of the contributing entity. Here is step by step set up and snapshot.

· Create a unit monitor targeted to the Health Service class that is always going to be green

· Create a dependency monitor targeted to Health Service Watcher class

· Set this monitor to roll up health from the always healthy unit monitor (created in Step 1)

· Set this monitor MemberUnAvailable configuration to Error

· Configure the alert for this dependency monitor

This will ensure that alert is generated when the Health Service is actually not available even after the computer leaves maintenance mode. One more benefit is it will also alert when the Health Service is not started after computer comes back up.

Unfortunately this implementation still affects workflow count on RMS. There will be an addition of that many workflows as many health service watcher instances your enterprise contains. This means that you need to decide if want to live with increased TCO with possible duplicate alert from previous post, or if your RMS is able to withstand heavier workflow load.

To test and use this implementation, please import attached management pack in your test environment to evaluate if this workaround works for you. It is not sealed and can be further customized if you wish to do so. Test entering MM for all entities as described here (you can use script from this post), shutdown server and wait for alert.

Maintenance Mode And Server Availability

MSutara — Fri, 04 Jan 2008 22:11:00 GMT

Maintenance mode expired, monitored server is still down, but there is no alert in the alert view of OpsMgr 2007 console indicating server being offline. This scenario is happening too often with current implementation of availability monitoring. What our customer face and fear is that if server which undergo maintenance mode each night is not rebooted successfully, there is no indication about its unavailability and this could lead to at least bad user experience if not more …

As described in this knowledge base article, one needs to put computer, health service hosted by maintained computer as well as watcher monitoring this health service remotely into maintenance mode to avoid unexpected unavailability alerts. But “unfortunately”, this is also a direct reason why alert is not generated when maintenance mode terminated.

When instance of the managed entity enters MM, all its monitoring is suspended. When it leaves MM, monitoring is “restarted” with clean slate. As there is not a probe detecting the current availability of instance implemented with current version of OpsMgr 2007, there is no indication about its unavailability. Luckily, there is a runtime component which is always aware of current availability of instances it cares about. This component is dependency monitor. It is always aware of availability status of its contributing instances. This gives us hope and chance to implement alert mechanism which notifies about possible instance unavailability.

Back to the subject of this post, workaround for health service availability. Instance of health service indicates the possibility to monitor discovered instances of other entities. Instance of health service watcher monitors availability of health service instance. There is already a relationship between these two entities and that allows for creation of the dependency monitor. If we would inspect monitor topology for health service watcher, we can see that there is such dependency monitor already.

Unfortunately this dependency monitor doesn’t allow for customization of alert, if one was enabled thru “Generate Alert” override, simply because there is no alert configuration with released dependency monitor.

Such “issues” indicate that workaround should implement its own dependency monitor. This new monitor will be “equal” to existing one, only difference is it defines alert configuration. It also means we could disable original dependency monitor (Local Health Service Availability) because new monitor will provide same monitoring for us.

How this new monitor changes state to error? One possible way is that instance of contributing health service experiences problem and contributes with state error. Another possibility is to use health state “error” in the case contributing instance is not available.

Here is a copy of snapshot from the “Knowledge base” I used with monitor implemented in this workaround. It tries to explain how to troubleshoot its error state and how to recognize (to some extent) that health service is not available.

Causes

The error health state for this dependency monitor could be caused by unavailability of watched health service. Alert is generated each time.

Resolutions

The state change event tab and context of the state change of this monitor needs to be investigated. ICMP ping diagnostic should be executed and its result carries information about availability of computer which is hosting watched health service.

Computer is online when there is no diagnostic output available. Contributing monitor hierarchy should be expanded and will point to problem with local state of watched health service.

You can try the following to diagnose and remediate issue when computer is not online: (when diagnostic output exists)

· Perform a trace route to check if there is packet loss to a router or switch to the target Health Service. You can use the tracert.exe utility in Windows. If you have Window XP or Windows Server 2003 and higher, you can use pathping.exe

· Physically check the target computer to ensure it is connected or plugged into the network.

· Check with your network administrators if there are any known issues or outages that may be affecting the target Health Service and its parent Management Server.

Please import attached management pack in your test environment to evaluate if this workaround works for you. It is not sealed and can be further customized if you wish to do so. Test entering MM for all entities as described here (you can use script from this post), shutdown server and wait for alert. Steps from knowledge base article should be self explanatory then.

Fix state related data type transform in OpsMgr2007 SP1 RC

MSutara — Sat, 08 Dec 2007 04:20:00 GMT

Operations Manager displays alerts, task results and other data thru different dialogs and views. Such are usually results of the transform applied to XML representation of said data types.

It came to my attention that transform used to generate output for state related data is broken (State data could be displayed as a part of diagnostic output, as the result of some task etc). Problem with transform is that it doesn’t allow display of the state change context when such is present for particular state change. (Remember state change context is usually the XML representation of data type which caused said state change in the first place). Another minor issue is also caused by somewhat ambiguous display of state changes where it is little confusing to recognize states of individual monitors contributing to the final health state of the instance of particular managed entity type.

Attached file provides the fix for said issue. It needs to be executed on Operations Manager database by database owner. It will update stored transforms with fixes necessary for proper display/functionality.

“Usual suspects:”

This fix is for Operations Manager 2007 SP1 RC only as later releases will be fixed and prior ones do not need it.

This posting is also provided "AS IS" with no warranties, and confers no rights. Use of included T-SQL script is subject to the terms specified at Microsoft plus is not supported thru any official CSS channel. If support is important, you need to wait for official release of Operations Manager 2007 SP1 RTM. Due to all these facts I strongly recommend TO TEST in pre-production environment as using this confers no rights in the case of any harm to your monitoring infrastructure. Please consider real carefully all these aspects and possibilities prior using attached file!