Notes on System Center Operations Manager : MP Authoring

OpsMgr 2007 R2 - Alert Storm Recognition (possible rule misconfiguration) for SDK alert

MSutara — Tue, 29 Sep 2009 21:18:47 GMT

I described the mitigation for alert storm in the one of my previous post. This mitigation is included out of the box with current deployment of OpsMgr 2007 R2 after system management packs were imported.

As I failed to remember this post (which talks about how to insert alerts from SDK), I also failed to realize that this alert suppression feature will not work unless a change to your custom management pack is made.

I was reminded by Markus, that I need to provide an update in order for this feature to work properly for SDK inserted data generating too much alerts within short period of the time.

History of SDK inserted alert requirements:

When we designed this SDK feature, we decided that it is best for MP (and connector) author to make a copy of condition detection module System.Connectors.GenericAlertMapper, publisher write action System.Connectors.PublishAlert and final composite module type System.Connectors.GenerateAlertFromSdkEvent into their own management pack.

That was the reason why we did not include such definition(s) inside of Microsoft management packs (well some things had changed since then so it is possible we will make this available as public module from one of our system MP in the future).

Management pack with sample modules can be also downloaded directly by clicking this.

Unfortunately this decision has a negative impact on “alert storm mitigation” feature which was delivered as a part current OpsMgr release. This is because implementation of this feature included minor change in the configuration of the module converting input data type to alert (System.Connectors.GenericAlertMapper).

By seeing your current implementation still working, it is clear that such change was not breaking backward compatibility.

Unfortunately, if configuration is not changed for your modules generating alert for SDK inserted data as well, runtime will not initialize component responsible for alert storm recognition and suspension.

Customization needed:

It is required to change the version for all referenced system management packs to OpsMgr R2. This ensures that MP will not be imported to previous releases of OpsMgr where feature is not implemented and would cause alert generating workflow unload. (This is because of failure during configuration processing while loading a module responsible for alert generation.)

<Version>6.1.7221.0</Version>

It is also required to add ManagementGroupName tag into configuration of the module(s). Highlighted areas in XML representation of module types should be self descriptive both for alert generating module as well as final composite module type.

<ConditionDetectionModuleType ID="System.Connectors.GenericAlertMapper" Accessibility="Internal" Batching="false" Stateful="false" PassThrough="false">
  <Configuration>
    <IncludeSchemaTypes>
      <SchemaType>Health!System.Health.AlertSchema</SchemaType>
    </IncludeSchemaTypes>
    <xsd:element name="Priority">
      <xsd:simpleType>
        <xsd:restriction base="xsd:integer">
          <xsd:minInclusive value="0" />
          <xsd:maxInclusive value="2" />
        </xsd:restriction>
      </xsd:simpleType>
    </xsd:element>
    <xsd:element name="Severity">
      <xsd:simpleType>
        <xsd:restriction base="xsd:integer">
          <xsd:minInclusive value="0" />
          <xsd:maxInclusive value="2" />
        </xsd:restriction>
      </xsd:simpleType>
    </xsd:element>
    <xsd:element name="ManagedEntityId" type="xsd:string" />
    <xsd:element name="AlertName" type="xsd:string" minOccurs="0" maxOccurs="1" />
    <xsd:element name="AlertDescription" type="xsd:string" minOccurs="0" maxOccurs="1" />
    <xsd:element name="AlertOwner" type="xsd:string" minOccurs="0" maxOccurs="1" />
    <xsd:element name="AlertMessageId" type="xsd:string" minOccurs="0" maxOccurs="1" />
    <xsd:element name="AlertParameters" type="System.Health.AlertParameters" minOccurs="0" maxOccurs="1" />
    <xsd:element name="Suppression" type="System.Health.Suppression" minOccurs="0" maxOccurs="1" />
    <xsd:element name="WorkflowId" type="xsd:string" />
    <xsd:element name="Custom1" type="xsd:string" minOccurs="0" maxOccurs="1" />
    <xsd:element name="Custom2" type="xsd:string" minOccurs="0" maxOccurs="1" />
    <xsd:element name="Custom3" type="xsd:string" minOccurs="0" maxOccurs="1" />
    <xsd:element name="Custom4" type="xsd:string" minOccurs="0" maxOccurs="1" />
    <xsd:element name="Custom5" type="xsd:string" minOccurs="0" maxOccurs="1" />
    <xsd:element name="Custom6" type="xsd:string" minOccurs="0" maxOccurs="1" />
    <xsd:element name="Custom7" type="xsd:string" minOccurs="0" maxOccurs="1" />
    <xsd:element name="Custom8" type="xsd:string" minOccurs="0" maxOccurs="1" />
    <xsd:element name="Custom9" type="xsd:string" minOccurs="0" maxOccurs="1" />
    <xsd:element name="Custom10" type="xsd:string" minOccurs="0" maxOccurs="1" />
    <xsd:element name="ManagementGroupName" type="xsd:string" />
  </Configuration>
  <ModuleImplementation Isolation="Any">
    <Native>
      <ClassID>2325018e-eef4-41a3-8c17-db831b85c93b</ClassID>
    </Native>
  </ModuleImplementation>
  <OutputType>Health!System.Health.AlertUpdateData</OutputType>
  <InputTypes>
    <InputType>System!System.BaseData</InputType>
  </InputTypes>
</ConditionDetectionModuleType>
 
<WriteActionModuleType ID="System.Connectors.GenerateAlertFromSdkEvent" Accessibility="Public" Batching="false">
  <Configuration>
    <IncludeSchemaTypes>
      <SchemaType>Health!System.Health.AlertSchema</SchemaType>
    </IncludeSchemaTypes>
    <xsd:element name="Priority">
      <xsd:simpleType>
        <xsd:restriction base="xsd:integer">
          <xsd:minInclusive value="0" />
          <xsd:maxInclusive value="2" />
        </xsd:restriction>
      </xsd:simpleType>
    </xsd:element>
    <xsd:element name="Severity">
      <xsd:simpleType>
        <xsd:restriction base="xsd:integer">
          <xsd:minInclusive value="0" />
          <xsd:maxInclusive value="2" />
        </xsd:restriction>
      </xsd:simpleType>
    </xsd:element>
    <xsd:element name="AlertName" type="xsd:string" minOccurs="0" maxOccurs="1" />
    <xsd:element name="AlertDescription" type="xsd:string" minOccurs="0" maxOccurs="1" />
    <xsd:element name="AlertOwner" type="xsd:string" minOccurs="0" maxOccurs="1" />
    <xsd:element name="AlertMessageId" type="xsd:string" minOccurs="0" maxOccurs="1" />
    <xsd:element name="AlertParameters" type="System.Health.AlertParameters" minOccurs="0" maxOccurs="1" />
    <xsd:element name="Suppression" type="System.Health.Suppression" minOccurs="0" maxOccurs="1" />
    <xsd:element name="Custom1" type="xsd:string" minOccurs="0" maxOccurs="1" />
    <xsd:element name="Custom2" type="xsd:string" minOccurs="0" maxOccurs="1" />
    <xsd:element name="Custom3" type="xsd:string" minOccurs="0" maxOccurs="1" />
    <xsd:element name="Custom4" type="xsd:string" minOccurs="0" maxOccurs="1" />
    <xsd:element name="Custom5" type="xsd:string" minOccurs="0" maxOccurs="1" />
    <xsd:element name="Custom6" type="xsd:string" minOccurs="0" maxOccurs="1" />
    <xsd:element name="Custom7" type="xsd:string" minOccurs="0" maxOccurs="1" />
    <xsd:element name="Custom8" type="xsd:string" minOccurs="0" maxOccurs="1" />
    <xsd:element name="Custom9" type="xsd:string" minOccurs="0" maxOccurs="1" />
    <xsd:element name="Custom10" type="xsd:string" minOccurs="0" maxOccurs="1" />
  </Configuration>
  <ModuleImplementation Isolation="Any">
    <Composite>
      <MemberModules>
        <ConditionDetection ID="Mapper" TypeID="System.Connectors.GenericAlertMapper">
          <Priority>$Config/Priority$</Priority>
          <Severity>$Config/Severity$</Severity>
          <ManagedEntityId>$Data/ManagedEntityId$</ManagedEntityId>
          <AlertName>$Config/AlertName$</AlertName>
          <AlertDescription>$Config/AlertDescription$</AlertDescription>
          <AlertOwner>$Config/AlertOwner$</AlertOwner>
          <AlertMessageId>$Config/AlertMessageId$</AlertMessageId>
          <AlertParameters>$Config/AlertParameters$</AlertParameters>
          <Suppression>$Config/Suppression$</Suppression>
          <WorkflowId>$MPElement$</WorkflowId>
          <Custom1>$Config/Custom1$</Custom1>
          <Custom2>$Config/Custom2$</Custom2>
          <Custom3>$Config/Custom3$</Custom3>
          <Custom4>$Config/Custom4$</Custom4>
          <Custom5>$Config/Custom5$</Custom5>
          <Custom6>$Config/Custom6$</Custom6>
          <Custom7>$Config/Custom7$</Custom7>
          <Custom8>$Config/Custom8$</Custom8>
          <Custom9>$Config/Custom9$</Custom9>
          <Custom10>$Config/Custom10$</Custom10>
          <ManagementGroupName>$Target/ManagementGroup/Name$</ManagementGroupName>
        </ConditionDetection>
        <WriteAction ID="WA" TypeID="System.Connectors.LibraryPublishAlert" />
      </MemberModules>
      <Composition>
        <Node ID="WA">
          <Node ID="Mapper" />
        </Node>
      </Composition>
    </Composite>
  </ModuleImplementation>
  <InputType>System!System.BaseData</InputType>
</WriteActionModuleType>

Enjoy alert storm mitigation (customization is possible as described in original post) after importing newly customized management pack!

What is new: OpsMgr 2007 R2 - How to reset monitor state with recovery?

MSutara — Mon, 20 Apr 2009 20:28:00 GMT

Cameron had a nice example of using new R2 feature process monitoring in real life, but that raised a question about feature he wanted to use. Following is report of his issue and how we can help him to address his challenges using already existing feature of OpsMgr 2007.

Scenario: Monitoring a system with a process monitor. Define a recovery to reboot the system if it’s not running the process required. Run this recovery automatically on critical state.

Problem: In OpsMgr prior to R2 when a Recovery was created it had an option to “Reset monitor” which would put it to a healthy state. In R2, this now says “Recalculate State Monitor”. This is representing a challenge as described below:

Challenge: Recalculating the state may keep the monitor in a critical state until the system has been rebooted successfully and is in fact running the process. If the process does not start correctly after reboot, it gets stuck in the critical state and the recovery will not run again. With a Reset of this monitor to a Healthy state, this would work properly, but without that option available I am not seeing an effective way to make this work.

Workaround: Recovery is no different than other workflows loaded by OpsMgr and is rather similar to task. It consists of modules that are chained together and should provide some corrective action in order for monitor to fix its state. For that reason, first module could be the module which resets state of the monitor.

Following is module that could be used with recovery directly. It will reset the state of the monitor specified in configuration.

<xsd:element minOccurs="1" name="MonitorId" type="xsd:string" />

</Configuration>

</OverrideableParameters>

<ManagementGroupId>$Target/ManagementGroup/Id$</ManagementGroupId>

<ManagedEntityId>$Target/Id$</ManagedEntityId>

<MonitorId>$Config/MonitorId$</MonitorId>

</WriteAction>

</MemberModules>

</Composition>

</Composite>

</ModuleImplementation>

<OutputType>System!System.BaseData</OutputType>

<InputType>System!System.BaseData</InputType>

</WriteActionModuleType>

Next is another module which can be used as well. It resets the state of the monitor first and then executes command.

<SchemaType>System!System.CommandExecuterSchema</SchemaType>

</IncludeSchemaTypes>

<xsd:element minOccurs="1" name="ApplicationName" type="xsd:string" />

<xsd:element minOccurs="1" name="WorkingDirectory" type="xsd:string" />

<xsd:element minOccurs="1" name="CommandLine" type="xsd:string" />

<xsd:element minOccurs="1" name="TimeoutSeconds" type="xsd:integer" />

<xsd:element minOccurs="1" name="RequireOutput" type="xsd:boolean" />

<xsd:element minOccurs="1" name="MonitorId" type="xsd:string" />

</Configuration>

<ApplicationName>$Config/ApplicationName$</ApplicationName>

<WorkingDirectory>$Config/WorkingDirectory$</WorkingDirectory>

<CommandLine>$Config/CommandLine$</CommandLine>

<TimeoutSeconds>$Config/TimeoutSeconds$</TimeoutSeconds>

<RequireOutput>$Config/RequireOutput$</RequireOutput>

</WriteAction>

<MonitorId>$Config/MonitorId$</MonitorId>

</WriteAction>

</MemberModules>

</Node>

</Composition>

</Composite>

</ModuleImplementation>

<OutputType>System!System.BaseData</OutputType>

<InputType>System!System.BaseData</InputType>

</WriteActionModuleType>

Sealed MP with both modules is attached to this post.

Sample: Attached is also example providing use of modules with simple event based monitor. Monitor targets instance of “Root Management Server” and that is a reason why management pack also defines a view for state of this entity. When you choose to display “Health explorer”, you should be easily able to locate sample monitor.

One of the recoveries present in attached MP runs automatically with WARNING state. Highlighted is MPElement replacement representing monitor you want to reset. (It should be same as value of the attribute Monitor! Also, please observe that using just a reset module causes its output to be displayed in “Context” tab as well as two state changes will appear to have “same” time of change in Health Explorer.

<Category>Maintenance</Category>

<MonitorId>$MPElement[Name="Microsoft.SystemCenter.Community.Monitors.RecoverySample.EventBasedMonitor"]$</MonitorId>

</WriteAction>

</Recovery>

DISCLAIMER:

Please evaluate in your test environment first! As expected, this solution is provided AS-IS, with no warranties and confers no rights. Use is subject to the terms specified at Microsoft.

How to: Reset monitor when closing alert?

MSutara — Tue, 03 Feb 2009 06:59:00 GMT

Not something I recommend, one would say I almost regret we did not prohibit closing alerts generated by monitor (especially when auto-resolve feature was used). But recently I learned about some ticketing systems closing alerts where it is unclear if issue was corrected, so I see some necessity to automate the action of resetting monitor health state to re-generate monitor state change when issue still present after ticket was closed.

Problem Description:

Again, as said, there may be legitimate situations where a customer needs to reset monitor health once an alert generated by its state change has been resolved. Such scenario would include automated ticketing systems resolving alerts without providing enough evidence that issue was indeed addressed, situation where operator resolves batch of alert without investigating their root cause (after network outage) and/or by mistake.

Recently I saw this type of request from multiple sources independently of each other so I decided to provide what I believe may be the only solution on how to achieve this functionality – OpsMgr connector.

Analyzing proposal:

OpsMgr connector is nice feature allowing subscribing to alert changes happening for members of specified group. It also allows reacting on such change, in our case by locating monitor associated with alert and requesting its state reset thru SDK call.

Note:
I will not discuss connector internals (registration, used subscription …) but will provide source code for possible reverse engineering of my implementation.

Solution:

Attached, you can find source code for my solution as well as binary you should copy into your RMS product folder. You need to initialize connector when you start it. Such action will import MP with group definition (if MP was not imported already), it creates connector and its subscription (again if such actions are necessary) and starts worker thread to receive monitor raised alerts.

You should see connector created (in Administration section of operations console) after successful initialization.

Currently connector uses group which is populated with instances of computer. This can be adjusted (steps described later) and you should be able to see all members after group calculation rule finishes (in Authoring section of the operations console).

Bellow is an example of alert raised by test event based monitor. When this alert is resolved, state of the monitor resets.

Customization:

As mentioned, this connector will respond to all alerts generated for any monitors which belong to the instance of Windows Computer. It is rather simple to customize the managed entity type you want to use though.

First export connector management pack:

Then edit management pack in XML editor of your choice. You need to change type used with relationship as well as group population rule:

After changes are saved and you imported your management pack, please restart connector application (initialization button will not overwrite changes to MP, but remove button will delete MP from OpsMgr when removing connector from your environment). You can see in the source code that worker thread starts in 3 minutes (to give group calculation time to populate group) and subscription uses 1 minute polling interval to retrieve all alerts as per subscription definition.

One more word of caution to be said is that connector like this may not be fully scalable in big environments and additional work could be needed. This post can serve as nice example and base stone for such more advanced application though.

DISCLAIMER:

Please evaluate in your test environment first! As expected, this solution is provided AS-IS, with no warranties and confers no rights. Use is subject to the terms specified at Microsoft. Future versions of this tool may be created based on time and requests.

Computer group and dependency health rollup in OpsMgr 2007 R2 Beta release

MSutara — Wed, 26 Nov 2008 23:09:00 GMT

… will be broken when you need to create group while inheriting from Microsoft.SystemCenter.ComputerGroup.

</ClassTypes>

This was caused by disabling all dependency monitors defined for abstract managed entity type representing computer group while trying to minimize traffic from agent to server.

<Category>AvailabilityHealth</Category>

<Algorithm>WorstOf</Algorithm>

<MemberUnAvailable>Error</MemberUnAvailable>

</DependencyMonitor>

<Category>ConfigurationHealth</Category>

<Algorithm>WorstOf</Algorithm>

<MemberUnAvailable>Error</MemberUnAvailable>

</DependencyMonitor>

<Category>PerformanceCollection</Category>

<Algorithm>WorstOf</Algorithm>

<MemberUnAvailable>Error</MemberUnAvailable>

</DependencyMonitor>

<Category>SecurityHealth</Category>

<Algorithm>WorstOf</Algorithm>

<MemberUnAvailable>Error</MemberUnAvailable>

</DependencyMonitor>

We enabled all instances of computer groups we needed, but unfortunately you must do the same if you want to see your rollup. Following is a sample of MonitoringPropertyOverride enabling availability section rollup for TestComputerGroup defined above. (you will need all necessary MP references in your override management pack)

</MonitorPropertyOverride>

I’m not sure if this problem is described in release notes, but my post should offer quick insight and easy “fix” solution (same can be actually achieved thru UI when you enable particular dependency monitor for your group)

Where is thy instance monitored and how that affects dependency monitor state?

MSutara — Fri, 14 Nov 2008 10:16:00 GMT

This area is rather complex and multiple factors weight in while configuration service makes the final decision, but I will try to provide somewhat simplified, rule of thumbs examples/points to help to make an educated guess while troubleshooting some state issues. I will start with dependency monitor.

Dependency monitor

Dependency monitor is used to roll up health of other objects (contributors) across a relationship and changing the state of target instance according to used algorithm. (Following is an article describing how to create this monitor using authoring section of operations console MSDN).

Roll up is an important term here! You could “visualize” our monitoring topology as some kind of root system with RMS at the top, followed by layer of management servers and with agents at the bottom. Such perspective then evokes that health state only rolls up from agent to server. Such observation is very correct, BUT one must add that rollup works within self as well.

When rollup works

In other words, dependency monitor is able to rollup state when contributor resides on (is monitored by) the agent while target instance is monitored by server. Monitor is also able to roll up state when both contributors and target are monitored by same health service (regardless of it being agent or server)

When rollup doesn’t work

Above statement also means that rollup won’t work when target is monitored by agent while contributor resides on sibling agent or on server! (Just imagine “root system” again please).

NOTE: This statement is correct at least for all up to date releases of Operations Manager including OpsMgr2007 R2 future release.

Where is this instance monitored?

1000000$ question right there! If you are an operator, you should not need to worry, unfortunately once you become MP author, answering this question may happen to “explain” some of your issues. As I tried to hint, there is many factors affecting final place where instance is monitored, but there are some hints that provide educated guess.

1. Hosting: Due to its nature, hosting is relationship that binds life time of the host to life time of the hosted instance. For that reason, same health service monitoring host instance will also monitor hosted instance. To put it into perspective and provide real life sample, everything hosted by the instance of the computer will be monitored by health service running on that computer.

2. Group and singleton: instances of such managed entity type are almost always monitored by Root Management Server. This is true for all extensions of original base class. Real life example could use “All managed computers group” as something that is monitored by RMS.

3. Logical entity and some others: (as well as all types extending those types like System.ApplicationComponent and Microsoft.Windows.ApplicationComponent) are possibly monitored by RMS. Instances of those ME types are “tricky” as long as they do not bring in “Hosted=’true’” attribute. What I mean by this is the fact that if agent discovers instance of this ME type, proxying must be enabled for discovery to succeed and insert the instance into operational DB (not very known fact, but fact it is!). If proxy was enabled and no additional work done, then instance is always monitored by RMS.

NOTE: I will discuss how to make discovering health service the one to perform monitoring in some future article!

Real life example of how to use this knowledge

Kevin Holman was asked by customer to provide state view which displays state of “Computer Down” monitor. (This was really not the first time I saw similar request, and vast majority of the times, same solution was tried and failed – well, it is an obvious one especially when you are not aware of the dependency rollup limitation – and original plan probably was that you as an author shouldn’t have to be aware of that limitation, but it is here now and needs to be considered.)

Solution which failed

· Created a custom class derived from local application (Microsoft.Windows.LocalApplication) and discover its instance based on registry while targeting instance of the computer (Microsoft.Windows.Computer).

· Create containment relationship between this class and health service watcher (requested “Computer Unreachable” monitor is part of HSW health state) followed by the creation of the dependency monitor to rollup health so health of the custom instance changes with the health changes on requested monitor.

· Create a state view to visualize particular monitor’s state (thru the health of the instance of custom class)

So why would this solution fail? Here is the recap.

· Custom class extends ME type which is already hosted. It is hosted by the computer (that is why discovery required targeting instance of the computer – to provide PrincipalName value for one of the keys). Based on this post, it means that instance of the custom class will be monitored by the same health service as one running on the computer.

· Relationship is starting at RMS. Instances of health service watcher are monitored by RMS. ME type used for HSW is hosted by group and as I said above, instance of group is almost always monitored by RMS.

· Place where contributor is monitored breaks dependency monitor - remember its limitation about rolling up and not down!

· MP verification is currently UNABLE TO WARN you about this problem, its implementation has no knowledge about instance space, it only understands type space of the installation.

Here are snapshots I received from Kevin. They display the problem as observed in health explorer.

How should this be solved?

By evaluation of the failure for original proposal, we can see that instances of health service watcher are monitored by RMS. This article also suggests that dependency monitor is able to roll up the health state when both contributor and target are monitored by same health service. So in fact, we are able to “reuse” the idea from original approach. Only change required is that we simply need to assure that instances of the custom class are monitored by RMS as well. There is a couple of possible ways how to do that, I chose following:

Class: extending ApplicationComponent. Such managed entity is not hosted so is easily discovered by RMS (while targeting instance of the Health Service Watcher).

</ClassType>

Relationship: is containment with HSW as target. This is base stone for dependency monitor.

<Source>Microsoft.SystemCenter.Community.InstanceSpace.ComputerUnreachable.Holder</Source>

<Target>SCLibrary!Microsoft.SystemCenter.HealthServiceWatcher</Target>

</RelationshipType>

Dependency monitor: uses requested monitor as contributor. In this case it is “Computer No Reachable” monitor belonging to HealthServiceWatcher. This monitor also uses state Error when contributing instance is in Maintenance Mode – there is no way to insert custom class into MM when health service watcher class enters maintenance mode, so leveraging this feature of the dependency monitor helps (at least in my opinion).

<Category>StateCollection</Category>

<Algorithm>WorstOf</Algorithm>

<MemberInMaintenance>Error</MemberInMaintenance>

</DependencyMonitor>

Once attached MP is imported, you can observe it picks state from “Computer Down” monitor and presents such in separate view. Below is state view with health explorer.

WARNING: this solution may not scale well in big environment!

Reason for scalability issue lies in the fact of having as many instances of health service watchers as many health services are present in management group (except health service for RMS). Provided solution uses instance of HSW as target for discovery of the custom class, which ultimately means that many workflows are loaded by RMS. There is also additional pressure from dependency monitor (each dependency monitor contributes with another (yet hidden) workflow).

PLEASE consider this an sample, more complex and advanced solution may be needed for real life situation. (Please contact and I will try to help if my bandwidth allows)

DISCLAIMER:

Please evaluate in your test environment first! As expected, this solution is provided AS-IS, with no warranties and confers no rights. Use is subject to the terms specified at Microsoft.

HealthServiceWatcher alerts and connector.

MSutara — Wed, 15 Oct 2008 06:00:00 GMT

Recently I had heard from couple of customers (internal and external) about their trouble with ticketing alerts created from Health Service Watcher. Main complaint was related to the fact that alert is raised by instance of the watcher rather than instance...(read more)

“Net view” as diagnostic used when heartbeat is missing

MSutara — Fri, 25 Jul 2008 01:25:00 GMT

This all started as question about how “Computer Not Reachable” works. It has been asked in newsgroup and I thought I will help and provide some inside. First, I would like to repeat, that in my opinion regarding the fact that computer not reachable detection is based on the fact that heart beat is missing while targeting “Health Service Watcher” is at least questionable and not very fortunate (it is almost like we try to find causality without root causality engine). Due to performance limitations (Main reasons lying in the fact we did not want to end up having as many workflows doing ICM Ping (which is from OpsMgr perspective a blocking operation) as many computers are present within topology, or not having efficient targeting story for such computer not reachable recognition) we now have this “nasty beast” where we execute diagnostic on heartbeat monitor state change which is later supposed to (thru some set of recoveries) make a monitor state change for “Computer Not Reachable”. This all design was not very reliable in RTM and I had to introduce some mitigations and reliability improvements in SP1 (which off course was not too positive because now two alerts are raised instead of one which was raised in RTM, but believe me, you want reliability J).

Regardless of my runt, I decided to show how to add custom diagnostics for Heartbeat monitor and eventually answer a question and provide management pack which will set the state of some custom monitor based on recovery output. This all could be advance talk, please comment and/or ask question thru this blog if something requires further explanation (this I will not discuss on newsgroup, all this provided info is about undocumented functionality which may break in any future release!).

So to adding custom diagnostic is really easy. In this case all we need (monitor and target) is public and accessible outside of management pack where such elements are defined. At the end it is just adding following raw XML into custom management pack (assuming you have all MP references set properly).

<Diagnostic ID="Microsoft.SystemCenter.Community.Diagnostic.NetViewDiagnostic" Comment="In response to heartbeat failure, net view machine" Accessibility="Internal" Enabled="true" Target="SCLibrary!Microsoft.SystemCenter.HealthServiceWatcher" Monitor="SC2007!Microsoft.SystemCenter.HealthService.Heartbeat" ExecuteOnState="Error" Remotable="true" Timeout="300">

<Category>Maintenance</Category>

<ProbeAction ID="Command" TypeID="System!System.CommandExecuterProbe">

<ApplicationName><![CDATA[%windir%\System32\net.exe]]></ApplicationName>

<WorkingDirectory />

<CommandLine>view $Target/Property[Type="SCLibrary!Microsoft.SystemCenter.HealthServiceWatcher"]/HealthServiceName$</CommandLine>

<TimeoutSeconds>30</TimeoutSeconds>

<RequireOutput>true</RequireOutput>

</ProbeAction>

</Diagnostic>

Now acting based on the result is where real fun starts! First I need to mention that “Computer Not Reachable” is defined as internal, so is not accessible and new monitor must be defined rather then disable original thru override included with management pack. I will not provide much information about why it is “aggregate” monitor, the only thing I will say is that in OpsMgr, aggregate monitor is only monitor without a workflow and runtime is magically making all the state changes. In our case we are setting the state thru recovery, so we do not want to “WASTE” a workflow (and there would be as many of it as many computers within enterprise are monitored) if we never expect that workflow to set the state anyway. Also, there are some “magic” modules I WILL NOT DISCUSS (maybe ever, but we will see in next release), where those modules will set the state of the monitor. (There might be some of you willing to do some reverse engineering and you may get an idea how set state critical when result of recovery provides info about command “net view” failing, and how we set state when command succeeded though.) So here is the recap thru screenshots:

1. After import, monitor state is NEVER set until net view command is executed

2. When RMS recognizes that heart beat is missing, we execute “net view” command inside of diagnostic. When net view succeeds, we set state of monitor to “Healthy”, we set to “Critical” otherwise

Attached is management pack that provides this functionality. It may be used AS IS and confers no rights and support. Enjoy!

Customizing monitor from sealed MP ("Device Status Check").

MSutara — Fri, 18 Jul 2008 21:45:00 GMT

I used Device Status Check monitor in my last post when providing realistic sample of how to define alert settings thru overrides. Unfortunately, and mainly because of override value restriction regarding use of parameter replacement, description of such alert was not able to benefit from property values of its target instance (could not use device name, IP Address and others with alert description). This led me to work on this post, although it is rather possible some other guys already did similar work …

So to help Tim, I decided to customize that monitor without forcing customization thru overrides. I had used regular text editor to create everything in raw XML, but here is how I would approach this problem if had no access to “internal” unsealed version of shipped MP and would need to use our authoring tools.

· Create new empty management pack in our authoring console. I only need to do this if I do not have MP with customization and/or overrides for that particular managed entity type already.

· Next I need to find management pack holding the definition of monitor I want to customize. Then I need to export this management pack. (UI won’t allow exporting sealed MP, but we have Powershell!)

$monitor = Get-Monitor -Criteria "DisplayName = 'Device Status Check'"

$monitor.GetManagementPack() | export-managementpack –path c:\

· Open this MP in regular text editor (it is stored in path used with commandlet) and copy unit monitor I want to customize into my custom MP file. This all work raises and falls on accessibility of managed entity used as target as well as accessibility of monitor type used with unit monitor implementation. Target of your monitor MUST be public (to be accessible outside of MP where it is defined) and same goes for monitor type you are implementing (monitor type is driven by TypeID attribute). Please be aware of referencing. When copying monitor into your custom MP, you need to provide reference to MP where those elements are defined. Please observe how it is done in attached file. Also, ID for this monitor needs to be changed; it is sufficient to simply add string Custom to the end of original ID. Also, this monitor needs to go to Monitoring -> Monitors section, please create those tags when not present.

NOTE: When you do not make this effort, your MP will not pass verification and authoring console will not load your MP for further customization!

It is also very good idea to copy KnowledgeBase article for your new custom monitor (if KB exists for original monitor) into your custom MP. Just remember to adjust ID of this article to match ID you had chosen for your monitor. (Attached MP file should be sufficient sample to follow for this paragraph.)

· Now save this file and re-open in authoring console. Go to section Monitors and provide all display strings to help SDK displaying user friendly strings rather than “ugly” MP element ID strings.

· Make changes to alert settings of your custom monitor. Remember that you can benefit from all replacements here.

· Disable original monitor. This is possible to do in UI thru operations console or you can manually add override thru editing raw XML (again we do not have overrides in shipped authoring console yet). Later is in my opinion much better option allowing deployment of compact solution to your customer with disabling original monitor and providing customized approach in one MP import.

· “MPVerify” your file (you can do this thru Tools section of authoring console), save the file, import into product and test its functionality. Once satisfied with results, MP seal and ship to your customers …

Next is a screenshot that displays alert after customized monitor picked a problem with SNMP. My customized MP is attached to benefit some of you (I hope); just please remember that its use is bound to usual AS-IS clause…

Dynamic alert settings for monitor which never had any (done thru overrides)

MSutara — Thu, 17 Jul 2008 19:00:00 GMT

Tim from QuickenLoans contacted me and wanted verification about ability to use AlertMessage override (discussed in my previous post) with unit monitor for SNMP network device. He was using “Device Status Check” monitor and we can see that initial requirement for override use is met – monitor is PUBLIC.

<Category>PerformanceHealth</Category>

</OperationalStates>

<IsWriteAction>false</IsWriteAction>

<IP>$Target/Property[Type="Microsoft.SystemCenter.NetworkDevice"]/IPAddress$</IP>

<CommunityString>$Target/Property[Type="Microsoft.SystemCenter.NetworkDevice"]/CommunityString$</CommunityString>

<Version>$Target/Property[Type="Microsoft.SystemCenter.NetworkDevice"]/Version$</Version>

</SnmpVarBind>

</SnmpVarBinds>

</Configuration>

</UnitMonitor>

As evident above (and to my surprise), this monitor doesn’t have any alert associated with its state change! Luckily, plenty of overrides which are required to raise alert are accessible thru UI, which leads me to believe that some of you were able to raise alert already. For those who never did, nor needed to do such work, we will use GenerateAlert, AlertOnState, AlertPriority and AlertServerity overrides. I’m going to include those overrides in my Management Pack directly and will not spend much time with them as they should be self-explanatory.

</MonitorPropertyOverride>

</MonitorPropertyOverride>

<Value>Error</Value>

</MonitorPropertyOverride>

<Value>Normal</Value>

</MonitorPropertyOverride>

<Value>MatchMonitorHealth</Value>

</MonitorPropertyOverride>

But as some of you are probably aware, using those overrides only helps alert to be raised, but alert will not have any user friendly description. This is where my previous post comes handy. It provides the guide on how to customize alert description. Our new alert would need description, but that is equal to customizing anyway, so same process applies here as well. Initially we need to define alert description and retrieve its GUID from database after it was imported (described in my previous post).

<Name>Network device is down</Name>

<Description>Network device identified by community string {0} is down.</Description>

</DisplayString>

Override to be used for alert message is then defined:

</MonitorPropertyOverride>

You can also see that in my definition I wanted to use alert parameter, the one holding device identification. WHAT I FAILED TO MENTION in my previous post is that no configuration replacement is allowed in override values. Only replacement that works is dynamic replacement. Here is what this means. $Target and $MPElement replacements are not working with override value, only $Data works because replacement is performed by module and not by runtime! This is important information, sorry to leave it out last time! So I need to use data item triggering monitor state change to retrieve required information thru $Data replacement. So here is override that helps runtime to contain alert parameter within alert data.

<Value>$Data/Context/DataItem/CommunityString$</Value>

</MonitorPropertyOverride>

Attached is MP that delivers on this guide. Due to restriction about parameter replacement use, alert description may not have enough information to identify failing device (although IP Address is present as source). There is an event 11001 in "Operations Manager" log that can be correlated to alert's time creation, where such event carries all neccessary information. (It is almost like that event should be used for alert creation in this particular case, but again, this post was supposed to serve as example of alert customization :))

Also remember that similar steps could be used for different monitors as well, only unfortunate requirement is authoring thru XML directly ...

ENJOY! (but remember that usual AS IS paradigm (described in side frame) applies.).

Monitor from sealed MP generated alert and I do not like its description …

MSutara — Wed, 16 Jul 2008 03:58:00 GMT

… Is there anything I can do? Our usual answer was that not really and you need to create your custom monitor while disabling original one. I would like to take this opportunity and rephrase that answer with this post. First, couple of requirements must be met, before I can say, that there is a way to customize your alert.

1. Monitor defined inside of sealed management pack MUST be accessible outside of such MP. In other words this means that monitor MUST be public. This requirement might be already met for most of the monitors, especially when diagnostics, recovery are defined outside or when using this monitor as contributor for some random dependency monitor residing inside of the different management pack.

2. You should be familiar with authoring management pack thru direct XML editing. I know that sounds highly user unfriendly, but unfortunately, current version of Authoring Console doesn’t support overrides creation (this I heard will change in next release)

What else do we need to know? There is an override for monitor which is not FULLY visible thru UI. It is MonitorPropertyOverride and there are multiple properties that are override-able for monitor. Some of them you might be familiar with as they are accessiblethru UI (Generate Alert, Alert On State, Alert Priority, Alert Severity …).

There are multiple properties that are not displayed but override-able too. Those that I will explain here are AlertMessage and AlertParameter1 … AlertParameter10. In my guide I will use unsealed management pack. Only difference will be the fact, that my monitor and required overrides are defined within same MP. In your custom solution, you will likely follow best practice of having overrides in unsealed MP different than “default user”, probably in unsealed MP where overrides for similar ME type are stored.

Please REMOVE Overrides section from attached MP to repro next steps.

1. Let’s assume I have simple event monitor. This monitor picks event 111 from Application log and generates state change to “Warning”. Let’s also assume that I will have alert generated when state of the monitor is at least “Warning” and that I will match alert severity with monitor health. (None of this should be too strange as is already something you are required to specify when creating new unit monitor).

2. After writing event 111 into Application log, state change is recorded and alert is raised as showed in the next picture.

3. This was all possible because runtime simply followed what was defined inside of AlertSettings for my sample the monitor. Here is XML fragment of that section.

<AlertOnState>Warning</AlertOnState>

<AlertPriority>Normal</AlertPriority>

<AlertSeverity>MatchMonitorHealth</AlertSeverity>

<AlertParameter1>$Target/Property[Type="Windows!Microsoft.Windows.Computer"]/PrincipalName$</AlertParameter1>

<AlertParameter2>$Data/Context/DataItem/EventDisplayNumber$</AlertParameter2>

</AlertParameters>

</AlertSettings>

4. Let’s look closer what particular parts mean. It is rather simply, really. Attribute AlertMessage says that there is StringResource with ID equal to Microsoft.SystemCenter.Community.SimpleEventBasedMonitor_AlertMessageID. It also says (somewhat indirectly, but clearly) that such MP element is defined within same management pack (reason is that it is not preceded by MPReferenceAliasName!). If you inspect attached MP, you will see that following is present.

5. As every resource, there will be DisplayString associated with that MP element. It is defined as follows inside of my sample MP.

<Name>Alert raised for Demo Event Based Monitor</Name>

<Description>This is original alert description. Provides info that target is {0} and that this alert was raised for event ID {1}</Description>

</DisplayString>

6. We can observer that Description part of this display string is what is displayed by UI after SDK performed string replacement. It might be obvious what was the replacement using. Yes, it was using AlertParameters section from alert settings definition.

How do I customize my alert description?

1. New AlertMessage needs to be created. Its display strings must use correct language used with locale of server installation. AlertMessage and its data is what I would call “static” SDK data. It can be part of “any” management pack. It is only imported into operational database and is used by SDK when alert is retrieved. One MUST be aware of parameter replacement, especially if alert parameter values or order are to be changed. In the section above I showed original AlertMessage and its display strings, here is value that I want to use next time alert is generated:

<Name>Alert raised for Demo Event Based Monitor with overriden alert description</Name>

<Description>This is OVERRIDEN alert description. It shows that I'm able to override AlertMessageID (which is ultimately how SDK retrieves description) as well as alert parameters. So new description provides info that event description is {0}.</Description>

</DisplayString>

2. Now having this data in DB is just a first step. We need to help runtime to understand the need to use it next time alert for monitor is generated. This is where overrides come to place. The initial override which is needed is override for AlertMessage. Runtime must use GUID representing this new description when creating alert data. To get such GUID, querying table StringResource is required after new alert message was imported into DB.

Once GUID is retrieved from DB, it can be used. As you can see below, override is using my demo monitor, it is used with context of original monitor target and its value is holding GUID representation of alert message I want to use.

</MonitorPropertyOverride>

Please observe, when compared with original alert description, that I changed what my alert parameter (which is to be replaced) is. So once again, runtime needs help to recognize what is/are parameters to be used with next generated alert. This is done by AlertParameter override. Indexing of alert parameters is 1-10, while indexing of replacement strings is {x} where x is 0-9. Please be aware of that, otherwise you have unexpected result! In my example, override is again using my demo monitor in proper context (original target, but grouping and all other nice things related to overrides are allowed, but out of scope of this post). Its value is replacement I want to use when retrieving value from input data type, but it can be any string including static constant string really.

<Value>$Data/Context/DataItem/EventDescription$</Value>

</MonitorPropertyOverride>

3. If successfully followed this guide, next alert generated by monitor will use customized alert description like visible in next picture. Again, it is important to understand that this customization will not be visible until new active alert and it doesn't affect already present or resolved alerts!

RECAP

1. Customization of alert description is possible for alert generated by “public” monitor.

2. This work is not for “faint of heart”. Big effort including DB query is required.

3. Using un-documented overrides (AlertMessage, AlertParameters) as well as understanding SDK replacement is required as main success driver.

I hope this guide can help some of you. I’m not aware any work happening on our side, to simplify this process by allowing our authorization tools to understand and use this. I might be wrong but it is possible these steps will be required when customizing monitor in next release too. I will also try to post more guides like this about some hidden treasures of our product in the future, permitting my time and bandwidth. Enjoy!

HOWTO: Registry Attribute Definition explained

MSutara — Thu, 28 Feb 2008 23:45:00 GMT

Maybe you already needed to use probe module to retrieve data from registry, check the existence of registry key or similar task involving registry when for example discovering your custom managed entity space. In that case you definitely met

<xsd:element name="RegistryAttributeDefinition" minOccurs="1" maxOccurs="unbounded">

<xsd:complexType>

<xsd:sequence>

<xsd:element name="AttributeName" type="xsd:ID" />

<xsd:element name="Path" type="xsd:string" />

<xsd:element name="PathType" type="xsd:integer" />

<xsd:element name="AttributeType" type="xsd:integer" />

</xsd:sequence>

</xsd:complexType>

</xsd:element>

It appears there is no explanation of PathType and AttributeType tags and their mapping to number available anywhere yet, so I will try to help there, as well as post an example of registry use to check key existence and how it should be done for OpsMgr 2007 …

PathType

· 0 – Key. Path is full path to the key.

· 1 – Value of specified registry key. Path includes name of the value.

· 2 – Default value of specified registry key. Path is full path to the key.

AttributeType

· 0 – Boolean: used when checking existence of the key or value.

· 1 – String: converting result value to string if possible.

· 2 – Integer: converting result value to number if possible.

· 3 – Float: converting result value to float.

Example to recognize Vista

Following is XML fragment when retrieving the value CurrentVersion under hive HKLM\Software\Microsoft\Windows NT\CurrentVersion helps to recognize Windows SKU. Similarly I could test existence of the value under some key (or key itself), in which case I need to provide full path to the value and change PathType to 1 (or ) in the case when checking the existence of key).

<AttributeName>WindowsCurrentVersion</AttributeName>

<Path>SOFTWARE\Microsoft\Windows NT\CurrentVersion\CurrentVersion</Path>

</RegistryAttributeDefinition>

</RegistryAttributeDefinitions>

Result registry data type can be used with subsequent module thru $Data/…$ replacement, or direct XPath query when used with expression filter. To recognize Vista, compare with its version number (6.0)

<XPathQuery>Values/WindowsCurrentVersion</XPathQuery>

</ValueExpression>

<Operator>Equal</Operator>

</ValueExpression>

</SimpleExpression>

</Expression>

Example to test existence

Following is XML fragment when testing the existence of the key CustomApplication under hive HKLM\Software.

<AttributeName>CustomAppExists</AttributeName>

<Path>SOFTWARE\CustomApplication</Path>

</RegistryAttributeDefinition>

</RegistryAttributeDefinitions>

To test use following expression

<XPathQuery>Values/CustomAppExists</XPathQuery>

</ValueExpression>

<Operator>Equal</Operator>

</ValueExpression>

</SimpleExpression>

</Expression>

Maintenance Mode And Server Availability (Another Solution)

MSutara — Sun, 06 Jan 2008 07:50:00 GMT

Maintenance mode expired, monitored server is still down, but there is no alert in the alert view of OpsMgr 2007 console indicating server being offline.

In my previous post I tried to walk through one possible implementation of the workaround to recognize server unavailability. That post also contains information about reasons behind scenario described above so I won’t repeat myself here. I will spend short time to discuss impact of that workaround to RMS and compare with solution I will suggest in this post though. One of the biggest “issues” with previous implementation is potential duplication of alert. Dependency monitor rolls up health state of the instance of its contributing health service and it is likely that alert was raised already. Another “issue” is duplication of health service watcher dependency monitor just to get alert for monitored health service unavailability. I suggested disabling original dependency monitor (Local Health Service Availability), but if that was not done, workflow count is affected and increased by count of health services within enterprise.

This post will try to suggest different solution to avoid alert duplication problem. It still uses dependency monitor and its feature to set the state its unavailable contributing entity uses. Difference is that dependency monitor will use always healthy monitor of contributing entity to guarantee that unhealthy state is a result of unavailability only and is never a result of health state of the contributing entity. Here is step by step set up and snapshot.

· Create a unit monitor targeted to the Health Service class that is always going to be green

· Create a dependency monitor targeted to Health Service Watcher class

· Set this monitor to roll up health from the always healthy unit monitor (created in Step 1)

· Set this monitor MemberUnAvailable configuration to Error

· Configure the alert for this dependency monitor

This will ensure that alert is generated when the Health Service is actually not available even after the computer leaves maintenance mode. One more benefit is it will also alert when the Health Service is not started after computer comes back up.

Unfortunately this implementation still affects workflow count on RMS. There will be an addition of that many workflows as many health service watcher instances your enterprise contains. This means that you need to decide if want to live with increased TCO with possible duplicate alert from previous post, or if your RMS is able to withstand heavier workflow load.

To test and use this implementation, please import attached management pack in your test environment to evaluate if this workaround works for you. It is not sealed and can be further customized if you wish to do so. Test entering MM for all entities as described here (you can use script from this post), shutdown server and wait for alert.

Maintenance Mode And Server Availability

MSutara — Fri, 04 Jan 2008 22:11:00 GMT

Maintenance mode expired, monitored server is still down, but there is no alert in the alert view of OpsMgr 2007 console indicating server being offline. This scenario is happening too often with current implementation of availability monitoring. What our customer face and fear is that if server which undergo maintenance mode each night is not rebooted successfully, there is no indication about its unavailability and this could lead to at least bad user experience if not more …

As described in this knowledge base article, one needs to put computer, health service hosted by maintained computer as well as watcher monitoring this health service remotely into maintenance mode to avoid unexpected unavailability alerts. But “unfortunately”, this is also a direct reason why alert is not generated when maintenance mode terminated.

When instance of the managed entity enters MM, all its monitoring is suspended. When it leaves MM, monitoring is “restarted” with clean slate. As there is not a probe detecting the current availability of instance implemented with current version of OpsMgr 2007, there is no indication about its unavailability. Luckily, there is a runtime component which is always aware of current availability of instances it cares about. This component is dependency monitor. It is always aware of availability status of its contributing instances. This gives us hope and chance to implement alert mechanism which notifies about possible instance unavailability.

Back to the subject of this post, workaround for health service availability. Instance of health service indicates the possibility to monitor discovered instances of other entities. Instance of health service watcher monitors availability of health service instance. There is already a relationship between these two entities and that allows for creation of the dependency monitor. If we would inspect monitor topology for health service watcher, we can see that there is such dependency monitor already.

Unfortunately this dependency monitor doesn’t allow for customization of alert, if one was enabled thru “Generate Alert” override, simply because there is no alert configuration with released dependency monitor.

Such “issues” indicate that workaround should implement its own dependency monitor. This new monitor will be “equal” to existing one, only difference is it defines alert configuration. It also means we could disable original dependency monitor (Local Health Service Availability) because new monitor will provide same monitoring for us.

How this new monitor changes state to error? One possible way is that instance of contributing health service experiences problem and contributes with state error. Another possibility is to use health state “error” in the case contributing instance is not available.

Here is a copy of snapshot from the “Knowledge base” I used with monitor implemented in this workaround. It tries to explain how to troubleshoot its error state and how to recognize (to some extent) that health service is not available.

Causes

The error health state for this dependency monitor could be caused by unavailability of watched health service. Alert is generated each time.

Resolutions

The state change event tab and context of the state change of this monitor needs to be investigated. ICMP ping diagnostic should be executed and its result carries information about availability of computer which is hosting watched health service.

Computer is online when there is no diagnostic output available. Contributing monitor hierarchy should be expanded and will point to problem with local state of watched health service.

You can try the following to diagnose and remediate issue when computer is not online: (when diagnostic output exists)

· Perform a trace route to check if there is packet loss to a router or switch to the target Health Service. You can use the tracert.exe utility in Windows. If you have Window XP or Windows Server 2003 and higher, you can use pathping.exe

· Physically check the target computer to ensure it is connected or plugged into the network.

· Check with your network administrators if there are any known issues or outages that may be affecting the target Health Service and its parent Management Server.

Please import attached management pack in your test environment to evaluate if this workaround works for you. It is not sealed and can be further customized if you wish to do so. Test entering MM for all entities as described here (you can use script from this post), shutdown server and wait for alert. Steps from knowledge base article should be self explanatory then.

Alert Severity and Priority use with override

MSutara — Mon, 17 Dec 2007 22:20:00 GMT

Some of you may know already that with Operations Manager SP1 RC we allow for overriding alert severity and priority. What is little unfortunate is inconsistency between creating alert generating rule and overriding especially as first uses string describing severity or priority used, while later simply expects integer value. Mapping between string and integer is what this post will try to explain.

Mapping:

Alert Severity – Its corresponding integer value

Critical – 2
Warning - 1
Information – 0

Alert Priority – Its corresponding integer value

High – 2
Medium – 1
Low – 0

Let’s assume we have following alert with severity CRITICAL:

Let’s override severity. As we can see effective value used for severity is 2 and priority is 1.

We can now override severity. Let’s pick incorrect value outside of allowed range to demonstrate the ability of runtime to adjust and generate corrected alert.

After such override was applied and configuration reloaded new values are used when alert should be raised. Following is sample of such alert.

We can see that new alert is raised with severity INFORMATION. Value of information as per mapping above is 0. So runtime had to adjust and used closest value to -1 which we used while overriding severity. In order to help with recognition of incorrect value used for severity or priority override, Operations Manager 2007 SP1 introduced new events (5404, 5405) to inform the user when runtime was unable to recognize used value and had to adjust to closest allowed one. There is also alert raised to allow OpsMgr2007 operator to recognize this problem without inspecting event log.

Sample alert:

Sample event (this raised alert above):

Event Type:             Warning
Event Source:          HealthService
Event Category:       Health Service
Event ID:                 5405
Date:                      12/17/2007
Time:                      11:16:25 AM
User:                      N/A
Computer:               CUPIDDP13D

Description:

Invalid value for alert severity used with configuration of the rule. It was outside of allowed range and had to be adjusted to closest valid value.

Alert: F2732EEA-A871-27EA-1639-143269DE3CEF
Workflow: MomUIGeneratedRule34de90f776f541db8ccb625bf338af58
Instance: OpsMgr Health Service
Instance ID: {4140A3FC-F165-F0BC-E036-4B2FFB93C622}
Management Group: marius

Used severity value: -1

How to troubleshoot:

There are a couple of options. Operator may remember the alert for which he used value “-1” in override and he needs to go back and use correct value from range as discussed in this post. In the case override was done long ago, or by someone, following steps could help to locate what the name and target for particular alert are and use that information to fix alert override thru authoring section of UI.

In the case of sample event used above one should run:

1. Open Windows Power Shell

2. Get and store monitoring object

$mo = Get-MonitoringObject –Id “4140A3FC-F165-F0BC-E036-4B2FFB93C622”

3. Get an alert rule to recognize where to change

$rule = Get-Alert -MonitoringObject $mo -Criteria "Id='f0ccb13e-22f1-47de-ba98-2c54975089dc'"

4. Retrieve name and target to adjust value for particular overridden alert property later

$rule.Name

$rule.MonitoringObjectFullName

Parameter replacements and its use with alert

MSutara — Mon, 10 Dec 2007 23:03:00 GMT

Many of you probably know already, that alert’s description can use parameter replacements to reference the value of any of the properties of the data type causing the alert to be raised in the first place. I tried to comment on some most commonly used data types and its XML representation in order to help with writing replacement XPath query, which then yields such value as a result of its execution (see older postings about parameter replacements and data item XML).

Unfortunately what I forgot to stress out is that alert description is the only place where such replacement works. Recently I saw couple of requests to help with a replacement inside of alert name (to customer the use of replacement seemed as a good way to create more unique alert names especially when there could be a multiple triggers raising such alert and proper suppression had been used in order to "preserve" the alert for each one of them).

Here is a sample of incorrect alert configuration causing replacement string being treated as constant alert name:

Here is a sample of above alert configuration inside of active alert:

Conclusion:

For alert, data replacement works only inside of alert description with release of Operations Manager 2007 and Operations Manager 2007 SP1.