Notes on System Center Operations Manager : HOWTO

OpsMgr 2007 R2 - Alert Storm Recognition (possible rule misconfiguration) for SDK alert

MSutara — Tue, 29 Sep 2009 21:18:47 GMT

I described the mitigation for alert storm in the one of my previous post. This mitigation is included out of the box with current deployment of OpsMgr 2007 R2 after system management packs were imported.

As I failed to remember this post (which talks about how to insert alerts from SDK), I also failed to realize that this alert suppression feature will not work unless a change to your custom management pack is made.

I was reminded by Markus, that I need to provide an update in order for this feature to work properly for SDK inserted data generating too much alerts within short period of the time.

History of SDK inserted alert requirements:

When we designed this SDK feature, we decided that it is best for MP (and connector) author to make a copy of condition detection module System.Connectors.GenericAlertMapper, publisher write action System.Connectors.PublishAlert and final composite module type System.Connectors.GenerateAlertFromSdkEvent into their own management pack.

That was the reason why we did not include such definition(s) inside of Microsoft management packs (well some things had changed since then so it is possible we will make this available as public module from one of our system MP in the future).

Management pack with sample modules can be also downloaded directly by clicking this.

Unfortunately this decision has a negative impact on “alert storm mitigation” feature which was delivered as a part current OpsMgr release. This is because implementation of this feature included minor change in the configuration of the module converting input data type to alert (System.Connectors.GenericAlertMapper).

By seeing your current implementation still working, it is clear that such change was not breaking backward compatibility.

Unfortunately, if configuration is not changed for your modules generating alert for SDK inserted data as well, runtime will not initialize component responsible for alert storm recognition and suspension.

Customization needed:

It is required to change the version for all referenced system management packs to OpsMgr R2. This ensures that MP will not be imported to previous releases of OpsMgr where feature is not implemented and would cause alert generating workflow unload. (This is because of failure during configuration processing while loading a module responsible for alert generation.)

<Version>6.1.7221.0</Version>

It is also required to add ManagementGroupName tag into configuration of the module(s). Highlighted areas in XML representation of module types should be self descriptive both for alert generating module as well as final composite module type.

<ConditionDetectionModuleType ID="System.Connectors.GenericAlertMapper" Accessibility="Internal" Batching="false" Stateful="false" PassThrough="false">
  <Configuration>
    <IncludeSchemaTypes>
      <SchemaType>Health!System.Health.AlertSchema</SchemaType>
    </IncludeSchemaTypes>
    <xsd:element name="Priority">
      <xsd:simpleType>
        <xsd:restriction base="xsd:integer">
          <xsd:minInclusive value="0" />
          <xsd:maxInclusive value="2" />
        </xsd:restriction>
      </xsd:simpleType>
    </xsd:element>
    <xsd:element name="Severity">
      <xsd:simpleType>
        <xsd:restriction base="xsd:integer">
          <xsd:minInclusive value="0" />
          <xsd:maxInclusive value="2" />
        </xsd:restriction>
      </xsd:simpleType>
    </xsd:element>
    <xsd:element name="ManagedEntityId" type="xsd:string" />
    <xsd:element name="AlertName" type="xsd:string" minOccurs="0" maxOccurs="1" />
    <xsd:element name="AlertDescription" type="xsd:string" minOccurs="0" maxOccurs="1" />
    <xsd:element name="AlertOwner" type="xsd:string" minOccurs="0" maxOccurs="1" />
    <xsd:element name="AlertMessageId" type="xsd:string" minOccurs="0" maxOccurs="1" />
    <xsd:element name="AlertParameters" type="System.Health.AlertParameters" minOccurs="0" maxOccurs="1" />
    <xsd:element name="Suppression" type="System.Health.Suppression" minOccurs="0" maxOccurs="1" />
    <xsd:element name="WorkflowId" type="xsd:string" />
    <xsd:element name="Custom1" type="xsd:string" minOccurs="0" maxOccurs="1" />
    <xsd:element name="Custom2" type="xsd:string" minOccurs="0" maxOccurs="1" />
    <xsd:element name="Custom3" type="xsd:string" minOccurs="0" maxOccurs="1" />
    <xsd:element name="Custom4" type="xsd:string" minOccurs="0" maxOccurs="1" />
    <xsd:element name="Custom5" type="xsd:string" minOccurs="0" maxOccurs="1" />
    <xsd:element name="Custom6" type="xsd:string" minOccurs="0" maxOccurs="1" />
    <xsd:element name="Custom7" type="xsd:string" minOccurs="0" maxOccurs="1" />
    <xsd:element name="Custom8" type="xsd:string" minOccurs="0" maxOccurs="1" />
    <xsd:element name="Custom9" type="xsd:string" minOccurs="0" maxOccurs="1" />
    <xsd:element name="Custom10" type="xsd:string" minOccurs="0" maxOccurs="1" />
    <xsd:element name="ManagementGroupName" type="xsd:string" />
  </Configuration>
  <ModuleImplementation Isolation="Any">
    <Native>
      <ClassID>2325018e-eef4-41a3-8c17-db831b85c93b</ClassID>
    </Native>
  </ModuleImplementation>
  <OutputType>Health!System.Health.AlertUpdateData</OutputType>
  <InputTypes>
    <InputType>System!System.BaseData</InputType>
  </InputTypes>
</ConditionDetectionModuleType>
 
<WriteActionModuleType ID="System.Connectors.GenerateAlertFromSdkEvent" Accessibility="Public" Batching="false">
  <Configuration>
    <IncludeSchemaTypes>
      <SchemaType>Health!System.Health.AlertSchema</SchemaType>
    </IncludeSchemaTypes>
    <xsd:element name="Priority">
      <xsd:simpleType>
        <xsd:restriction base="xsd:integer">
          <xsd:minInclusive value="0" />
          <xsd:maxInclusive value="2" />
        </xsd:restriction>
      </xsd:simpleType>
    </xsd:element>
    <xsd:element name="Severity">
      <xsd:simpleType>
        <xsd:restriction base="xsd:integer">
          <xsd:minInclusive value="0" />
          <xsd:maxInclusive value="2" />
        </xsd:restriction>
      </xsd:simpleType>
    </xsd:element>
    <xsd:element name="AlertName" type="xsd:string" minOccurs="0" maxOccurs="1" />
    <xsd:element name="AlertDescription" type="xsd:string" minOccurs="0" maxOccurs="1" />
    <xsd:element name="AlertOwner" type="xsd:string" minOccurs="0" maxOccurs="1" />
    <xsd:element name="AlertMessageId" type="xsd:string" minOccurs="0" maxOccurs="1" />
    <xsd:element name="AlertParameters" type="System.Health.AlertParameters" minOccurs="0" maxOccurs="1" />
    <xsd:element name="Suppression" type="System.Health.Suppression" minOccurs="0" maxOccurs="1" />
    <xsd:element name="Custom1" type="xsd:string" minOccurs="0" maxOccurs="1" />
    <xsd:element name="Custom2" type="xsd:string" minOccurs="0" maxOccurs="1" />
    <xsd:element name="Custom3" type="xsd:string" minOccurs="0" maxOccurs="1" />
    <xsd:element name="Custom4" type="xsd:string" minOccurs="0" maxOccurs="1" />
    <xsd:element name="Custom5" type="xsd:string" minOccurs="0" maxOccurs="1" />
    <xsd:element name="Custom6" type="xsd:string" minOccurs="0" maxOccurs="1" />
    <xsd:element name="Custom7" type="xsd:string" minOccurs="0" maxOccurs="1" />
    <xsd:element name="Custom8" type="xsd:string" minOccurs="0" maxOccurs="1" />
    <xsd:element name="Custom9" type="xsd:string" minOccurs="0" maxOccurs="1" />
    <xsd:element name="Custom10" type="xsd:string" minOccurs="0" maxOccurs="1" />
  </Configuration>
  <ModuleImplementation Isolation="Any">
    <Composite>
      <MemberModules>
        <ConditionDetection ID="Mapper" TypeID="System.Connectors.GenericAlertMapper">
          <Priority>$Config/Priority$</Priority>
          <Severity>$Config/Severity$</Severity>
          <ManagedEntityId>$Data/ManagedEntityId$</ManagedEntityId>
          <AlertName>$Config/AlertName$</AlertName>
          <AlertDescription>$Config/AlertDescription$</AlertDescription>
          <AlertOwner>$Config/AlertOwner$</AlertOwner>
          <AlertMessageId>$Config/AlertMessageId$</AlertMessageId>
          <AlertParameters>$Config/AlertParameters$</AlertParameters>
          <Suppression>$Config/Suppression$</Suppression>
          <WorkflowId>$MPElement$</WorkflowId>
          <Custom1>$Config/Custom1$</Custom1>
          <Custom2>$Config/Custom2$</Custom2>
          <Custom3>$Config/Custom3$</Custom3>
          <Custom4>$Config/Custom4$</Custom4>
          <Custom5>$Config/Custom5$</Custom5>
          <Custom6>$Config/Custom6$</Custom6>
          <Custom7>$Config/Custom7$</Custom7>
          <Custom8>$Config/Custom8$</Custom8>
          <Custom9>$Config/Custom9$</Custom9>
          <Custom10>$Config/Custom10$</Custom10>
          <ManagementGroupName>$Target/ManagementGroup/Name$</ManagementGroupName>
        </ConditionDetection>
        <WriteAction ID="WA" TypeID="System.Connectors.LibraryPublishAlert" />
      </MemberModules>
      <Composition>
        <Node ID="WA">
          <Node ID="Mapper" />
        </Node>
      </Composition>
    </Composite>
  </ModuleImplementation>
  <InputType>System!System.BaseData</InputType>
</WriteActionModuleType>

Enjoy alert storm mitigation (customization is possible as described in original post) after importing newly customized management pack!

What is new: OpsMgr 2007 R2 - How to reset monitor state with recovery?

MSutara — Mon, 20 Apr 2009 20:28:00 GMT

Cameron had a nice example of using new R2 feature process monitoring in real life, but that raised a question about feature he wanted to use. Following is report of his issue and how we can help him to address his challenges using already existing feature of OpsMgr 2007.

Scenario: Monitoring a system with a process monitor. Define a recovery to reboot the system if it’s not running the process required. Run this recovery automatically on critical state.

Problem: In OpsMgr prior to R2 when a Recovery was created it had an option to “Reset monitor” which would put it to a healthy state. In R2, this now says “Recalculate State Monitor”. This is representing a challenge as described below:

Challenge: Recalculating the state may keep the monitor in a critical state until the system has been rebooted successfully and is in fact running the process. If the process does not start correctly after reboot, it gets stuck in the critical state and the recovery will not run again. With a Reset of this monitor to a Healthy state, this would work properly, but without that option available I am not seeing an effective way to make this work.

Workaround: Recovery is no different than other workflows loaded by OpsMgr and is rather similar to task. It consists of modules that are chained together and should provide some corrective action in order for monitor to fix its state. For that reason, first module could be the module which resets state of the monitor.

Following is module that could be used with recovery directly. It will reset the state of the monitor specified in configuration.

<xsd:element minOccurs="1" name="MonitorId" type="xsd:string" />

</Configuration>

</OverrideableParameters>

<ManagementGroupId>$Target/ManagementGroup/Id$</ManagementGroupId>

<ManagedEntityId>$Target/Id$</ManagedEntityId>

<MonitorId>$Config/MonitorId$</MonitorId>

</WriteAction>

</MemberModules>

</Composition>

</Composite>

</ModuleImplementation>

<OutputType>System!System.BaseData</OutputType>

<InputType>System!System.BaseData</InputType>

</WriteActionModuleType>

Next is another module which can be used as well. It resets the state of the monitor first and then executes command.

<SchemaType>System!System.CommandExecuterSchema</SchemaType>

</IncludeSchemaTypes>

<xsd:element minOccurs="1" name="ApplicationName" type="xsd:string" />

<xsd:element minOccurs="1" name="WorkingDirectory" type="xsd:string" />

<xsd:element minOccurs="1" name="CommandLine" type="xsd:string" />

<xsd:element minOccurs="1" name="TimeoutSeconds" type="xsd:integer" />

<xsd:element minOccurs="1" name="RequireOutput" type="xsd:boolean" />

<xsd:element minOccurs="1" name="MonitorId" type="xsd:string" />

</Configuration>

<ApplicationName>$Config/ApplicationName$</ApplicationName>

<WorkingDirectory>$Config/WorkingDirectory$</WorkingDirectory>

<CommandLine>$Config/CommandLine$</CommandLine>

<TimeoutSeconds>$Config/TimeoutSeconds$</TimeoutSeconds>

<RequireOutput>$Config/RequireOutput$</RequireOutput>

</WriteAction>

<MonitorId>$Config/MonitorId$</MonitorId>

</WriteAction>

</MemberModules>

</Node>

</Composition>

</Composite>

</ModuleImplementation>

<OutputType>System!System.BaseData</OutputType>

<InputType>System!System.BaseData</InputType>

</WriteActionModuleType>

Sealed MP with both modules is attached to this post.

Sample: Attached is also example providing use of modules with simple event based monitor. Monitor targets instance of “Root Management Server” and that is a reason why management pack also defines a view for state of this entity. When you choose to display “Health explorer”, you should be easily able to locate sample monitor.

One of the recoveries present in attached MP runs automatically with WARNING state. Highlighted is MPElement replacement representing monitor you want to reset. (It should be same as value of the attribute Monitor! Also, please observe that using just a reset module causes its output to be displayed in “Context” tab as well as two state changes will appear to have “same” time of change in Health Explorer.

<Category>Maintenance</Category>

<MonitorId>$MPElement[Name="Microsoft.SystemCenter.Community.Monitors.RecoverySample.EventBasedMonitor"]$</MonitorId>

</WriteAction>

</Recovery>

DISCLAIMER:

Please evaluate in your test environment first! As expected, this solution is provided AS-IS, with no warranties and confers no rights. Use is subject to the terms specified at Microsoft.

Tool: OpsMgr 2007 R2 - What to do with Secure Reference Override Alert?

MSutara — Fri, 10 Apr 2009 02:05:00 GMT

Subject of this post is an advanced authoring combining usage of the security features of OpsMgr 2007 with workflows while trying to explain how to troubleshoot alerts which may be raised at the end of such process. On the simple example, I display tool I developed to help resolving ambiguous or unclear obstacles which may surface with this scenario.

I’m not going to discuss why, let’s just say I have a need to create my own Run As profile. This profile is then be populated with custom Run As account I created as well. These steps need to be done manually.

· Open OpsMgr console

· Navigate to “Administration”, then “Run As Configuration”

· Please create “Windows Credentials” account (do not distribute to any computer)

· Please create new profile and associate with previously created account.

Just to note that this post doesn’t aim to explain the internals of association between profile and account nor account distribution details, there are (or will be) official guides available for that exact reason.

Let’s also assume simple rule which generates alert when event 123 is raised in Application log by EventCreate. When created profile is used with this rule while run as account was not distributed to computer where target instance is monitored, event 1108 is raised during configuration load and workflow for this profile is not loaded until issue is corrected.

· Open OpsMgr authoring console

· Create NT event based rule and use this profile with Event data source module.

Because we are using unsealed MP, this rule must be created in same file as initially created profile.

This event 1108 is picked by OpsMgr MP and alert is raised to notify that distribution was not set when Run As account was associated with Run As profile

Dialogs and wizards were re-designed in this milestone to notify about the need to distribute during the creation!

Unfortunately, this new alert may at cases contain somewhat cryptic information increasing TCO of its investigation. If alert is closed without investigating the root cause, it will appear again either after 24 hours from its original creation or when health service restarted.

To simplify investigation of affected Run As profile (where querying a DB would be a necessity), I created SDK tool and associated with the product as “console task”. Upon its execution, tool retrieves all alerts related to Run As Profile and provides user friendly information about affected Run As profile (as long as it was present in the DB).

Another alert that such tool is able to help investigate is based on event 1107 and can be simulated by importing attached MP.

DISCLAIMER:

Please evaluate in your test environment first! As expected, this solution is provided AS-IS, with no warranties and confers no rights. Use is subject to the terms specified at Microsoft. Future versions of this tool may be created based on time and requests.

x86 installation package

x64 installation package

What is new: OpsMgr 2007 R2 - Discover multiple cluster network names (Virtual Server)

MSutara — Fri, 27 Mar 2009 23:34:00 GMT

Multiple network names scenario

Customer has cluster resource group with multiple network name resources associated with IP address resource. This scenario was described by one of the customers:

Configuring DTC in cluster resource group that also contains the default instance of SQL is our standard practice. This is based on documentation written by Mike Grasso (Microsoft). This practice was never question during the Best Practice Review done by Microsoft either and is the reason why it remains same over the years.

Cluster aware application monitoring requires discovering instance of “Virtual Server” while targeting discovery of such application to this instance. In OpsMgr 2007, definition of Virtual Server is cluster resource group with network name and IP address properties.

Previous releases discovered only single instance of Virtual Server and assigned first network name resource as this instance property. For case study mentioned above, such network name was frequently the one belonging to DTC and not SQL. That is the reason why this approach was often error prone with multiple network names assigned to group and was often causing “loss of monitoring”. Workaround consisting of changing the order of the network name properties is not really feasible and extremely costly to attempt in enterprise environment.

Solution at glance

Override has been introduced to allow each IP address -> Network name dependency discovered as instance of Virtual Server. This override must be turned on. Decision was taken not to change default discovery behavior, based on CSS cases up to date, only specific installations of SQL Server required this feature.

How to

1. Open OpsMgr2007 R2 console.

2. Navigate to Authoring, then Object Discoveries

3. Change the scope to Virtual Server (to simplify location of discovery)

4. Select Windows Clustering Discovery and then select “Override the discovery”

5. Pick Multiple Servers Discovery and change override value to true

Upon dialog closure and configuration reload, multiple Virtual Servers should be discovered where feasible.

Another override that is worth to be mentioned!

Imaging that undesired instance of Virtual Server is discovered after override has been applied. Can I remove it? Definitely …

Use Excluded Servers override (also visible in the picture above)! That override has always been present on all up-to-date releases of OpsMgr 2007 (although I’m not aware of anyone using it). Upon selection, reading description should be self explanatory for user to know what the “value” to use with override is:

String which contains semicolon delimited fully qualified names of virtual servers to be excluded from discovery.

Applying this to customer scenario above, when instance of Virtual Server representing DTC is not required, simply place it’s FQDN into Excluded Servers override value and wait for configuration reload. Such instance will then be deleted and will disappear from views.

HOTFIX allowing this same functionality on OpsMgr 2007 SP1 is at works too for those who will not have OpsMgr 2007 R2 depployed in their environments.

What is new: OpsMgr 2007 R2 - Alert Storm Recognition (possible rule misconfiguration)

MSutara — Thu, 26 Mar 2009 07:51:00 GMT

What is new?

OpsMgr 2007 R2 Release candidate finally released and can be downloaded from Connect. What is new in this release? PLENTY! Some of you get a glimpse at those features while evaluating Beta, some will see most improvement for the first time … very exiting!

That is the reason why I would like to start small series where I comment on some of the changes or additions. With this post, I would like to mention a design change to suspend alert creation in order to prevent alert storm – yes, we did bring MOM2005 feature (at least for the rule) back!

Alert storm mitigation at glance:

I need to clarify we are not trying to solve generic data storm problem – that is vNext scenario. We were only addressing possible “rogue” alert generating rule to flood our operational DB and/or raise too many notifications.

Settings to recognize such problem are per agent (across all targeting instances) per individual management group (there are multiple groups settings in registry in multi-homed scenario). Default throttle settings are 50/60/10. This means that if one rule generates more than 50 alerts within 60s, such rule is suspended for 10 minutes (alert generation is disabled)

Option to customize threshold values still exist … Customization will not work in very special deployment scenario – having OpsMgr2007 R2 agent multi-homed to at least one management group monitored by OpsMgr2007 SP1 server (reason is that such agent is forced to use SP1 management packs – and those obviously miss new configuration required when threshold customization was requested). In order for runtime to recognize customized values, health service must be restarted!

When runtime recognizes that possible storm is happening, event 5399 is raised. Following is English snap of such event:

;// Suspend alert generating rule
;// %1 = management group name
;// %2 = workflow name
;// %3 = name of targeted instance
;// %4 = instance id
;// %5 = alert origin (name or message id)
;// %6 = count
;// %7 = time
;// %8 = disabled time

MessageId=5399
SymbolicName=MSG_HS_HM_ALERT_SUSPENDED
Severity=Warning
Language=English

A rule has generated %6 alerts in the last %7 seconds. Usually, when a rule generates this many alerts, it is because the rule definition is misconfigured. Please examine the rule for errors. In order to avoid excessive load, this rule will be temporarily suspended until %8.
%nRule: %2
%nInstance: %3
%nInstance ID: %4
%nManagement Group: %1.

OpsMgr 2007 R2 health monitoring will recognize this event and will raise an alert to notify operator about this problem. Alert needs to be manually closed when corrective action is taken or when conditions causing possible storm are mitigated

Following is an example of customized threshold values. It shows customization 15/30/5 (15 alerts within 30 seconds will cause suspension for 5 minutes (300 seconds). It also shows where in registry such customization should be done. One must create “Alert Count”, “Alert Count Interval” and ”Alert Suspend Interval” under “HKLM\System\CurrentControlSet\Services\HealthService\Parameters\Management Groups\<name of MG>”.

I hope you enjoy this product as much as we hope you would. I always feel happy, this time I also feel rather confident about its quality and value! Questions, comments, feedback (anything) please let me know, I will try to continue this series often (so any things in particular, scream and I move it higher in my TODO list!)

How to: Reset monitor when closing alert?

MSutara — Tue, 03 Feb 2009 06:59:00 GMT

Not something I recommend, one would say I almost regret we did not prohibit closing alerts generated by monitor (especially when auto-resolve feature was used). But recently I learned about some ticketing systems closing alerts where it is unclear if issue was corrected, so I see some necessity to automate the action of resetting monitor health state to re-generate monitor state change when issue still present after ticket was closed.

Problem Description:

Again, as said, there may be legitimate situations where a customer needs to reset monitor health once an alert generated by its state change has been resolved. Such scenario would include automated ticketing systems resolving alerts without providing enough evidence that issue was indeed addressed, situation where operator resolves batch of alert without investigating their root cause (after network outage) and/or by mistake.

Recently I saw this type of request from multiple sources independently of each other so I decided to provide what I believe may be the only solution on how to achieve this functionality – OpsMgr connector.

Analyzing proposal:

OpsMgr connector is nice feature allowing subscribing to alert changes happening for members of specified group. It also allows reacting on such change, in our case by locating monitor associated with alert and requesting its state reset thru SDK call.

Note:
I will not discuss connector internals (registration, used subscription …) but will provide source code for possible reverse engineering of my implementation.

Solution:

Attached, you can find source code for my solution as well as binary you should copy into your RMS product folder. You need to initialize connector when you start it. Such action will import MP with group definition (if MP was not imported already), it creates connector and its subscription (again if such actions are necessary) and starts worker thread to receive monitor raised alerts.

You should see connector created (in Administration section of operations console) after successful initialization.

Currently connector uses group which is populated with instances of computer. This can be adjusted (steps described later) and you should be able to see all members after group calculation rule finishes (in Authoring section of the operations console).

Bellow is an example of alert raised by test event based monitor. When this alert is resolved, state of the monitor resets.

Customization:

As mentioned, this connector will respond to all alerts generated for any monitors which belong to the instance of Windows Computer. It is rather simple to customize the managed entity type you want to use though.

First export connector management pack:

Then edit management pack in XML editor of your choice. You need to change type used with relationship as well as group population rule:

After changes are saved and you imported your management pack, please restart connector application (initialization button will not overwrite changes to MP, but remove button will delete MP from OpsMgr when removing connector from your environment). You can see in the source code that worker thread starts in 3 minutes (to give group calculation time to populate group) and subscription uses 1 minute polling interval to retrieve all alerts as per subscription definition.

One more word of caution to be said is that connector like this may not be fully scalable in big environments and additional work could be needed. This post can serve as nice example and base stone for such more advanced application though.

DISCLAIMER:

HOWTO: “restart” monitoring of my environment? (another version UPDATE)

MSutara — Tue, 20 Jan 2009 08:30:00 GMT

My initial and last posts described restart monitoring tool representing my idea how one can approach “clean up” of the monitoring (scenario described in the first post).

Since then I received some additional and valid feedback and mainly thanks to Microsoft PFE (premier field engineering) team, I can now offer somewhat (mainly speed-vice) improved version of this tool. On top of improving speed, feedback I received was related to the fact that for vast majority, recalculate monitoring task causes no change and has no visible effect. (For those interested, this is due the fact that most monitor types do not define “On-Demand” detection – which is real pity as many would, in my opinion, benefit from having ability o get the state of the monitor “now” rather than wait for next regular monitor detection to make a state change). Such feedback led to the addition of “pure” RESET task (for any instance of the group).

Here is the list of changes:

1. Fixed deployment and support upgrade for both tools (SDK application and Web application).

2. Addition of “Reset Monitoring” task, for plain monitor reset which is not followed by request to recalculate instance state.

3. Change task invocation from PartialMonitoringObject to MonitoringState: reason driven by the fact that timeout can be specified and one do not need to wait for task completion – runtime (currently ) doesn’t return task status to SDK, which means that once task is spawned, it will asynchronously finish its action (well unless low memory or other system type errors occur).

4. Task now provides output as DisplayName of instances which were affected by such action directly – this means those reset (or eventually recalculated). Indirect result of that should be visible thru dependency - one can locate all instances to which affected instance contributed its state and observe if state change for those was necessary (do-able thru Health Explorer)!

Additional info – command line options:

Location:
%Program Files%\System Center Operations Manager 2007 Restart Monitoring Tool

Usage:
Microsoft.SystemCenter.Community.RestartMonitoring.App [/s][/o][/r] /instances id [, id ...]

Options:
/s            “Pure” command line tool. Option missing means, that WinForm version of the tool will be executed.
/o           Flag to recognize if to include information about affected instance(s) in output.
/r            Request instance recalculation after its state reset has performed.

Reset:
Microsoft.SystemCenter.Community.RestartMonitoring.App /s /o /instances <guid – get from PowerShell (all groups (like root management group) are visible thru initial DIR)>

Restart:
Microsoft.SystemCenter.Community.RestartMonitoring.App /s /o /r /instances <guid>

Attached, you can find NEW VERSION of this tool. Attachment for my old posts will update automatically thru my ISP site.

DISCLAIMER:

HOWTO: “restart” monitoring of my environment? (UPDATE)

MSutara — Tue, 13 Jan 2009 23:18:00 GMT

My last post tried to introduce a tool which restarts monitoring of OpsMgr environment. Cameron (and others) is (are) actively looking at its use and here is an update based on some feedback.

1. Following was a state view for computer group which I achieved by selecting “Discovered Inventory” followed by “Change Target Type …” from “State Action” pane. I then selected “View all targets” and picked “Computer Group”.

To avoid this manual intervation, my next version is adding plain state view for target of my “Restart Monitoring” task.

2. “Restart Monitoring” task was using Timeout attribute set to 5 minutes. This is not long enough for task to complete so it often fails with “Timeout Expired” error. This timeout was for task execution which one cannot set thru authoring UI. I did manual change and bumped it to 1 hour. Please customize if that is still not long enough (remember that complete contaiment relationship tree is crawled prior restarting environment monitoring although tool would be flawed if it takes that long – remember: feedback is ALWAYS appreciated)

<Task ID="Microsoft.SystemCenter.Community.RestartMonitoring.Task" Accessibility="Internal" Enabled="true" Target="SCLibrary!Microsoft.SystemCenter.ComputerGroup" Timeout="3600" Remotable="true">

<Category>Maintenance</Category>

<WriteAction ID="PA" TypeID="System!System.CommandExecuter" Target="SCLibrary!Microsoft.SystemCenter.RootManagementServer">

<ApplicationName><![CDATA[%ProgramFiles%\System Center Operations Manager 2007 Restart Monitoring Tool\Microsoft.SystemCenter.Community.RestartMonitoring.App.exe]]></ApplicationName>

<WorkingDirectory>.</WorkingDirectory>

<CommandLine>/s $Target/Id$</CommandLine>

<TimeoutSeconds>3600</TimeoutSeconds>

<RequireOutput>true</RequireOutput>

</WriteAction>

</Task>

3. I fixed x64 deployment. My old packages were installing into “Program Files(x86)” and that broke task targetted to “Computer Group” from being succesfull. Having old version installed, you can simply move “System Center Operations Manager 2007 Restart Monitoring Tool” into “Program Files” (istallation removal thru Windows installer will break though with such approach).

4. It appears that deployment of WebApp needs to be executed as an user who is allowed to create WebSite as well as AppPool.

Attached, you can find NEW VERSION of this tool. Attachment for my old post will update automatically thru my ISP site..

DISCLAIMER:

HOWTO: “restart” monitoring of my environment?

MSutara — Sat, 10 Jan 2009 01:45:00 GMT

My friend Cameron and I discussed following issue as one of the challenges he occasionally faces with his customers. In order to minimize his TCO and manual interventions, I promised to help even when such design request is not making it in feature set for our next release.

Problem Description:

There may be legitimate situations where a customer needs to reset many health monitors at once. For example after a network outage, there might be a significant number of alerts which may have been generated as well as the health state of various items becomes unhealthy. Another case is incorrect approach to Maintenance Mode may cause similar outcome, especially when manual reset monitors or alerts generated without “auto-resolve” feature are present in instances involved with maintenance.

To address this type of situation, the bulk of alerts from the outage need to be closed (which can be done with a PowerShell script). Also, resetting of the health state for multiple systems is required but not viable and manual intervention is needed.

His proposal was that it should be possible to select multiple servers and force their health back to green. Specifically, the health model for those instances would be walked and each monitor not Healthy is reset. This would “restart” the environment to green so that only real issues would resurface as alerts recurred and the states would be updated.

Analyzing proposal:

It is already possible to use SDK tasks to accomplish this proposal. It is event achievable to “speed” up the recognition of real issues by submitting additional “recalculate” state task for given instance (where this task forces to recalculate what the state of given instance should be (at the time of the execution) by working with on-demand detection (assuming that such detection is defined for monitor types used for monitoring of that same instance)).

My approach to implementing this proposal was little different than stated above. I’m not finding every unhealthy monitor, but crawl relationship tree for selected instance recursively adding each instance contributing to the overall health. While making sure instance is present just once, result of reset request against each of those instances affects the health state of all other instances that depend on its state either directly or indirectly.

Note:
Following post contains video trying to describe the difference between Reset and Recalculate tasks. It also touches bases on what does “on-Demand” detection means etc. Please contact me thru comments if I should try to provide additional/different explanation of those monitor features.

Solution:

Attached, you can find source code for my solution as well as installers for deployment of already built binaries. I provide two types of integration with our operations console.

First is having a task associated with managed entity “Microsoft.SystemCenter.ComputerGroup”. This will become present when installation of “RestartMonitoringSetup” for particular SKU succeeds. Following is screenshot providing self-descriptive use of the task:

Second possible integration is using the fact that console is able to act like a browser. Deployment is performed by RestartMonitoringWebSetup and consists of creating Web application and MP import. Web application allows regular web browser to act as the tool which triggers requested restart action. MP associated with this approach contains following WEB view to allow integration with console:

Choosing option with group allows “restart” of the monitoring for all instances contained within all selected groups. Such operation may become rather consuming, as I hinted above, instance space is crawled and all necessary instances (contributing directly or indirectly) are asked to reset and then recalculate their state.

Option to restart monitoring for which active alert is present is doing similar operation as the one made for group, only difference is that likely-hood of having many instances contributing to overall health state is smaller that it is with group (or multiple groups for that matter).

Note:
Additional warning is that tool is not smart enough to recognize if there is an alert raised by monitor, which means that restarting may have no effect as in fact monitors were healthy and alert has been generated by rule. This may change in future versions.

DISCLAIMER:

Clustered Virtual Server 2005 R2 and Operations Manager 2007

MSutara — Thu, 08 Jan 2009 02:20:00 GMT

Problem:

Monitoring of clustered virtual machines (guests) is unreliable with Operations Manager 2007. Instances of Virtual Machine are not monitored without any apparent reason.

Scenario:

Having simple wolfpack cluster (implemented thru Microsoft Cluster Services – MSCS) with just a quorum cluster resource group and clustered Virtual Server 2005 R2 (following this guide). Operations Manager 2007 is installed and health service is pushed to every cluster node. Agents are also pushed to every virtual machine (and those computers become agent managed computers). At the end, Server Virtualization Management Pack for Microsoft System Center Operations Manager 2007 is imported after successful installation and deployment of OpsMgr.

Following is the list of issues one can observe with such setup:

1. Instance of virtual machine (guest) or instance representing windows computer is NEVER monitored when virtual machine cluster resource group is active on the same cluster node as quorum resource group

2. Instance of virtual machine (guest) or instance representing windows computer is monitored fine when its adequate cluster resource group is not active on the same cluster node as quorum resource group

3. Everything works as expected when Server Virtualization MP is not present!

Root cause:

Based on investigation, all instances that are to be monitored by health service running on the virtual machine agent (guest) are disabled. It appears that Server Virtualization MP has not been designed to work under cluster scenarios, mainly due to the fact of associating Virtual Machine Host with virtual computer (cluster).

Possible solution:

Let’s quickly review some MP implementation details. Main discovery targets Microsoft.Windows.Server.Computer (and cluster is discovered as instance of Microsoft.Windows.Cluster.VirtualServer where this type extends Windows Server type). This means that discovery executed on cluster nodes is discovering instances of the VMHost while their virtual machines are active on particular node, but it also means that discovery is executed against quorum cluster resource group (virtual computer).

As mentioned in “Root cause” section, this is, I believe, where MP author might make a mistake with health modeling, as in fact you do not need to associate quorum with virtual machine group as they are able to coexist independently. I base such comment on the fact that virtual machine resource group will stay online when quorum moves to different cluster node. It also remains fully functional and accessible thru VS Administration Website with name of the physical computer (cluster node) used.

Translating this to real life example, having VM1 on node1 and VM2 plus quorum on node2, following seems discovered:

· Instance of VM host for VM1 associated with physical computer 1.

· Instance of VM host for VM2 associated with physical computer 2.

· Instance of VM host for VM2 associated with virtual computer (cluster = quorum)

Investigation revealed that avoiding an association of VM host with virtual computer will fix the issue. This can be done by having DiscoveryPropertyOverride to disable VM host discovery in the context of Microsoft.Windows.Cluster.VirtualServer. (Solution verified while investigating issue.)

<Value>false</Value>

</DiscoveryPropertyOverride>

Implementing workaround:

Really should be as simple as importing attached MP. But I had an environment where following steps had to be taken:

1. Open Operations Console

2. Go to “Administration” -> ManagementPack

3. Select “Microsoft Virtualization Reports” -> right click -> delete

4. Select “Microsoft Virtual Server 2005 R2” -> right click -> delete. If another dependency exist, delete as well (please be careful if such dependency is default MP as diff steps needs to be taken first)

5. Right click in “Administration” pane -> Import management packs

6. Add Microsoft.Virtualization.Reports.mp

7. Add Microsoft.Virtualization.VirtualServer.2005R2.mp

8. Add attached MP

When all management packs are imported at the same time, override is properly applied and such allowed my test environment to work and monitor virtual machines

DISCLAIMER:

Please evaluate in your test environment first! As expected, this solution is provided AS-IS, with no warranties and confers no rights. Use is subject to the terms specified at Microsoft.

Computer group and dependency health rollup in OpsMgr 2007 R2 Beta release

MSutara — Wed, 26 Nov 2008 23:09:00 GMT

… will be broken when you need to create group while inheriting from Microsoft.SystemCenter.ComputerGroup.

</ClassTypes>

This was caused by disabling all dependency monitors defined for abstract managed entity type representing computer group while trying to minimize traffic from agent to server.

<Category>AvailabilityHealth</Category>

<Algorithm>WorstOf</Algorithm>

<MemberUnAvailable>Error</MemberUnAvailable>

</DependencyMonitor>

<Category>ConfigurationHealth</Category>

<Algorithm>WorstOf</Algorithm>

<MemberUnAvailable>Error</MemberUnAvailable>

</DependencyMonitor>

<Category>PerformanceCollection</Category>

<Algorithm>WorstOf</Algorithm>

<MemberUnAvailable>Error</MemberUnAvailable>

</DependencyMonitor>

<Category>SecurityHealth</Category>

<Algorithm>WorstOf</Algorithm>

<MemberUnAvailable>Error</MemberUnAvailable>

</DependencyMonitor>

We enabled all instances of computer groups we needed, but unfortunately you must do the same if you want to see your rollup. Following is a sample of MonitoringPropertyOverride enabling availability section rollup for TestComputerGroup defined above. (you will need all necessary MP references in your override management pack)

</MonitorPropertyOverride>

I’m not sure if this problem is described in release notes, but my post should offer quick insight and easy “fix” solution (same can be actually achieved thru UI when you enable particular dependency monitor for your group)

Where is thy instance monitored and how that affects dependency monitor state?

MSutara — Fri, 14 Nov 2008 10:16:00 GMT

This area is rather complex and multiple factors weight in while configuration service makes the final decision, but I will try to provide somewhat simplified, rule of thumbs examples/points to help to make an educated guess while troubleshooting some state issues. I will start with dependency monitor.

Dependency monitor

Dependency monitor is used to roll up health of other objects (contributors) across a relationship and changing the state of target instance according to used algorithm. (Following is an article describing how to create this monitor using authoring section of operations console MSDN).

Roll up is an important term here! You could “visualize” our monitoring topology as some kind of root system with RMS at the top, followed by layer of management servers and with agents at the bottom. Such perspective then evokes that health state only rolls up from agent to server. Such observation is very correct, BUT one must add that rollup works within self as well.

When rollup works

In other words, dependency monitor is able to rollup state when contributor resides on (is monitored by) the agent while target instance is monitored by server. Monitor is also able to roll up state when both contributors and target are monitored by same health service (regardless of it being agent or server)

When rollup doesn’t work

Above statement also means that rollup won’t work when target is monitored by agent while contributor resides on sibling agent or on server! (Just imagine “root system” again please).

NOTE: This statement is correct at least for all up to date releases of Operations Manager including OpsMgr2007 R2 future release.

Where is this instance monitored?

1000000$ question right there! If you are an operator, you should not need to worry, unfortunately once you become MP author, answering this question may happen to “explain” some of your issues. As I tried to hint, there is many factors affecting final place where instance is monitored, but there are some hints that provide educated guess.

1. Hosting: Due to its nature, hosting is relationship that binds life time of the host to life time of the hosted instance. For that reason, same health service monitoring host instance will also monitor hosted instance. To put it into perspective and provide real life sample, everything hosted by the instance of the computer will be monitored by health service running on that computer.

2. Group and singleton: instances of such managed entity type are almost always monitored by Root Management Server. This is true for all extensions of original base class. Real life example could use “All managed computers group” as something that is monitored by RMS.

3. Logical entity and some others: (as well as all types extending those types like System.ApplicationComponent and Microsoft.Windows.ApplicationComponent) are possibly monitored by RMS. Instances of those ME types are “tricky” as long as they do not bring in “Hosted=’true’” attribute. What I mean by this is the fact that if agent discovers instance of this ME type, proxying must be enabled for discovery to succeed and insert the instance into operational DB (not very known fact, but fact it is!). If proxy was enabled and no additional work done, then instance is always monitored by RMS.

NOTE: I will discuss how to make discovering health service the one to perform monitoring in some future article!

Real life example of how to use this knowledge

Kevin Holman was asked by customer to provide state view which displays state of “Computer Down” monitor. (This was really not the first time I saw similar request, and vast majority of the times, same solution was tried and failed – well, it is an obvious one especially when you are not aware of the dependency rollup limitation – and original plan probably was that you as an author shouldn’t have to be aware of that limitation, but it is here now and needs to be considered.)

Solution which failed

· Created a custom class derived from local application (Microsoft.Windows.LocalApplication) and discover its instance based on registry while targeting instance of the computer (Microsoft.Windows.Computer).

· Create containment relationship between this class and health service watcher (requested “Computer Unreachable” monitor is part of HSW health state) followed by the creation of the dependency monitor to rollup health so health of the custom instance changes with the health changes on requested monitor.

· Create a state view to visualize particular monitor’s state (thru the health of the instance of custom class)

So why would this solution fail? Here is the recap.

· Custom class extends ME type which is already hosted. It is hosted by the computer (that is why discovery required targeting instance of the computer – to provide PrincipalName value for one of the keys). Based on this post, it means that instance of the custom class will be monitored by the same health service as one running on the computer.

· Relationship is starting at RMS. Instances of health service watcher are monitored by RMS. ME type used for HSW is hosted by group and as I said above, instance of group is almost always monitored by RMS.

· Place where contributor is monitored breaks dependency monitor - remember its limitation about rolling up and not down!

· MP verification is currently UNABLE TO WARN you about this problem, its implementation has no knowledge about instance space, it only understands type space of the installation.

Here are snapshots I received from Kevin. They display the problem as observed in health explorer.

How should this be solved?

By evaluation of the failure for original proposal, we can see that instances of health service watcher are monitored by RMS. This article also suggests that dependency monitor is able to roll up the health state when both contributor and target are monitored by same health service. So in fact, we are able to “reuse” the idea from original approach. Only change required is that we simply need to assure that instances of the custom class are monitored by RMS as well. There is a couple of possible ways how to do that, I chose following:

Class: extending ApplicationComponent. Such managed entity is not hosted so is easily discovered by RMS (while targeting instance of the Health Service Watcher).

</ClassType>

Relationship: is containment with HSW as target. This is base stone for dependency monitor.

<Source>Microsoft.SystemCenter.Community.InstanceSpace.ComputerUnreachable.Holder</Source>

<Target>SCLibrary!Microsoft.SystemCenter.HealthServiceWatcher</Target>

</RelationshipType>

Dependency monitor: uses requested monitor as contributor. In this case it is “Computer No Reachable” monitor belonging to HealthServiceWatcher. This monitor also uses state Error when contributing instance is in Maintenance Mode – there is no way to insert custom class into MM when health service watcher class enters maintenance mode, so leveraging this feature of the dependency monitor helps (at least in my opinion).

<Category>StateCollection</Category>

<Algorithm>WorstOf</Algorithm>

<MemberInMaintenance>Error</MemberInMaintenance>

</DependencyMonitor>

Once attached MP is imported, you can observe it picks state from “Computer Down” monitor and presents such in separate view. Below is state view with health explorer.

WARNING: this solution may not scale well in big environment!

Reason for scalability issue lies in the fact of having as many instances of health service watchers as many health services are present in management group (except health service for RMS). Provided solution uses instance of HSW as target for discovery of the custom class, which ultimately means that many workflows are loaded by RMS. There is also additional pressure from dependency monitor (each dependency monitor contributes with another (yet hidden) workflow).

PLEASE consider this an sample, more complex and advanced solution may be needed for real life situation. (Please contact and I will try to help if my bandwidth allows)

DISCLAIMER:

Please evaluate in your test environment first! As expected, this solution is provided AS-IS, with no warranties and confers no rights. Use is subject to the terms specified at Microsoft.

HealthServiceWatcher alerts and connector.

MSutara — Wed, 15 Oct 2008 06:00:00 GMT

Recently I had heard from couple of customers (internal and external) about their trouble with ticketing alerts created from Health Service Watcher. Main complaint was related to the fact that alert is raised by instance of the watcher rather than instance...(read more)

Cluster and maintenance mode

MSutara — Sat, 06 Sep 2008 04:54:00 GMT

Maintenance mode and cluster became a big issue for most of our customers. Starting MM for physical computer serving as one of the cluster node causes FALSE-positives alerts raised from cluster infrastructure monitoring MP. Please accept our apologizes, let’s hope this post clears things out a bit as well as provides some insight/tools how to perform this maintenance mode process.

Following is class hierarchy and relationships (part of management pack guide, so it should not be completely new). We can see that managed entity Cluster contains all managed entities which create cluster infrastructure (remember that hosting is specialization of containment introducing lifetime dependency)

1. MANUAL PROCESS

Usual steps to maintenance physical computer that serves as one of the nodes for failover cluster should consist of finding cluster which contains this computer. The one must start maintenance mode for this managed entity (Cluster) while also starting MM for all contained entities as well. Inserting entity with all contained objects causes node (which extends (inherits from) Windows computer) to enter MM for every local application hosted by this computer (though in SP1, Health Service which is local app is exempt from this rule). Only thing left to do is to start MM for each health service and health service watcher associated with cluster node computers.

2. Powershell script

To automate above said work, you can use attached powershell script. It locates all the nodes related to cluster specified by name sent into script as first argument. After it locates cluster nodes, it also enters maintenance mode for heath service and health service watcher associated with those physical computers.

$clusterName = $args[0]

$HoursInMaintenance = $args[1]

$Description = $args[2]

& {

$ErrorActionPreference = "silentlycontinue"

Add-PSSnapin "Microsoft.EnterpriseManagement.OperationsManager.Client" -ErrorVariable errSnapin;

}

$pathOpsMgr = $env:ProgramFiles + "\System Center Operations Manager 2007"

cd $pathOpsMgr

.\Microsoft.EnterpriseManagement.OperationsManager.ClientShell.Startup.ps1

$startTime = (Get-Date).ToUniversalTime()

$endTime = $startTime.AddHours($HoursInMaintenance)

$clusterCriteria="Name='"+ $clusterName +"'"

$cluster = get-monitoringClass -name 'Microsoft.Windows.Cluster' | get-monitoringObject -criteria:$clusterCriteria

$clusterNodeClass=get-monitoringclass -name 'Microsoft.Windows.Cluster.Node'

$nodes=$cluster.GetRelatedMonitoringObjects($clusterNodeClass)

"Putting cluster " + $clusterName + " into maintenance mode recursively"

$cluster.ScheduleMaintenanceMode($startTime,$endTime,"PlannedOther",$Description,"Recursive")

foreach ($node in $nodes)

{

$healthServiceWatcherCriteria = "HealthServiceName='" + $node.Name + "'"

$healthServiceWatcher = get-monitoringclass -name 'Microsoft.SystemCenter.HealthServiceWatcher' | get-monitoringobject -criteria:$healthServiceWatcherCriteria

"Putting HSW " + $node.Name + " into maintenance mode"

& {

$ErrorActionPreference = "silentlycontinue"

New-MaintenanceWindow -startTime:$startTime -endTime:$endTime -monitoringObject:$healthServiceWatcher -comment:$Description

}

$computer = Get-Agent | Where-object {$_.PrincipalName –eq $node.Name}

$healthService = $computer.HostedHealthService

"Putting HS " + $node.Name + " into maintenance mode"

& {

$ErrorActionPreference = "silentlycontinue"

New-MaintenanceWindow -startTime:$startTime -endTime:$endTime -monitoringObject:$healthService -comment:$Description

}

3. Recap

These steps with entering MM for whole cluster will cause temporary suppression of all monitoring for said cluster infrastructure (including cluster nodes (physical computers)) because all workflows are unloaded. There will be no alerting on cluster infrastructure or computer and health service while instances are in maintenance mode. I will try to answer question if it is possible to enter just active cluster node into maintenance mode in some of my next posts.

Last contacted, better SQL query

MSutara — Mon, 28 Jul 2008 12:57:00 GMT

My post about how to receive approximate last contacted time spawn good feedback and thanks to Robin Drake for rewriting my SQL script without use of cursors while producing single grid. Following is his version:

use OperationsManager

declare @substract float

declare @numberOfMissing float

declare @interval float

-- Get the number of missing heartbeats

select @numberOfMissing = SettingValue from GlobalSettings GS

join ManagedTypeProperty MTP with(nolock) on GS.ManagedTypePropertyId = MTP.ManagedTypePropertyId

where MTP.ManagedTypePropertyName = 'NumberOfMissingHeartBeatsToMarkMachineDown'

-- Get the heartbeat interval

select @interval = SettingValue from GlobalSettings GS

join ManagedTypeProperty MTP with(nolock) on GS.ManagedTypePropertyId = MTP.ManagedTypePropertyId

where MTP.ManagedTypePropertyName = 'HeartbeatInterval'

-- Calculate the amount of lapsed time before a system is marked as non contactable

select @substract = (@numberOfMissing * @interval)/100000

select B.DisplayName, AH.TimeStarted, (cast((cast(tmp.MaxTimeStarted as float)- @substract) as datetime)) as [ApproxLastContactedTime (UTC)],

dateadd(hh, +9, (cast((cast(tmp.MaxTimeStarted as float)- @substract) as datetime))) as 'ApproxLastContactedTime (Pacific)' from Availability A

join BaseManagedEntity B with(nolock) on B.BaseManagedEntityId = A.BaseManagedEntityId

join AvailabilityHistory AH with(nolock) on AH.BaseManagedEntityId = A.BaseManagedEntityId

join

(

select MAX(AHTMP.TimeStarted) AS MaxTimeStarted, BME.BaseManagedEntityId from AvailabilityHistory AHTMP

join BaseManagedEntity BME with(nolock) on BME.BaseManagedEntityId = AHTMP.BaseManagedEntityId

where BME.IsDeleted = 0

group by BME.BaseManagedEntityId

)

TMP on AH.TimeStarted = MaxTimeStarted

where A.IsAvailable = 0 and B.IsDeleted = 0

Robin, Thanks again!