System Center WebLog by Russ Slaten

URL Monitoring Part III – The Monitors

rslaten — Wed, 08 Jul 2009 00:22:00 GMT

This entry is part of the following series:
URL Monitoring Part I – The Design
URL Monitoring Part II – The Classes and Discoveries
URL Monitoring Part III – The Monitors

Creating the monitors was by far the most complex part of this solution because monitors do not use data sources directly. Instead, they use Monitor Types which in turn use data sources. This is because data sources don’t understand the difference between what is considered healthy and what isn’t.

Again, I will use the System Center Configuration Manager Management Point monitor for this example since the other scenarios are built in the same way.

Step 1 – Creating the web page up/down data source. This custom data source is generic and can be used for any URL but its primary function is to return whether or not a web page returns with or without error.

I use the System.Scheduler and Microsoft.SystemCenter.WebApplication.UrlProbe modules so this MP does take a dependency on the Web Application Library MP. Make sure this is added as a reference under File\Management Pack Properties before continuing.

The System.Scheduler module is used for kicking off the URL Probe on a dynamic interval which in my example ends up being every 15 seconds. This is a parameter that is passed at the monitor level.

The Microsoft.SystemCenter.WebApplication.URLProbe module is very complex so I pasted the configuration below:

<ProbeAction ID="Probe" TypeID="MicrosoftSystemCenterWebApplicationLibrary!Microsoft.SystemCenter.WebApplication.UrlProbe">
<Proxy />
<ProxyUserName />
<ProxyPassword />
<ProxyAuthenticationScheme>None</ProxyAuthenticationScheme>
<CredentialUserName />
<CredentialPassword />
<AuthenticationScheme>None</AuthenticationScheme>
<FollowRedirects>true</FollowRedirects>
<RetryCount>0</RetryCount>
<RequestTimeout>0</RequestTimeout>
<Requests>
    <Request>
      <RequestID>1</RequestID>
      <URL>$Config/URLPath$</URL>
      <Verb>GET</Verb>
      <Version>HTTP/1.1</Version>
      <HttpHeaders>
        <HttpHeader>
          <Name>User-Agent</Name>
          <Value>Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)</Value>
        </HttpHeader>
      </HttpHeaders>
      <Body />
      <CheckContentChange>false</CheckContentChange>
      <ContentHash>00000000-0000-0000-0000-000000000000</ContentHash>
      <Depth>0</Depth>
      <ThinkTime>0</ThinkTime>
      <CheckInternalLinks>false</CheckInternalLinks>
      <CheckExternalLinks>false</CheckExternalLinks>
      <CheckResources>false</CheckResources>
      <RequestEvaluationCriteria>
        <StopProcessingIfWarningCriteriaIsMet>false</StopProcessingIfWarningCriteriaIsMet>
        <StopProcessingIfErrorCriteriaIsMet>false</StopProcessingIfErrorCriteriaIsMet>
        <BasePageEvaluationCriteria>
          <WarningCriteria />
          <ErrorCriteria>
            <NumericCriteriaExpressions>
              <NumericCriteriaExpression>
                <NumericRequestMetric>BasePageData/TotalResponseTime</NumericRequestMetric>
                <Operator>Greater</Operator>
                <Value>$Config/ResponseTime$</Value>
              </NumericCriteriaExpression>
            </NumericCriteriaExpressions>
          </ErrorCriteria>
        </BasePageEvaluationCriteria>
        <LinksEvaluationCriteria>
          <WarningCriteria>
            <StatusCodeCriteria>
              <ListNumericRequestMetric>StatusCode</ListNumericRequestMetric>
              <Operator>GreaterEqual</Operator>
              <Value>400</Value>
            </StatusCodeCriteria>
          </WarningCriteria>
          <ErrorCriteria />
        </LinksEvaluationCriteria>
        <ResourcesEvaluationCriteria>
          <WarningCriteria>
            <StatusCodeCriteria>
              <ListNumericRequestMetric>StatusCode</ListNumericRequestMetric>
              <Operator>GreaterEqual</Operator>
              <Value>400</Value>
            </StatusCodeCriteria>
          </WarningCriteria>
          <ErrorCriteria />
        </ResourcesEvaluationCriteria>
        <WebPageTotalEvaluationCriteria>
          <WarningCriteria />
          <ErrorCriteria />
        </WebPageTotalEvaluationCriteria>
        <DepthEvaluationCriteria>
          <WarningCriteria />
          <ErrorCriteria />
        </DepthEvaluationCriteria>
      </RequestEvaluationCriteria>
      <FormsAuthCredentials />
    </Request>
</Requests>
</ProbeAction>

This is passed to another data source in the Web Application Library which uses internal libraries to do the actual HTTP request.

The Configuration Schema tab contains the parameters which are passed to this data source. Currently it only accepts the URL and the interval at which the URL is to be watched.

This Data Source should output WebApplicationData.

Step 2 – Creating the Monitor Type. The Monitor Type takes parameters from the Monitor and passes them onto the data source. It also evaluates the data source to help the monitor determine what criteria equals healthy versus unhealthy.

For States just enter something that makes sense for the type of monitor you will be using.

The Member Modules start with the data source we just created. The Condition Detections are used to evaluate the results of the data source and the Probe Action is used to recalculate the state of the monitor manually.

The configuration of the data source module is fairly simple in that we are accepting parameters from the monitor (which we haven’t created yet) and are just re-passing them to the data source.

The first expression filter evaluates the web page as down.

The second expression filter evaluates the web page as up.

The OnDemandReset Probe Action does not contain any custom configuration information.

Next we have to specify what order the modules will be executed for the regular detection when there is a failure.

Next we have to specify what order the modules will be executed for the regular detection when there is not a failure.

For On Demand we just use the ErrorCodeOK state.

The Configuration Schema defines what parameters this Monitor Type accepts.

Step 3 – Create the Monitor. The monitor is a custom monitor which uses the monitor type we just created. It is also targeted to the SCCMMP class which we created in Part II of this series.

The Configuration is how often we want to query the URL and the URL itself. In this case we want the URL to be the registry value(s) that we discovered in the SCCMMP class. This is how one monitor can be used to watch thousands of websites.

We must tell the monitor which state means Critical and which means Healthy.

I set the example monitor to not alert but this can obviously be configured. Also, in the final solution for my customer we sealed the data sources and monitor types and used a different MP for the classes, discoveries, relationships, and monitors.

To test out this solution the MP you just created can be imported and if the agent from Part II still exists the discovered inventory should look like this:

I don’t have a website with this name so it shows as critical. This is what Health Explorer looks like:

If you want to do some more testing add some real URLs to the SCCMMP registry key and add that key to some more watcher nodes. Also, open the MP attached to Part I and look at the other classes and data sources to understand how the text parsing and response time solutions work.

If you have any questions about this series please let me know.

URL Monitoring Part II – The Classes and Discoveries

rslaten — Wed, 08 Jul 2009 00:14:00 GMT

This entry is part of the following series:
URL Monitoring Part I – The Design
URL Monitoring Part II – The Classes and Discoveries
URL Monitoring Part III – The Monitors

Each of the 3 web applications has its own class. The applicable discovery queries a registry key and based on the values in that key creates an instance of the proper web application’s class. Also, each web application can contain multiple URLs if necessary.

I will use the class I created to monitor System Center Configuration Manager Management Points for this example because they are all built the same way.

Step 1 - The first thing I did was create a new custom class in the authoring console. The base class is System.Perspective. Notice that one of the attributes is “Hosted”. Do not worry about checking this yet because once we add a relationship it will be checked automatically. Also, I always make what I create “Public”. This is important if you decide to seal the MP and want to access the class from a separate management pack.

I added an attribute called “URL” and made it a key value. When the instance of this class is created the “URL” attribute will contain the URL we will be monitoring.

Step 2 – Next I created a relationship so that the URL we are monitoring is tied to the Watcher Node. I made this a hosting relationship and made it public.

Step 3 – I then created a new custom data source that the discovery will end up using to create instances of the SCCM MP Class in addition to the other classes I use in the solution.

I use the System.Scheduler and Microsoft.Windows.ScriptDiscoveryProbe modules in this data source

The System.Scheduler module takes a parameter which is passed to the data source by the discovery itself. This parameter is the interval at which the discovery should be run in seconds.

The Microsoft.Windows.ScriptDiscoveryProbe module is used to run a script that returns discovery data.

The Microsoft.Windows.ScriptDiscoveryProbe module is a bit more complex so I will paste the configuration below:

<ProbeAction ID="DiscoveryProbe" TypeID="Windows!Microsoft.Windows.ScriptDiscoveryProbe">
<ScriptName>WatcherDiscovery.vbs</ScriptName>
<Arguments>$Config/FQDN$ $Config/SourceID$ $Config/ManagedEntityID$ $Config/Class$ $Config/Property$ $Config/URLRegKey$</Arguments>
<ScriptBody><![CDATA['Arg1 = FQDN of Watcher Node
'Arg2 = SourceID (MPElement)
'Arg3 = ManagedEntityID (Target/Id)
'Arg4 = Class ID
'Arg5 = Property ID
'Arg6 = URL Watcher reg key off hklm

Set oArgs = Wscript.Arguments
sFQDN = UCase(oArgs(0))
sSourceID = oArgs(1)
sManagedEntityID = oArgs(2)
sClassID = oArgs(3)
sPropertyID = oArgs(4)
sURLRegKey = oArgs(5)

Set oAPI = CreateObject("MOM.ScriptAPI")
Set oDiscoveryData = oAPI.CreateDiscoveryData(0,sSourceID, sManagedEntityID)

'Determine if this is a Watcher Node
const HKEY_LOCAL_MACHINE = &H80000002
bIsWatcher = False

Set oReg = GetObject("winmgmts:root\default:StdRegProv")
oReg.EnumValues HKEY_LOCAL_MACHINE, sURLRegKey, arrValues
If isarray(arrValues) Then
bIsWatcher = True
End If

If bIsWatcher Then
   For Each sValue in arrValues
      oReg.GetStringValue HKEY_LOCAL_MACHINE, sURLRegKey, sValue, sURL
      Set oWatcher = oDiscoveryData.CreateClassInstance(sClassID)
      oWatcher.AddProperty "$MPElement[Name='Windows!Microsoft.Windows.Computer']/PrincipalName$",sFQDN
      oWatcher.AddProperty sPropertyID,sURL
      oWatcher.AddProperty "$MPElement[Name='System!System.Entity']/DisplayName$",sURL
      oDiscoveryData.AddInstance(oWatcher)
   Next
End If
Call oAPI.Return(oDiscoveryData)]]></ScriptBody>
<TimeoutSeconds>240</TimeoutSeconds>
</ProbeAction>

As you can see this is a vbscript that takes 5 arguments. The script queries a registry key and if the key is found returns discovery data that is used by the discovery to create instances of the applicable class. This is the reason we only need to create a handful of monitors to watch thousands of URLs.

The Configuration Tab of the data source contains all the parameters that are passed into the data source by the discovery, which we will create next.

The Data Types tab should show that this data source outputs discovery data.

Step 4 – Now I need to create the discovery that will use this data source. I target the discovery to Microsoft.Windows.Computer because any computer can be a watcher. The way the data source identifies a watcher is by the registry key we pass it.

Next I added the discovered types. We are discovering instances of the SCCM MP Class so we just add that along with the appropriate attributes we will be populating.

Next I choose the data source we just created and plug in the appropriate parameters. Notice the interval is set to every 24 hours and the URLRegKey is HKLM by default – Software\URLWatcher\SCCMMP. So if the data source finds this registry key it will read the values underneath it. The value should contain a URL like “http://mySCCMMP/sms_mp/.sms_aut?mplist”.

That is it for the class and discovery, just 4 steps (class, relationship, data source, and discovery). To test this out add the following registry key and value to one of your OpsMgr agents and import the MP. Note, you can add more registry values under this key if you want that agent to watch more than one URL in this particular class (so more than one SCCM MP).

After importing the MP you just created you should see new discovered, but not monitored, inventory of type “SCCM MP Class”. Notice that the “Path” is actually the Watcher Node while the Display Name is the URL.

Custom URL Monitoring Part I – The Design

rslaten — Wed, 08 Jul 2009 00:11:00 GMT

The out of the box Web Application template that exists in OpsMgr 2007 SP1 and R2 is useful but there are situations where it just doesn’t fit.

Consider the following scenario: The Contoso Corporation’s global IT headquarters is located in Richardson, TX. They have over 2,000 retail stores spread out across the world and over slow connections. Each of these retail stores have two local servers. One of these servers is a web server that hosts 3 web applications which are used for internal tools critical to the operation of the retail store. The CIO wants the main page of SCCM Management Point site to be checked every 15 seconds to ensure the site is up. She wants the POS Web Application site to be checked every 15 seconds to ensure it is responding within 500ms. Finally, she also wants Store Inventory site to be checked for the text “store inventory web application” every 15 seconds to make sure the site is displaying the appropriate information. Because of the remote nature of these retail stores the CIO wants the check to be done from the non-web server at each retail store where the websites are going to be viewed.

The out-of-the-box Web Application template in OpsMgr doesn’t fit well in this scenario because we need to monitor 6,000 websites every 15 seconds from 2,000 different watcher nodes. One of my customers has a very similar scenario in their environment and asked for a custom solution that uses OpsMgr. I created a solution that does these checks every 15 seconds from the local watcher node at each retail store. The cool part about this solution is we only use 3 monitors to watch all 6,000 web sites, it is dynamic so once the solution is in place there is very little additional administrative overhead, and it scales incredibly well.

Because of the complexity of this management pack I decided to split this topic up into multiple blog entries. The primary purpose of this series is to offer an alternative to the Web Application template in OpsMgr and the secondary purpose is to show how I developed this solution. The series is broken up into the following posts:

Custom URL Monitoring Part I – The Design
Custom URL Monitoring Part II – The Classes and Discoveries
Custom URL Monitoring Part III – The Monitors

My design is generic and was designed for OpsMgr 2007 SP1 although it should also work in R2. It consists of 3 discoveries, 3 classes, 3 monitors, 3 monitor types, and 4 data sources. I attached the entire solution to this blog entry.

Here is a diagram of the design of this MP:

To test this MP in the lab do the following:

1. Create a test website with the text “store inventory web application” on the main page
2. On several OpsMgr agents (which would act as watcher nodes) create the following registry entries:
    - HKLM\Software\URLWatcher\SCCMMP (Key)
      - HKLM\Software\URLWatcher\SCCMMP\SCCMMP:http://<yourwebsite> (String Value)
    - HKLM\Software\URLWatcher\POS (Key)
      - HKLM\Software\URLWatcher\POS\POS:http://<yourwebsite> (String Value)
    - HKLM\Software\URLWatcher\StoreInv (Key)
      - HKLM\Software\URLWatcher\StoreInv\StoreInv:http://<yourwebsite> (String Value)
3. Import the MP and wait 5 minutes
4. Under Monitoring\Discovered Inventory look at the 3 classes in the Website Watcher Library MP
5. Change the text and bring the website up/down to view the results

Monitoring the Monitoring solution

rslaten — Thu, 26 Feb 2009 02:14:00 GMT

What if OpsMgr is in a state where it is running but not working properly? Maybe your RMS is gray or the notification and alerting systems are down. How do you detect this condition so someone can take action to resolve it? Monitoring the console all day isn't very efficient. This is where monitoring the monitoring solution is required.

Below is just one approach to detecting these types of failure conditions. First, we generate an alert on an interval which triggers a notification. The alert auto-resolves and the notification triggers a vbscript that updates a log file on a share. An OpsMgr agent (watcher node) is used to read the log file on the share periodically. If the log file is out of date then the agent triggers an external process which notifies the appropriate administrators that OpsMgr is down. Here is a diagram of this design:

The way the agent notifies an administrator is configurable on the rule that runs the script on the agent. The script writes an event 9898 to the agent's Operations Manager event log. This event can be picked up by any other method that can be used to notify the appropriate administrator of a problem. Also, there is an optional parameter of a process that the script can run. This process could me something like sendmail, net send, createticket, etc... which would be used to notify the appropriate administrator of a problem. This is a good design for determining the health of a management group because it uses the alerting and notification systems and if both of these are working then chances are you're going to be alerted of any issues that occur.

I have created an example MP using this design and provided the steps below to use such an MP.

Step 1 - Create a share that the RMS and agent watcher can write to

Step 2 - Copy the RMSMonitor.vbs script to this share

Step 3 - If an alerting or notification program is to be used then copy this program to the share as well

Step 4 - Configure Command Notification passing the management group name as a parameter to the VBScript. The MG name is used so if you have multiple management groups you can use the same share

Step 5 - Import the MP (it contains one group, one monitor, and one rule with an override)

Step 6 - Add an agent (which will be the watcher node) to the RMSHealthAgent group

Step 7 - Ensure the "Alert Generator" monitor is working by checking for the <mgmtgroupname>RMSMonitor.log. It should contain a new line every 10 minutes.

Step 8 - Configure the "LogWatcher" Rule. Create an override for this rule so that it is enabled for the RMSHealthAgent group. Modify the script so that the proper parameters are passed (the share path, the mgmt group name-optional, and the application you would like to call to alert you - optional).

Step 9 - Confirm it is working properly. If the appropriate RMSMonitor.log is updated within 30 minutes this rule will simply clear the log to keep the size small. If it hasn't then it will update the log saying there is a problem, log an event 9898, and call the alerter program if one was configured. Make sure that the log is either empty or contains fewer than 3 or 4 lines. If it contains more then this rule isn't working properly.

Much of this example would probably need to be customized based on your environment. Things such as how often the alert is generated, how often the watcher node checks the log, how you get notified, changes in the script, etc...

What do you do here?

rslaten — Wed, 25 Feb 2009 05:17:07 GMT

Here at Microsoft I have had the opportunity to work in different roles on several teams all of which contained very smart people. I have to say though, I am currently working with some of the most talented engineers of my career. I work with a team of rock stars and we are part of a larger, very talented, organization called Premier Field Engineering.

My OpsMgr focused peers are DSE's (Dedicated Support Engineers). This means that we work with a single customer, or a very small group of customers, and are their primary resource for everything OpsMgr. We work with our customers on projects, provide training, and also provide solutions to problems that may arise.

My current OpsMgr DSE teammates are Steve Rachui, Kevin Holman, Jonathan Almquist, and Jimmy Harper. If you are reading my blog then chances are you've also read their blogs and know about the quality of work that these guys do.

If you would like to learn more about DSE check out the data sheet available on the Microsoft Services site or send me an e-mail.

Using a property bag to populate an alert description part II

rslaten — Wed, 25 Feb 2009 04:06:00 GMT

Over a year ago I blogged about the proper syntax required to populate an alert description with data returned from a property bag. The syntax in the blog is correct but the idea of doing this confused some. I will describe below how to accomplish this with a rule using the following scenario.

Lets say you have an application that creates an event (we'll use event 9999 as an example) when it has a certain problem. You want to get alerted when this event occurs, but you want to do some additional diagnostics before the alert is generated and populate the alert description with information collected from the diagnostics. I have put a diagram of this scenario below:

I have listed each step that I used to create this management pack below.

Step 1 - Create and name a new management pack in the Authoring Console

Step 2 - Create a new composite data source

Step 3 - Add the Microsoft.Windows.EventProvider member module to the data source

Step 4 - Configure the member module to look for an event id from the application event log where the id = a parameter that is passed in.

Step 5 - Add the Microsoft.Windows.ScriptPropertyBagProbe member module to the data source

Step 6 - Configure the member module to accept parameters for the script name, arguments, and script body. In my example I hard code the script timeout to 30 seconds but you can also make this configurable via a parameter.

Step 7 - Set the order of the member modules so the EventProvider runs first, the ScriptProvider runs second, and the Module Output is last.

Step 8 - Add all of the parameters that need to be passed into this data source. Your data source is now complete.

Step 9 - Create a new custom rule in the Authoring Console. I am targeting my rule to Microsoft.Windows.Computer for example purposes.

Step 10 - On the Modules tab, add our custom data source and then click "Edit" to configure the parameters we want to pass to this data source. My ScriptBody parameter contains a very simple VBScript that returns a string that I want to have appear in the alert description ("This is the information that I collected for my alert"). In the real world this string would obviously be additional useful information that was gathered by the vbscript. I use notepad as my XML editor so this is what my configuration looks like.

Step 11 - After the data source is configured go back to the Modules tab and add a new action of type System.Health.GenerateAlert. Once added click "Edit" and then "Configure" to pull up the Alerting User Interface. Now fill in the appropriate fields for the alert. Notice that the Alert description xpath is looking for the value in the property bag which is populated in the vbscript which we configured in the last step. After this step your rule should be created.

Step 12 - Import the MP and create an event of 9999 in the application event log on an agent

Step 13 - View the alert generated to ensure it contains the string from the VBScript

I hope this example helps explain how information from a property bag can appear in an alert description and might also be a good introduction to the authoring console for some. I have attached the MP that was created by these steps for your reference.

Exporting and importing user roles

rslaten — Tue, 04 Nov 2008 01:24:00 GMT

I have a customer who has many management groups and wants to synchronize the user roles between them. There is no easy way to do this in the UI, but it can be done via PowerShell. One of my co-workers, Jonathan Almquist, blogged about this some time ago. His script allows one to export the user roles, along with any associated user accounts, and import them in another management group. It, however, doesn't export any associated Group Scopes, Tasks, or Views. The script I have created exports everything associated with the user role and is a great example of creating generic types in PowerShell.

The script takes two parameters. The first is the name of the MS you want to connect to and the second is either "export" or "import". The export task creates a file called userroles.xml in the same directory as the PS script ran from. To use the "import" task you must place the userroles.xml in the same directory if it isn't already there. The script then looks for any user roles in the userroles.xml that don't exist and creates them. If the user role does exist, but is missing certain users, group scopes, views, or tasks then it will add them.

I have pasted the script below and also attached a copy to the blog.

#FileName: UserRoleExporter.ps1
#Created by: Russ Slaten (http://blogs.msdn.com/rslaten)
#Created on: 07/24/2008
#Modified by: xxx
#Modified on: xx/xx/xxxx

#===========================FUNCTIONS BEGIN===========================

function Main
{
    #Constants
    $MSPARAM = "MS"
    $TASKPARAM = "TASK"
    $EXPORT = "EXPORT"
    $IMPORT = "IMPORT"
    $XMLFILE = "userroles.xml"
    #Get each of the command line parameters passed by the caller
    $managementServer = GetParameter $MSParam
    $task = GetParameter $TASKPARAM
    Write-Host "Task =" $task
    Write-Host "Management Server =" $managementServer
    #Configure OpsMgr powershell environment
    ValidateSnapIn
    $drive = SetDrive
    if (!$drive){throw("Error configuring OpsMgr Environment")}
    #Connect to the Management Server
    $ms = GetManagementServer $managementServer
    if (!$ms){throw("Error connecting to Management Server")}
    #Start the applicable task
    if ($task.ToString() -eq $EXPORT.ToString()) {ExportRoles $XMLFILE} #Export Roles
    elseif ($task.ToString() -eq $IMPORT.toString()) {ImportRoles $XMLFILE} #Import Roles
    trap [Exception]{Write-Error $_.Exception.Message;ShowHelp;exit;}
} #Main

#Exports Roles from Management Group
function ExportRoles([string]$s)
{
    #Get all non-system user roles
    $userroles = get-userRole | where {$_.IsSystem -eq $False}
    #Make sure some custom user roles exist
    if ($userroles.count -eq 0) {Write-host "No custom user roles found";return 0;}
    #Create the XML object
    $doc = New-Object "System.Xml.XmlDocument"
    $doc.LoadXml("<?xml version='1.0' encoding='utf-8'?><UserRoles xmlns:xsi='http://www.w3.org/2001/XMLSchema-instance' xmlns:xsd='http://www.w3.org/2001/XMLSchema'></UserRoles>")

    #Loop through each user role instance
    foreach ($mo in $userRoles)
    {
        #Add single instance data to XML
        $elem = $doc.CreateElement("UserRole")
        $elem.SetAttribute("Name",$mo.Name)
        $elem.SetAttribute("DisplayName",$mo.DisplayName)
        $elem.SetAttribute("Description",$mo.Description)
        $elem.SetAttribute("Profile",$mo.MonitoringProfile)

        #Add users to XML if any are defined
        if ($mo.Users.count -ne 0)
        {
            foreach ($u in $mo.Users)
            {
                $elem2 = $doc.CreateElement("User")
                $elem2.SetAttribute("UserName",$u.ToString())
                $temp = $elem.AppendChild($elem2)
            }
        }
        #Loop through the groups if the user has explicitly defined group scopes
        if ($mo.scope.MonitoringObjects.count -gt 0)
        {
            foreach ($grp in $mo.scope.MonitoringObjects)
            {
                $elem2 = $doc.CreateElement("Group")
                $elem2.SetAttribute("GroupScope",$grp.ToString())
                $temp = $elem.AppendChild($elem2)
            }
        }
        #Loop through the views if the user has explicitly defined views
        if ($mo.scope.MonitoringViews.count -gt 0)
        {
            foreach ($view in $mo.scope.MonitoringViews)
            {
                $elem2 = $doc.CreateElement("MonitoringView")
                $elem2.SetAttribute("View",$view.First.ToString())
                $elem2.SetAttribute("Bool",$view.Second.ToString())
                $temp = $elem.AppendChild($elem2)
            }
        }
        #Loop through the non-credential tasks if the user has explicitly defined tasks
        if ($mo.scope.NonCredentialMonitoringTasks.count -gt 0)
        {
            foreach ($task in $mo.scope.NonCredentialMonitoringTasks)
            {
                $elem2 = $doc.CreateElement("NonCredentialMonitoringTask")
                $elem2.SetAttribute("Task",$task.First.ToString())
                $elem2.SetAttribute("Bool",$task.Second.ToString())
                $temp = $elem.AppendChild($elem2)
            }
        }
        #Loop through the credential tasks if the user has explicitly defined tasks
        if ($mo.scope.CredentialMonitoringTasks.count -gt 0)
        {
            foreach ($task in $mo.scope.CredentialMonitoringTasks)
            {
                $elem2 = $doc.CreateElement("CredentialMonitoringTask")
                $elem2.SetAttribute("Task",$task.First.ToString())
                $elem2.SetAttribute("Bool",$task.Second.ToString())
                $temp = $elem.AppendChild($elem2)
            }
        }
        #Write this new element to the XML document
        $temp = $doc.get_ChildNodes().Item(1).AppendChild($elem)
        Write-Host "---" $mo.DisplayName "-> exported"
    }
    #Save XML to a file
    $doc.save((Join-path $SCRIPTPATH $s))
} #ExportRoles

#Imports Roles from XML in Management Group
function ImportRoles([string]$s)
{
    #Get existing Non-System User Roles
    $existingUserRoles = get-userRole | where {$_.IsSystem -eq $False}
    #Open XML file
    $doc = New-Object "System.Xml.XmlDocument"
    $doc.load((Join-Path $SCRIPTPATH $s))
    #Loop through each user role
    $userRoles = $doc.SelectNodes("UserRoles/UserRole")
    foreach ($mo in $userRoles)
    {
        #Check to see if user already exists
        $bFound = $false
        foreach ($u in $existingUserRoles)
        {
            if ($mo.Name -eq $u.Name)
            {
                Write-Host $mo.Name ":Role already exists, adding permissions"
                $bFound = $true
                ReplicateUserRoleRights $mo
            }
        }
        if (!$bFound)
        {
            #Create new role
            CreateNewUserRole $mo
        }
    }
} #ImportRoles

function ReplicateUserRoleRights([System.Object]$xml)
{
    #Get the user
    $obj = get-userRole | where {$_.Name -eq $xml.Name}
    Write-Host $obj.Name ":Adding rights"
    #First add users
    foreach ($xmlConsoleUser in $xml.User)
    {
        $bFound = $false
        foreach ($consoleUser in $obj.Users)
        {
            if ($xmlConsoleUser.UserName -eq $consoleUser)
            {
                $bFound = $true
            }
        }
        if (!$bFound)
        {
            if ($xmlConsoleUser.UserName.length -gt 1)
            {
                $obj.Users.Add($xmlConsoleUser.UserName)
                $obj.Update()
                Write-Host $obj.Name "-User:" $xmlConsoleUser.UserName "->added"
            }
        }
        else
        {
            Write-Host $obj.Name "-User:" $xmlConsoleUser.UserName "->already exists"
        }
    }

    #Now Add Group Scopes
    foreach ($xmlGroupScope in $xml.Group)
    {
        $bFound = $false
        foreach ($consoleScope in $obj.Scope.MonitoringObjects)
        {
            if ($xmlGroupScope.GroupScope -eq $consoleScope.ToString())
            {
                $bFound = $true
            }
        }
        if (!$bFound)
        {
            $ret = $true
            [string]$sGuid = $xmlGroupScope.GroupScope
            if ($sGuid.length -ne 0)
            {
                $obj.Scope.MonitoringObjects.Add($sGuid)
                $ret = $obj.Update()
                trap [Exception]{continue}
                if (!$ret)
                {
                    Write-Host $obj.Name "-GroupScope:" $xmlGroupScope.GroupScope "->added"
                }
                else
                {
                    Write-Host $obj.Name "-GroupScope:" $xmlGroupScope.GroupScope "->no matching group"
                }
            }
        }
        else
        {
            Write-Host $obj.Name "-GroupScope:" $xmlGroupScope.GroupScope "->already exists"
        }
    }
    #Create generic type (used for views and tasks if there are any)
    $genericType = [Type] "Microsoft.EnterpriseManagement.Common.Pair``2"
    $typeParameters = "System.Guid","System.Boolean"
    [type[]] $typedParameters = $typeParameters
    $closedType = $genericType.MakeGenericType($typedParameters)

    #Now Add Views
    foreach ($xmlView in $xml.MonitoringView)
    {
        $bFound = $false
        foreach ($consoleView in $obj.Scope.MonitoringViews)
        {
            if ($xmlView.View.ToString() -eq $consoleView.First.ToString())
            {
                $bFound = $true
            }
        }
        if (!$bFound)
        {
            $ret = $true
            if ($xmlView.View.length -gt 1)
            {
                if ($xmlView.bool -eq $false)
                {
                    $second = $false
                }
                $params = [guid]$xmlView.View,$second
                $pair = [Activator]::CreateInstance($closedType, $params)
                $obj.Scope.MonitoringViews.Add($pair)
                $ret = $obj.Update()
                trap [Exception]{continue}
                if (!$ret)
                {
                    Write-Host $obj.Name "-View:" $xmlView.View "->added"
                }
                else
                {
                    Write-Host $obj.Name "-View:" $xmlView.View "->no matching group"
                }
            }
        }
        else
        {
            Write-Host $obj.Name "-View:" $xmlView.View "->already exists"
        }
    }
    #Now Add noncredentialmonitoringtasks
    foreach ($xmlNonCred in $xml.NonCredentialMonitoringTask)
    {
        $bFound = $false
        foreach ($consoleNonCred in $obj.Scope.NonCredentialMonitoringTasks)
        {
            if ($xmlNonCred.Task.ToString() -eq $consoleNonCred.First.ToString())
            {
                $bFound = $true
            }
        }
        if (!$bFound)
        {
            $ret = $true
            if ($xmlNonCred.Task.length -gt 1)
            {
                if ($xmlNonCred.bool -eq $false)
                {
                    $second = $false
                }
                $params = [guid]$xmlNonCred.Task,$second
                $pair = [Activator]::CreateInstance($closedType, $params)
                $obj.Scope.NonCredentialMonitoringTasks.Add($pair)
                $ret = $obj.Update()
                trap [Exception]{continue}
                if (!$ret)
                {
                    Write-Host $obj.Name "-NonCredTask:" $xmlNonCred.Task "->added"
                }
                else
                {
                    Write-Host $obj.Name "-NonCredTask:" $xmlNonCred.Task "->no matching group"
                }
            }
        }
        else
        {
            Write-Host $obj.Name "-NonCredTask:" $xmlNonCred.Task "->already exists"
        }
    }
    #Now Add credentialmonitoringtasks
    foreach ($xmlCred in $xml.CredentialMonitoringTask)
    {
        $bFound = $false
        foreach ($consoleCred in $obj.Scope.CredentialMonitoringTasks)
        {
            if ($xmlCred.Task.ToString() -eq $consoleCred.First.ToString())
            {
                $bFound = $true
            }
        }
        if (!$bFound)
        {
            $ret = $true
            if ($xmlCred.Task.length -gt 1)
            {
                if ($xmlCred.bool -eq $false)
                {
                    $second = $false
                }
                $params = [guid]$xmlCred.Task,$second
                $pair = [Activator]::CreateInstance($closedType, $params)
                $obj.Scope.CredentialMonitoringTasks.Add($pair)
                $ret = $obj.Update()
                trap [Exception]{continue}
                if (!$ret)
                {
                    Write-Host $obj.Name "-CredTask:" $xmlCred.Task "->added"
                }
                else
                {
                    Write-Host $obj.Name "-CredTask:" $xmlCred.Task "->no matching group"
                }
            }
        }
        else
        {
            Write-Host $obj.Name "-CredTask:" $xmlCred.Task "->already exists"
        }
    }
} #ReplicateUserRoleRights

function CreateNewUserRole([System.Object]$xml)
{
    #Create a new User Role Object
    $obj = new-object Microsoft.EnterpriseManagement.Monitoring.Security.MonitoringUserRole
    #Populate the common fields for the userrole
    $obj.Name = $xml.Name
    $obj.DisplayName = $xml.DisplayName
    $obj.Description = $xml.Description
    $profile = $mg.GetMonitoringProfiles() | where {$_.Name -eq $xml.Profile}
    $obj.MonitoringProfile = $profile
    $mg.InsertMonitoringUserRole($obj)
    #Now Replicate the rights associated with this role
    Write-Host $xml.Name ":New user role created"
    ReplicateUserRoleRights $xml
} #CreateNewUserRole

#Validates the parameters passed by the caller
#Pass either "MS" or "TASK" depending on which parameter you want
function GetParameter([string]$s)
{
    #First, make sure the caller passed at least two parameters to the script
    if ($cmdLineArgs.count -ne 2) {throw("Error getting command line parameters")}
    #Now, validate the contents of the parameter
    if ($s -eq "MS"){return $cmdLineArgs[0]}
    elseif ($s -eq "TASK")
    {
        if ($cmdLineArgs[1] -eq "EXPORT"){return "EXPORT"}
        elseif ($cmdLineArgs[1] -eq "IMPORT") {return "IMPORT"}
        else {throw("Error matching 2nd command line parameter")}
    }
    else {throw("Error getting command line parameters")}
} #GetParameters

#This function tests whether the opsmgr snap-in has been added
function ValidateSnapIn
{
    $snapins = PsSnapIn | select-Object name
    $added = $false
    foreach ($o in $snapins)
    {
        if ($o -like "*Microsoft.EnterpriseManagement.OperationsManager.Client*")
        {
            $added = $true
            break
        }
    }
    if (!$added)
    {
        add-PsSnapIn "Microsoft.EnterpriseManagement.OperationsManager.Client"
        write-Host "OpsMgr Snap-in added."
    }
    else
    {
        write-Host "OpsMgr Snap-in already added."
    }
} #ValidateSnapIn

function SetDrive
{
#Sets location
set-location "OperationsManagerMonitoring::"

    $drv = psdrive | select-Object name
    $added = $false
    foreach ($d in $drive)
    {
        if ($d -like "*Monitoring*")
        {
            $added = $true
        }
    }
    if (!$added)
    {
        New-PSDrive -Name: Monitoring -PSProvider: OperationsManagerMonitoring -Root: \
        write-Host "Monitoring Drive added."
    }
    else
    {
        write-Host "Monitoring Drive already added."
    }
    return $psdrive
} #SetDrive

function GetManagementServer([string]$s)
{
    New-ManagementGroupConnection -ConnectionString: $s
    cd Monitoring:\$s
    $mg = (get-item .).ManagementGroup
    return $mg
} #GetManagementServer

function ShowHelp
{
    Write-Host "-----UserRoleExporter.ps1 Help-----"
    Write-Host "This is an example script for exporting and importing user roles from OpsMgr"
    Write-Host ""
    Write-Host "UserRoleExporter.ps1 Usage:"
    Write-Host "Parameter 1: <Management Server Name>"
    Write-Host "Parameter 2: import or export"
    Write-Host "Example: UserRoleExporter.ps1 myRMS export"
    Write-Host ""
} #ShowHelp

#===========================FUNCTIONS END===========================

#Get CmdLine Args and set global
$CmdLineArgs = $Args

#Get path script was called from and set global
$SCRIPTPATH = $MyInvocation.Mycommand.Path | Split-Path -Parent

#Get Management Group Connection
$mg = (get-item .).ManagementGroup

#Calls the main program
Main

# End of Script

The Mystery of the stuck alerts

rslaten — Fri, 17 Oct 2008 22:13:00 GMT

Occasionally I run across an issue that requires immediate action be taken to resolve the problem while any root cause analysis is an afterthought. Last week I handled such an issue.

My customer has a connector that was continually sending around 20 alerts from OpsMgr to an event correlation product. There are many different monitoring solutions that send data to this product and the connector was flooding the event correlation product with the same 20 alerts which resulted in plugging up their alert ticketing system. We consulted with the 3rd party connector vendor for a few hours but progress was a bit slow so I tried to think of a way, from an OpsMgr perspective, to make this stop occurring.

I consulted a co-worked, Tim "god of everything connector" Helton, and he came up with an idea. His idea involved installing a bogus connector, changing the ConnectorId property on the "stuck" alerts to the bogus connector, and uninstalling the bogus connector. Since you can't change the ConnectorId to NULL through the SDK you have to install the bogus connector first. Once the bogus connector is uninstalled the ConnectorId will then change to NULL.

I started by getting a bogus connector. I used a simple connector from a class that Tim taught but you can easily create your own. Ambrose Wong wrote a good connector guide which contains some sample code for creating a connector.

I then installed the connector in my lab and obtained the GUID of the connector. Next I created a PS script to change the ConnectorId on all those pesky alerts. Here is my script:

##The alertID’s were ID’s of stuck alerts
##The Connector ID is Tim's demo connector

##This function does the set
function SetAlertDestination([string]$s)
{
$alert = get-alert | where {$_.ID -eq $s}
Set-AlertDestination -Alert $alert -Connector $connector
}

##Get the bogus connector using its GUID
$connector = get-connector | where {$_.ID -eq 'aaaaaaaa-1111-2222-3333-0305e82c3301'}

##Set it to the bogus connector
SetAlertDestination "5ce5fd8f-790e-4e87-aabc-c0aae3f935a6"
SetAlertDestination "20087c39-6669-4791-910d-14e9d89d5650"
SetAlertDestination "1e6362a8-cb06-4f2a-a4ec-90e2a1b6fa24"
SetAlertDestination "c5a748e8-d299-4743-b77d-75279716c6ea"
SetAlertDestination "a33c0582-f1b9-42de-9639-60057641a261"
SetAlertDestination "6ee1f643-5589-4481-a38e-b8d36b57b5cc"
SetAlertDestination "e93e0bf5-0563-4edc-b8df-fa0bc4f96e0a"
SetAlertDestination "339510ea-7dee-482c-9388-398f7982c72d"
SetAlertDestination "2703c193-3712-4eb4-9d16-5754a4b4d646"
SetAlertDestination "f1474a98-1383-4173-89b4-3f98dfa7a58b"
SetAlertDestination "6c70a0bf-0ca3-4039-9195-a4640d2edc01"
SetAlertDestination "97331f49-b535-4edb-ba83-722ed0c36f4d"
SetAlertDestination "203ff402-a3fd-4fe4-b672-2ff30449011a"
SetAlertDestination "4855f316-9bcf-447f-8888-5bd19106abd5"
SetAlertDestination "a0716409-3d8e-42a4-b70f-ecbf9dbfec92"
SetAlertDestination "3edc53e6-2d43-4a4d-98ce-b50261fe63e5"
SetAlertDestination "7b6dfd88-1d59-46d6-bbc9-351bd04e1349"
SetAlertDestination "56fea496-f4fe-4563-9b6e-2e614d75b601"
SetAlertDestination "3a71c969-7eb8-49fb-904f-6b9673d52396"
SetAlertDestination "fb51278a-8b01-4484-9470-d2bd1a1c9316"

After doing some testing in my lab I had my customer install the demo connector and run the script. The script ran and did end up changing the ConnectorId of the stuck alerts and the connector stopped trying to forward them. The connector was then uninstalled and the ConnectorId property went to NULL on all the bad alerts. We don't know the root cause here, but at least all is now well in ticketing system.

Get all alerts within a certain time period

rslaten — Tue, 02 Sep 2008 17:59:00 GMT

A co-worker of mine ran into an alert flood scenario and needed to resolve a large number of alerts. He tried the following PowerShell command:

Get-Alert | where {$_.ResolutionState -eq "New"} | Resolve-Alert

This eventually ended up throwing an out of memory exception. This command is obviously attempting to resolve all of the alerts from the flood at once. Another alternative would be to resolve them in chunks. I provided a script that retrieves all alerts that were raised within the last 30 minutes. This can be modified to get only new alerts, set the resolution state of each alert, and also get the alerts from the past hour, day, month, etc...

#rslaten 08/29/2008

#get current time
$now = Get-Date

#subtract 30 minutes
$newTime = $now.AddMinutes(-30)

#format time correctly
$timeFormat = $newTime.ToShortDateString() + " " + $newTime.ToLongTimeString()

#set criteria
$criteria = "TimeRaised >= '" + $timeFormat + "'"

#query OpsMgr
$alerts = Get-Alert -Criteria $criteria

#print to console
Write-Host "Command = Get-Alert - Criteria" + $criteria
foreach ($alert in $alerts) {$alert.name + ":" + $alert.TimeRaised}

Check out the PowerShell Team Blog for more commands that you can use on the DateTime object.

Adding Run AS accounts using powershell

rslaten — Wed, 27 Aug 2008 19:51:00 GMT

A customer of mine ran across a situation where they had to add a significant number of runas accounts to OpsMgr and didn't want to do it through the UI. The customer happened to have the account information handy and wanted to know how to script this. I provided the following example:

#Rslaten 08/26/2008

#Connect to MG
$mg = (Get-Item .).ManagementGroup

#Create new account object
$oAccount = New-Object Microsoft.EnterpriseManagement.Monitoring.Security.MonitoringWindowsCredentialSecureData

#Input basic account details
$oAccount.Domain = "MyDomain"
$oAccount.UserName = "testaccount"
$oAccount.Name = "Test Account"
$sPassword = "MyPassword"

#Set Password (more complicated than the previous settings)
$oPassword = New-Object System.Security.SecureString
foreach($char in $sPassword.ToCharArray())
{
$oPassword.AppendChar($char)
}
$oAccount.SecureData = $oPassword

#Finally add the account
$mg.InsertMonitoringSecureData($oAccount)

Obviously you could modify this to read the account information from a txt file. Another way (probably a more secure way) is to gather the account information via command line. Check out the Get-Credential command if you want to do this.

Event 5206

rslaten — Wed, 27 Aug 2008 19:42:00 GMT

I was doing a health check for a customer and ran across the following warning event which occurred almost daily:

Event ID: 5206
Event Description: In memory container (hash table System.Health.EntityStateChangeData) had to drop data because it reached max limit. Possible data loss.

I did some research and found that the default for this container is 1024. This can be configured through the following DWORD value:

HKLM\System\CurrentControlSet\Services\HealthService\Parameters:”State Queue Items”

I recommended that my customer set this to 2048 and if the errors continue to recur then try doubling it again. There my be some other underlying issues that cause this queue to fill up so this my need to be troubleshot further if errors continue after this value has been raised significantly.

New Role at Microsoft

rslaten — Wed, 27 Aug 2008 19:33:00 GMT

This is my first post in 2008 and definitely won't be my last. In April I transitioned to a new role here at Microsoft. I was previously in Escalation Engineer for most of the System Center product line. I am now a dedicated support engineer focused exclusively on System Center Operations Manager 2007. I provide support, training, and consulting to customers with the some of the largest and most complex OpsMgr deployments in the world. This means that most of my upcoming posts will be focused on OpsMgr. Many of my peers do the same thing for SMS and SCCM and I work with them from time to time so I will still continue to try and post some SMS/SCCM goodness.

MOM 2005 MCF throws an exception when UpdateAlerts() is called

rslaten — Thu, 08 Nov 2007 19:00:00 GMT

After calling UpdateAlerts() an exception is returned: "Value does not fall within the expected range". DBGView will show the following output:

[4680] [E] 04680.01796, (cdasmomconnectorframework.cpp::2114) Could not decode unknown vartype 36 !
[4680] [E] 04680.01796, (cdasmomconnectorframework.cpp::2045) Could not decode unknown vartype 36 !
[4680] [E] 04680.01796, (cdasmomconnectorframework.cpp::1991) Could not determine ADO type or typeLength for [0][0] Param Name is @AlertId and VARTYPE is 36 [hr=0x80070057]!
[4680] [E] 04680.01796, (cdasmomconnectorframework.cpp::1342) AppendParamsFromSafeArray() failed for AlertHistory # 0 [hr=0x80070057] !

This error occurred for a customer after compiling their custom application in Visual Studio 2005. This did not happen when it was compiled with Visual Studio 2003. It turns out that the MCF isn't designed to work with the .NET 2.0 runtime. The solution here is to compile the application in VS 2003 or us MSBee (http://www.codeplex.com/MSBee) to force the use of the .NET 1.1 runtime.

SNMPVarBinds and the Alert Description Field in OpsMgr 2007

rslaten — Wed, 31 Oct 2007 20:49:00 GMT

I recently had the opportunity to learn a lot about SNMPVarBind values and populating those values in the Alert description Field from a monitor and rule. A customer was having difficulty determining the proper syntax to use for this and with the help of some folks in the OpsMgr product group I've documented it below:

For a Rule:

Added "Simple Network Management Protocol" through Add/Remove programs on the OpsMgr server
Configured the community name as "public" and to accept packets from any host
In the OpsMgr console, ran the Network Devices Discovery Wizard
It discovered my SNMP device and I selected it to manage
Created a new SNMP Trap (Alert) rule and tested generating alerts using trapgen
Used the following syntax for the Alert description: $Data/EventData/DataItem/SnmpVarBinds/SnmpVarBind[1]/Value$
Tested by creating an SNMP Trap using trapgen.exe
Alert appeared and the alert description contained the value specified

For a Monitor:

Use the following syntax for the Alert description: $Data/Context/SnmpVarBinds/SnmpVarBind[1]/Value$

To figure out the proper value we used a collection rule and viewed the raw event data from the collection rule. Also, we found that if you enter more than 10 parameters for Alert description they won't get reported properly. This is currently a limitation in OpsMgr 07.

New System Center Product Group Blog

rslaten — Wed, 31 Oct 2007 18:07:00 GMT

The System Center Product Group has started a new blog that will contain some useful high level information about the System Center product line. The first post is by Brad Anderson who is "the man" when it comes to the System Center branded products. I expect to see more posts like this from management within the product team so if you're interested I would suggest subscribing to the feed.

http://blogs.technet.com/systemcenter

-Russ