November, 2006

  • Jakub@Work

    Differences between MCF in MOM 2005 and SCOM 2007

    • 78 Comments

    I wanted to go through and outline some of the changes we made for MCF since our last release. The things I really wanted to accomplish in redesigning MCF was to make it more integrated with SCOM 2007 than it was in MOM 2005, to make necessary performance improvements and maintain at least the same level of functionality as we had before, while keeping with the model-based approach of SCOM 2007.

    Integration with SCOM 2007

    With the significant changes that took place between these releases, we had an opportunity to integrate MCF more closely with the product than it was before. The first step here, was getting rid of resolution state as the mechanism to mark alerts for forwarding. Instead, the alert object has a special nullable field (ConnectorId) that represents the identifier of the connector that currently "owns" this alert.

    There are a few ways to mark alerts for forwarding. In the UI, you can right-click on an alert and there will be a fly-out menu with available connectors so you can explicitly mark an alert for forwarding. Programmatically, the ConnectorId field on a MonitoringAlert object is settable and once set can be applied by calling the Update() method on the alert. Lastly, we have introduced a concept of connector subscriptions that are exposed via the UI and programmatically (MonitoringConnectorSubscription) that allow users to "subscribe" to a particular set of alerts by using criteria that should be marked for a particular connector.

    Another aspect of more tight-knit integration is the fact that 100% of MCF functionality is available via the managed SDK. While we still ship with a web-service (using WCF) and a proxy to communicate with it, the recommendation is to use the SDK to do MCF functionality unless you are not running on windows; this provides a richer object model and a better programming experience. The namespace in the SDK that servers MCF functionality is Microsoft.EnterpriseManagement.ConnectorFramework and the main object the provides the root of much of the functionality is ConnectorFrameworkAdministration which can be retrieved off a ManagementGroup object via GetConnectorFrameworkAdministration().

    The last aspect of integration is the fact that a connector is modeled in SCOM 2007. While we aren't planning on shipping anything that provides health information for a connector out of the box (time constraints), there is an ability for customers and partners who write connectors to develop a health model for their connector. In a nutshell, creating a connector creates a new instance of Microsoft.SystemCenter.Connector. In order to develop a health model around a new connector, a user should extend this class with their own and develop a health model targeted at the new class. Once this is complete, the built-in setup process will need to be augmented by the connector author to discover their connector as an instance of their new class. Assuming the model is developed and working correctly, the connector object will now have state like anything else SCOM is managing.

    Performance Improvements

    In 2005 we have many performance issues at scale due to the fact that a lot of data is copied from place to place. In the MOM to MOM Connector scenario in particular, we physically moved alerts from bottom tiers to the top tier, replicating the data and keeping it in sync.

    The first change for performance was with respect to tiering. We no longer copy alerts from bottom tiers to the top tier, but instead make available alerts from multiple tiers on the top tier. A SCOM administrator can setup tiers in the UI and make them available to connectors. Once available, MCF provides special "tiered" versions of various methods that aggregate data across these tiers the administrator setup. For instance, calling GetMonitoringAlertsForTiers() will loop through all available tiers and aggregate the results, returning alerts from each. The recommended approach for tiering, however, is to manage the calls to each tier from within the connector code itself as this allows more robustness in terms of error handling and allows for parallel processing for multiple tiers. There is much more to talk about with respect to tiering and how to accomplish this, but it is not in the scope of this article. If there is interest, I will put together a more detailed post on tiering.

    The next improvement for performance has to do with how we process alerts and alert changes. In MOM 2005, we distinguished between "New" and "Updated" alerts and actually kept a cached copy of each change to an alert in a pending list for each connector. If an alert changed 3 times, we would have three independently acknowledgeable changes and three copies of the alert in the cache. This caused performance problems when dealing with large scale deployments. In order to remedy this problem, in SCOM we manage alerts for connectors in place. The alerts in the alert table have a timestamp associated with non-connector related changes (so you don't get an update to an alert you changed as a connector) and that timestamp is checked against the current bookmark of a connector when retrieving alerts. Acknowledging data is now a function of moving a connectors bookmark. In other words, when a connector retrieves 10 alerts, when they are done processing these alerts, they should acknowledge receipt and processing by calling AcknowledgeMonitoringAlerts() with the maximum time of the alerts in the batch that was retrieved. This will ensure no data is lost.

    Important Note: - Since SQL datetime is not particularly granular, it is possible that two alerts with the same datetime timestamp get split between two SELECT calls. Even worse, when acknowledging across tiers, the system times on the machines will not be the same which might cause data loss by acknowledging with a time from one alert on one tier to another tier that has a time slightly in the past. In order to remedy this we actually have a built-in "delay" when retrieving alerts so that we don't get the newest alert but instead get alerts older than 30 seconds. This actually causes some "weird" behavior in that if you mark an alert for forwarding and immediately call GetMonitoringAlerts() you won't get anything, even though the alert is marked. This value is configurable in the database. This query should do it, pending any changes to the property name at RTM: (The SettingValue is the number of seconds of "buffer" you want in GetMonitoringAlerts calls. It needs to be at least 1)

    UPDATE GlobalSettings SET SettingValue = 1 WHERE ManagedTypePropertyId in
    (SELECT ManagedTypePropertyId FROM ManagedTypeProperty WHERE ManagedTypePropertyName = 'TierTimeDifferenceThreshold')

    Other Random Things

    1. We no longer support discovery data insertion via the web-service. This can be done as described in my earlier post.
    2. We support batching (and in fact enforce it with a maximum size of 1000), however the batch size is an "approximation" as ties in the alert timestamp might require more than the requested number of alerts to be returned. Also, batch size for tiered methods is the total across all tiers, not per tier.
    3. The MonitoringConnector object in the SDK has an Initialized property and a Bookmark property available for reading. The web-service has GetConnectorState() and GetConnectorBookmark() for the same purpose.
    4. Alert insertion is not supported. Please see this post and this one.
    5. Alert history retrieval is not done as part of the alert retrieval. Via the SDK there is a GetMonitoringAlertHistory() method on the alert. Via the web-service we have GetMonitoringAlertHistoryByAlertIds() for the same purpose.
    6. We added a way to generically get alerts via the web-service: GetMonitoringAlertsByIds(). This is available in the SDK in a variety of methods.
    7. When updating alerts via the web-service, the connector is implicit. When updating alerts via the SDK on behalf of a connector, it is important to use the overload that accepts a MonitoringConnector object to indicate it is the connector that is changing the alert.
  • Jakub@Work

    Creating and Updating Groups

    • 22 Comments

    I've received several questions over the past few days about how to create and update groups in the SDK. Below is a commented sample of how to do it.

    What InsertCustomMonitoringObject group does, is create a new singleton class that derives from Microsoft.SystemCenter.InstanceGroup. It also creates a rule that populates this group which essentially defines the formula and inclusion and exlusion lists for the group. Hopefully the sample and comments in the code below explain this in more detail.

    Note: A quick apology on the formating of the strings for formulas below. I found that when I had extra spaces, the schema validation failed and caused some exceptions to be thrown. This way, a direct cut and paste should work.

    Edit: If you receive an ArgumentException for the references collection, this means that your default management pack already has a reference for the given management pack. When this happens, replace the alias I used with the existing alias for the management pack in the default management pack and don't add it to the newReferences collection.

    using System;

    using Microsoft.EnterpriseManagement;

    using Microsoft.EnterpriseManagement.Configuration;

    using Microsoft.EnterpriseManagement.ConnectorFramework;

    using Microsoft.EnterpriseManagement.Monitoring;

     

    namespace Jakub_WorkSamples

    {

        partial class Program

        {

            static void InsertAndUpdatingGroups()

            {

                // Connect to the sdk service on the local machine

                ManagementGroup localManagementGroup = new ManagementGroup("localhost");

     

                // Create the formula

                // There can be multiple membership rules, no need for a root tag.

                // The monitoring class is the class of instance you want in the group.

                // The Relationship class is a relationhip whose source type needs to

                //      be in the base class hierarchy of the class of your group (in this

                //      case we actually create a new class using the insert method on

                //      ManagementPack and this class will derive from InstanceGroup

                //      which is the source of InstanceGroupContainsEntities) and the target

                //      class needs to be in the base class hierarchy of the of the

                //      aforementioned MonitoringClass.

                //  You can also have an include and exclude list of specific entities that

                //      also must match the relationship class and monitoring class critiera.

                //      So in the example below, you could only include or exclude instances

                //      that derive from Microsoft.Windows.Computer.

                //      These look like this: (if both are specified the include list is first)

                //      <IncludeList>

                //        <MonitoringObjectId>f5E5F15E-3D7D-1839-F4C6-13E36BCD982a

                //        </MonitoringObjectId>

                //      </IncludeList>

                //      <ExcludeList>

                //        <MonitoringObjectId>f5E5F15E-3D7D-1839-F4C6-13E36BCD982a

                //        </MonitoringObjectId>

                //      </ExcludeList>

                //  Prior to the include list, you can also add an expression to include

                //  instances by. The element is defined as Expression and the schema for

                //  it (as well as this entire MembershipRule schema) is defined in the

                //  Microsoft.SystemCenter.Library management pack.

                //      An example of an expression:

                //      <Expression>

                //          <SimpleExpression>

                //           <ValueExpression>

                //             <Property>

                //              $MPElement[Name="Microsoft.SystemCenter.HealthService"]/IsRHS$

                //              </Property>

                //           </ValueExpression>

                //           <Operator>Equal</Operator>

                //           <ValueExpression>

                //              <Value>False</Value>

                //           </ValueExpression>

                //          </SimpleExpression>

                //      </Expression>

                //      This expression can reference properties of the class of the membership

                //      rule and in this case would include any health services that are not

                //      the root health service.

                //      Note: this example doesn't work with the rule I have below, it is simple

                //      for illustrative purposes. I would need to filter by a

                //      Microsoft.Windows.Computer property in order to use it below.

     

                string formula =

                    @"<MembershipRule>

                    <MonitoringClass>$MPElement[Name=""Windows!Microsoft.Windows.Computer""]$</MonitoringClass>

                    <RelationshipClass>$MPElement[Name=""InstanceGroup!Microsoft.SystemCenter.InstanceGroupContainsEntities""]$</RelationshipClass>

                    </MembershipRule>";

     

                // Create the custom monitoring object group

                CustomMonitoringObjectGroup allComputersGroup =

                    new CustomMonitoringObjectGroup("Jakub.At.Work.Namespace",

                    "AllComputerGroup",

                    "Jakub@Work Sample All Computers Group",

                    formula);

     

                // Get the default management pack.

                ManagementPack defaultManagementPack =

                    localManagementGroup.GetManagementPacks(

                    "Microsoft.SystemCenter.OperationsManager.DefaultUser")[0];

     

                // Get the management packs for references

                ManagementPack windowsManagementPack = localManagementGroup.

                    GetManagementPack(SystemManagementPack.Windows);

                ManagementPack instanceGroupManagementPack = localManagementGroup.

                    GetManagementPack(SystemManagementPack.Group);

                ManagementPackReferenceCollection newReferences =

                    new ManagementPackReferenceCollection();

                newReferences.Add("Windows", windowsManagementPack);

                newReferences.Add("InstanceGroup", instanceGroupManagementPack);

     

                defaultManagementPack.InsertCustomMonitoringObjectGroup(allComputersGroup,

                    newReferences);

     

                // Get the class that represents my new group

                MonitoringClass myNewGroup = localManagementGroup.

                    GetMonitoringClasses("Jakub.At.Work.Namespace.AllComputerGroup")[0];

     

                // Get the discovery rule that populates this group

                // For the purposes of this sample, I know there is only 1 in the template

                MonitoringDiscovery groupPopulateDiscovery = myNewGroup.

                    GetMonitoringDiscoveries()[0];

     

                // This is the full configuration of the discovery of which the

                // membership rule is one part that you can configure and update

                Console.WriteLine("The discovery configuration: {0}",

                    groupPopulateDiscovery.DataSource.Configuration);

     

                // Update the configuration in some fasion

                string newConfiguration =

                    @"<RuleId>$MPElement$</RuleId>

                    <GroupInstanceId>$MPElement[Name=""Jakub.At.Work.Namespace.AllComputerGroup""]$</GroupInstanceId>

                    <MembershipRules>

                        <MembershipRule>

                    <MonitoringClass>$MPElement[Name=""Windows!Microsoft.Windows.Computer""]$</MonitoringClass>

                    <RelationshipClass>$MPElement[Name=""InstanceGroup!Microsoft.SystemCenter.InstanceGroupContainsEntities""]$</RelationshipClass>

                    <ExcludeList>

                        <MonitoringObjectId>f5E5F15E-3D7D-1839-F4C6-13E36BCD982a</MonitoringObjectId>

                    </ExcludeList>

                    </MembershipRule>

                    </MembershipRules>";

     

                // Now we want to update the group membership for this group

                groupPopulateDiscovery.Status = ManagementPackElementStatus.PendingUpdate;

                groupPopulateDiscovery.DataSource.Configuration = newConfiguration;

                groupPopulateDiscovery.GetManagementPack().AcceptChanges();

            }

        }

    }

     

  • Jakub@Work

    Command Shell

    • 0 Comments

    A co-worker of mine has started a blog on the SCOM Power Shell. The power shell is built on top of the SDK and provides command line functionality for many common administration tasks and much more. I've added the link to the list of SCOM blogs and also pasted it here for reference:

    System Center Operations Manager Command Shell

  • Jakub@Work

    Inserting Discovery Data

    • 28 Comments

    We've gone over how to drive state and insert operational data for existing entities, but how do you insert your own objects into the system? That's what this post will briefly touch on, as well as providing sample code (below) and a management pack to use (attached) with the code.

    Discovery data insertion via the SDK revolves around connectors as discovery sources. In order to insert data, you first need to create a connector with the system that all the data you insert will be associated with. This allows us to control the lifetime of the discovery data as a function of the lifetime of the connector.

    Once the connector is setup, you can use one of two modes for insertion; Snapshot or Incremental. Snapshot discovery indicates to the system that for this particular connector (read: discovery source), this is the definite snapshot of everything it has discovered. It will essentially delete anything that was previously discovered, and treat this snapshot as authoritative. Incremental, as the name would indicate, simply merges the existing discovery data with the discovery information provided in the incremental update. This can include additions, as well as deletions.

    Users can insert CustomMonitoringObjects and CustomMonitoringRelationshipObjects which, once inserted, map to MonitoringObjects and MonitoringRelationshipObjects. In order to insert either, you have to provide, at a minimum, the key values for objects, and the source and target for relationships. When dealing with a hosting relationship, the key values of the host must also be populated as part of the CustomMonitoringObject and no explicit CustomMonitoringRelationshipObject needs to be created. The example below should guide you through this.

    A quick discussion on managed vs. unmanaged instances. Our system will only run workflows against instances that are managed. The discovery process I talked about in the last post will insert "managed" data. Top-level instances (computers for instance) are inserted via the install agent APIs in the SDK and result in managed computers. It is also possible for rules to insert discovery data, however, this data will not be managed unless hosted by a managed instance.

    In order to be able to target workflows to your newly created instances and have them actually run, DiscoveryDataIsManaged needs to be set to true on the ConnectorInfo object when creating the connector. Alternatively, if you insert an instance as hosted by a managed instance, that instance will also be managed. For the former case, all workflows would run on the primary management server, while the latter would have them all running on the health service that manages the host. If something is not managed, you can still insert events and performance data about it, although the workflow that collects these will need to targeted against something other than the class of the instance. State change information would not be available for non-managed instances.

    using System;

    using Microsoft.EnterpriseManagement;

    using Microsoft.EnterpriseManagement.Configuration;

    using Microsoft.EnterpriseManagement.ConnectorFramework;

    using Microsoft.EnterpriseManagement.Monitoring;

     

    namespace Jakub_WorkSamples

    {

        partial class Program

        {

            static void InsertDiscoveryData()

            {

                // Connect to the sdk service on the local machine

                ManagementGroup localManagementGroup = new ManagementGroup("jakubo-test");

     

                // Get the connnector framework administration object

                ConnectorFrameworkAdministration connectorFrameworkAdministration =

                    localManagementGroup.GetConnectorFrameworkAdministration();

     

                // Create a connector

                ConnectorInfo info = new ConnectorInfo();

                info.Name = "TestConnector";

                info.DisplayName = "Test connector for discovery data";

                MonitoringConnector connector = connectorFrameworkAdministration.Setup(info);

     

                // First create an instance of SampleClassiHostedByComputer and

                // SampleClass2HostedBySampleClass1   

                // Find a computer

                MonitoringObject computer = localManagementGroup.GetMonitoringObjects(

                    localManagementGroup.GetMonitoringClass(

                    SystemMonitoringClass.WindowsComputer))[0];

     

                // Get the SampleClassiHostedByComputer class

                MonitoringClass sampleClass1HostedByComputer =

                    localManagementGroup.GetMonitoringClasses(

                    "SampleClass1HostedByComputer")[0];

     

                // Get the SampleClass2HostedBySampleClass1 class

                MonitoringClass sampleClass2HostedBysampleClass1 =

                    localManagementGroup.GetMonitoringClasses(

                    "SampleClass2HostedBySampleClass1")[0];

     

                // Get the key properties for each

                MonitoringClassProperty keyPropertyForSampleClass1 =

                    (MonitoringClassProperty)sampleClass1HostedByComputer.

                    PropertyCollection["KeyProperty1"];

                MonitoringClassProperty keyPropertyForSampleClass2 =

                    (MonitoringClassProperty)sampleClass2HostedBysampleClass1.

                    PropertyCollection["KeyProperty1SecondClass"];

     

                // Create the CustomMonitoringObjects to represent the new instances

                CustomMonitoringObject sampleClass1HostedByComputerInstance =

                    new CustomMonitoringObject(sampleClass1HostedByComputer);

                CustomMonitoringObject sampleClass2HostedBysampleClass1Instance =

                    new CustomMonitoringObject(sampleClass2HostedBysampleClass1);

     

                // Set the key property value for the first instance

                sampleClass1HostedByComputerInstance.SetMonitoringPropertyValue(

                    keyPropertyForSampleClass1, "MySampleInstance1");

     

                // Set the key property values for the second instance, which includes the

                // key property values of the host in order to populate the hosting relationship

                sampleClass2HostedBysampleClass1Instance.SetMonitoringPropertyValue(

                    keyPropertyForSampleClass1, "MySampleInstance1");

                sampleClass2HostedBysampleClass1Instance.SetMonitoringPropertyValue(

                    keyPropertyForSampleClass2, "MySampleInstance2");

     

                // In order to populate the hosting relationship, you need to also set

                // the key properties of the host. This will automatically create the hosting

                // relationship and is in fact the only way to create one programmatically.

                foreach (MonitoringClassProperty property in computer.GetMonitoringProperties())

                {

                    if (property.Key)

                    {

                        sampleClass1HostedByComputerInstance.SetMonitoringPropertyValue(

                            property, computer.GetMonitoringPropertyValue(property));

     

                        // Even though the relationship between

                        // sampleClass1HostedByComputerInstance and the computer is already

                        // defined, we need to add this key property to

                        // sampleClass2HostedBysampleClass1Instance as the entire hosting

                        // hierarchy is what uniquely identifies the instance. Without this,

                        // we wouldn't know "where" this instance exists and where it should be

                        // managed.

                        sampleClass2HostedBysampleClass1Instance.SetMonitoringPropertyValue(

                            property, computer.GetMonitoringPropertyValue(property));

                    }

                }

     

                // Let's insert what we have so far

                // We'll use Snapshot discovery to indicate this is a full snapshot of

                // all the discovery data this connector is aware of.

                SnapshotMonitoringDiscoveryData snapshot = new SnapshotMonitoringDiscoveryData();

                snapshot.Include(sampleClass1HostedByComputerInstance);

                snapshot.Include(sampleClass2HostedBysampleClass1Instance);

                snapshot.Commit(connector);

     

                // Let's retrieve the objects and ensure they were created

                MonitoringObject sampleClass1HostedByComputerMonitoringObject =

                    localManagementGroup.GetMonitoringObject(

                    sampleClass1HostedByComputerInstance.Id.Value);

                MonitoringObject sampleClass2HostedBySampleClass1MonitoringObject =

                    localManagementGroup.GetMonitoringObject(

                    sampleClass2HostedBysampleClass1Instance.Id.Value);

     

                // Now we create a relationship that isn't hosting

                MonitoringRelationshipClass computerContainsSampleClass2 =

                    localManagementGroup.GetMonitoringRelationshipClasses(

                    "ComputerContainsSampleClass2")[0];

     

                // Create the custom relationship object

                CustomMonitoringRelationshipObject customRelationship =

                    new CustomMonitoringRelationshipObject(computerContainsSampleClass2);

     

                // Do an incremental update to add the relationship.

                IncrementalMonitoringDiscoveryData incrementalAdd =

                    new IncrementalMonitoringDiscoveryData();

                customRelationship.SetSource(computer);

                customRelationship.SetTarget(sampleClass2HostedBySampleClass1MonitoringObject);

                incrementalAdd.Add(customRelationship);

                incrementalAdd.Commit(connector);

     

                // Make sure the relationship was inserted

                MonitoringRelationshipObject relationshipObject =

                    localManagementGroup.GetMonitoringRelationshipObject(

                    customRelationship.Id.Value);

     

                // Cleanup the connector. This should remove all the discovery data.

                connectorFrameworkAdministration.Cleanup(connector);

            }

        }

    }

     

Page 1 of 1 (4 items)