System Center WebLog by Russ Slaten : SCCM 2007

URL Monitoring Part III – The Monitors

rslaten — Wed, 08 Jul 2009 00:22:00 GMT

This entry is part of the following series:
URL Monitoring Part I – The Design
URL Monitoring Part II – The Classes and Discoveries
URL Monitoring Part III – The Monitors

Creating the monitors was by far the most complex part of this solution because monitors do not use data sources directly. Instead, they use Monitor Types which in turn use data sources. This is because data sources don’t understand the difference between what is considered healthy and what isn’t.

Again, I will use the System Center Configuration Manager Management Point monitor for this example since the other scenarios are built in the same way.

Step 1 – Creating the web page up/down data source. This custom data source is generic and can be used for any URL but its primary function is to return whether or not a web page returns with or without error.

I use the System.Scheduler and Microsoft.SystemCenter.WebApplication.UrlProbe modules so this MP does take a dependency on the Web Application Library MP. Make sure this is added as a reference under File\Management Pack Properties before continuing.

The System.Scheduler module is used for kicking off the URL Probe on a dynamic interval which in my example ends up being every 15 seconds. This is a parameter that is passed at the monitor level.

The Microsoft.SystemCenter.WebApplication.URLProbe module is very complex so I pasted the configuration below:

<ProbeAction ID="Probe" TypeID="MicrosoftSystemCenterWebApplicationLibrary!Microsoft.SystemCenter.WebApplication.UrlProbe">
<Proxy />
<ProxyUserName />
<ProxyPassword />
<ProxyAuthenticationScheme>None</ProxyAuthenticationScheme>
<CredentialUserName />
<CredentialPassword />
<AuthenticationScheme>None</AuthenticationScheme>
<FollowRedirects>true</FollowRedirects>
<RetryCount>0</RetryCount>
<RequestTimeout>0</RequestTimeout>
<Requests>
    <Request>
      <RequestID>1</RequestID>
      <URL>$Config/URLPath$</URL>
      <Verb>GET</Verb>
      <Version>HTTP/1.1</Version>
      <HttpHeaders>
        <HttpHeader>
          <Name>User-Agent</Name>
          <Value>Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)</Value>
        </HttpHeader>
      </HttpHeaders>
      <Body />
      <CheckContentChange>false</CheckContentChange>
      <ContentHash>00000000-0000-0000-0000-000000000000</ContentHash>
      <Depth>0</Depth>
      <ThinkTime>0</ThinkTime>
      <CheckInternalLinks>false</CheckInternalLinks>
      <CheckExternalLinks>false</CheckExternalLinks>
      <CheckResources>false</CheckResources>
      <RequestEvaluationCriteria>
        <StopProcessingIfWarningCriteriaIsMet>false</StopProcessingIfWarningCriteriaIsMet>
        <StopProcessingIfErrorCriteriaIsMet>false</StopProcessingIfErrorCriteriaIsMet>
        <BasePageEvaluationCriteria>
          <WarningCriteria />
          <ErrorCriteria>
            <NumericCriteriaExpressions>
              <NumericCriteriaExpression>
                <NumericRequestMetric>BasePageData/TotalResponseTime</NumericRequestMetric>
                <Operator>Greater</Operator>
                <Value>$Config/ResponseTime$</Value>
              </NumericCriteriaExpression>
            </NumericCriteriaExpressions>
          </ErrorCriteria>
        </BasePageEvaluationCriteria>
        <LinksEvaluationCriteria>
          <WarningCriteria>
            <StatusCodeCriteria>
              <ListNumericRequestMetric>StatusCode</ListNumericRequestMetric>
              <Operator>GreaterEqual</Operator>
              <Value>400</Value>
            </StatusCodeCriteria>
          </WarningCriteria>
          <ErrorCriteria />
        </LinksEvaluationCriteria>
        <ResourcesEvaluationCriteria>
          <WarningCriteria>
            <StatusCodeCriteria>
              <ListNumericRequestMetric>StatusCode</ListNumericRequestMetric>
              <Operator>GreaterEqual</Operator>
              <Value>400</Value>
            </StatusCodeCriteria>
          </WarningCriteria>
          <ErrorCriteria />
        </ResourcesEvaluationCriteria>
        <WebPageTotalEvaluationCriteria>
          <WarningCriteria />
          <ErrorCriteria />
        </WebPageTotalEvaluationCriteria>
        <DepthEvaluationCriteria>
          <WarningCriteria />
          <ErrorCriteria />
        </DepthEvaluationCriteria>
      </RequestEvaluationCriteria>
      <FormsAuthCredentials />
    </Request>
</Requests>
</ProbeAction>

This is passed to another data source in the Web Application Library which uses internal libraries to do the actual HTTP request.

The Configuration Schema tab contains the parameters which are passed to this data source. Currently it only accepts the URL and the interval at which the URL is to be watched.

This Data Source should output WebApplicationData.

Step 2 – Creating the Monitor Type. The Monitor Type takes parameters from the Monitor and passes them onto the data source. It also evaluates the data source to help the monitor determine what criteria equals healthy versus unhealthy.

For States just enter something that makes sense for the type of monitor you will be using.

The Member Modules start with the data source we just created. The Condition Detections are used to evaluate the results of the data source and the Probe Action is used to recalculate the state of the monitor manually.

The configuration of the data source module is fairly simple in that we are accepting parameters from the monitor (which we haven’t created yet) and are just re-passing them to the data source.

The first expression filter evaluates the web page as down.

The second expression filter evaluates the web page as up.

The OnDemandReset Probe Action does not contain any custom configuration information.

Next we have to specify what order the modules will be executed for the regular detection when there is a failure.

Next we have to specify what order the modules will be executed for the regular detection when there is not a failure.

For On Demand we just use the ErrorCodeOK state.

The Configuration Schema defines what parameters this Monitor Type accepts.

Step 3 – Create the Monitor. The monitor is a custom monitor which uses the monitor type we just created. It is also targeted to the SCCMMP class which we created in Part II of this series.

The Configuration is how often we want to query the URL and the URL itself. In this case we want the URL to be the registry value(s) that we discovered in the SCCMMP class. This is how one monitor can be used to watch thousands of websites.

We must tell the monitor which state means Critical and which means Healthy.

I set the example monitor to not alert but this can obviously be configured. Also, in the final solution for my customer we sealed the data sources and monitor types and used a different MP for the classes, discoveries, relationships, and monitors.

To test out this solution the MP you just created can be imported and if the agent from Part II still exists the discovered inventory should look like this:

I don’t have a website with this name so it shows as critical. This is what Health Explorer looks like:

If you want to do some more testing add some real URLs to the SCCMMP registry key and add that key to some more watcher nodes. Also, open the MP attached to Part I and look at the other classes and data sources to understand how the text parsing and response time solutions work.

If you have any questions about this series please let me know.

URL Monitoring Part II – The Classes and Discoveries

rslaten — Wed, 08 Jul 2009 00:14:00 GMT

This entry is part of the following series:
URL Monitoring Part I – The Design
URL Monitoring Part II – The Classes and Discoveries
URL Monitoring Part III – The Monitors

Each of the 3 web applications has its own class. The applicable discovery queries a registry key and based on the values in that key creates an instance of the proper web application’s class. Also, each web application can contain multiple URLs if necessary.

I will use the class I created to monitor System Center Configuration Manager Management Points for this example because they are all built the same way.

Step 1 - The first thing I did was create a new custom class in the authoring console. The base class is System.Perspective. Notice that one of the attributes is “Hosted”. Do not worry about checking this yet because once we add a relationship it will be checked automatically. Also, I always make what I create “Public”. This is important if you decide to seal the MP and want to access the class from a separate management pack.

I added an attribute called “URL” and made it a key value. When the instance of this class is created the “URL” attribute will contain the URL we will be monitoring.

Step 2 – Next I created a relationship so that the URL we are monitoring is tied to the Watcher Node. I made this a hosting relationship and made it public.

Step 3 – I then created a new custom data source that the discovery will end up using to create instances of the SCCM MP Class in addition to the other classes I use in the solution.

I use the System.Scheduler and Microsoft.Windows.ScriptDiscoveryProbe modules in this data source

The System.Scheduler module takes a parameter which is passed to the data source by the discovery itself. This parameter is the interval at which the discovery should be run in seconds.

The Microsoft.Windows.ScriptDiscoveryProbe module is used to run a script that returns discovery data.

The Microsoft.Windows.ScriptDiscoveryProbe module is a bit more complex so I will paste the configuration below:

<ProbeAction ID="DiscoveryProbe" TypeID="Windows!Microsoft.Windows.ScriptDiscoveryProbe">
<ScriptName>WatcherDiscovery.vbs</ScriptName>
<Arguments>$Config/FQDN$ $Config/SourceID$ $Config/ManagedEntityID$ $Config/Class$ $Config/Property$ $Config/URLRegKey$</Arguments>
<ScriptBody><![CDATA['Arg1 = FQDN of Watcher Node
'Arg2 = SourceID (MPElement)
'Arg3 = ManagedEntityID (Target/Id)
'Arg4 = Class ID
'Arg5 = Property ID
'Arg6 = URL Watcher reg key off hklm

Set oArgs = Wscript.Arguments
sFQDN = UCase(oArgs(0))
sSourceID = oArgs(1)
sManagedEntityID = oArgs(2)
sClassID = oArgs(3)
sPropertyID = oArgs(4)
sURLRegKey = oArgs(5)

Set oAPI = CreateObject("MOM.ScriptAPI")
Set oDiscoveryData = oAPI.CreateDiscoveryData(0,sSourceID, sManagedEntityID)

'Determine if this is a Watcher Node
const HKEY_LOCAL_MACHINE = &H80000002
bIsWatcher = False

Set oReg = GetObject("winmgmts:root\default:StdRegProv")
oReg.EnumValues HKEY_LOCAL_MACHINE, sURLRegKey, arrValues
If isarray(arrValues) Then
bIsWatcher = True
End If

If bIsWatcher Then
   For Each sValue in arrValues
      oReg.GetStringValue HKEY_LOCAL_MACHINE, sURLRegKey, sValue, sURL
      Set oWatcher = oDiscoveryData.CreateClassInstance(sClassID)
      oWatcher.AddProperty "$MPElement[Name='Windows!Microsoft.Windows.Computer']/PrincipalName$",sFQDN
      oWatcher.AddProperty sPropertyID,sURL
      oWatcher.AddProperty "$MPElement[Name='System!System.Entity']/DisplayName$",sURL
      oDiscoveryData.AddInstance(oWatcher)
   Next
End If
Call oAPI.Return(oDiscoveryData)]]></ScriptBody>
<TimeoutSeconds>240</TimeoutSeconds>
</ProbeAction>

As you can see this is a vbscript that takes 5 arguments. The script queries a registry key and if the key is found returns discovery data that is used by the discovery to create instances of the applicable class. This is the reason we only need to create a handful of monitors to watch thousands of URLs.

The Configuration Tab of the data source contains all the parameters that are passed into the data source by the discovery, which we will create next.

The Data Types tab should show that this data source outputs discovery data.

Step 4 – Now I need to create the discovery that will use this data source. I target the discovery to Microsoft.Windows.Computer because any computer can be a watcher. The way the data source identifies a watcher is by the registry key we pass it.

Next I added the discovered types. We are discovering instances of the SCCM MP Class so we just add that along with the appropriate attributes we will be populating.

Next I choose the data source we just created and plug in the appropriate parameters. Notice the interval is set to every 24 hours and the URLRegKey is HKLM by default – Software\URLWatcher\SCCMMP. So if the data source finds this registry key it will read the values underneath it. The value should contain a URL like “http://mySCCMMP/sms_mp/.sms_aut?mplist”.

That is it for the class and discovery, just 4 steps (class, relationship, data source, and discovery). To test this out add the following registry key and value to one of your OpsMgr agents and import the MP. Note, you can add more registry values under this key if you want that agent to watch more than one URL in this particular class (so more than one SCCM MP).

After importing the MP you just created you should see new discovered, but not monitored, inventory of type “SCCM MP Class”. Notice that the “Path” is actually the Watcher Node while the Display Name is the URL.

Custom URL Monitoring Part I – The Design

rslaten — Wed, 08 Jul 2009 00:11:00 GMT

The out of the box Web Application template that exists in OpsMgr 2007 SP1 and R2 is useful but there are situations where it just doesn’t fit.

Consider the following scenario: The Contoso Corporation’s global IT headquarters is located in Richardson, TX. They have over 2,000 retail stores spread out across the world and over slow connections. Each of these retail stores have two local servers. One of these servers is a web server that hosts 3 web applications which are used for internal tools critical to the operation of the retail store. The CIO wants the main page of SCCM Management Point site to be checked every 15 seconds to ensure the site is up. She wants the POS Web Application site to be checked every 15 seconds to ensure it is responding within 500ms. Finally, she also wants Store Inventory site to be checked for the text “store inventory web application” every 15 seconds to make sure the site is displaying the appropriate information. Because of the remote nature of these retail stores the CIO wants the check to be done from the non-web server at each retail store where the websites are going to be viewed.

The out-of-the-box Web Application template in OpsMgr doesn’t fit well in this scenario because we need to monitor 6,000 websites every 15 seconds from 2,000 different watcher nodes. One of my customers has a very similar scenario in their environment and asked for a custom solution that uses OpsMgr. I created a solution that does these checks every 15 seconds from the local watcher node at each retail store. The cool part about this solution is we only use 3 monitors to watch all 6,000 web sites, it is dynamic so once the solution is in place there is very little additional administrative overhead, and it scales incredibly well.

Because of the complexity of this management pack I decided to split this topic up into multiple blog entries. The primary purpose of this series is to offer an alternative to the Web Application template in OpsMgr and the secondary purpose is to show how I developed this solution. The series is broken up into the following posts:

Custom URL Monitoring Part I – The Design
Custom URL Monitoring Part II – The Classes and Discoveries
Custom URL Monitoring Part III – The Monitors

My design is generic and was designed for OpsMgr 2007 SP1 although it should also work in R2. It consists of 3 discoveries, 3 classes, 3 monitors, 3 monitor types, and 4 data sources. I attached the entire solution to this blog entry.

Here is a diagram of the design of this MP:

To test this MP in the lab do the following:

1. Create a test website with the text “store inventory web application” on the main page
2. On several OpsMgr agents (which would act as watcher nodes) create the following registry entries:
    - HKLM\Software\URLWatcher\SCCMMP (Key)
      - HKLM\Software\URLWatcher\SCCMMP\SCCMMP:http://<yourwebsite> (String Value)
    - HKLM\Software\URLWatcher\POS (Key)
      - HKLM\Software\URLWatcher\POS\POS:http://<yourwebsite> (String Value)
    - HKLM\Software\URLWatcher\StoreInv (Key)
      - HKLM\Software\URLWatcher\StoreInv\StoreInv:http://<yourwebsite> (String Value)
3. Import the MP and wait 5 minutes
4. Under Monitoring\Discovered Inventory look at the 3 classes in the Website Watcher Library MP
5. Change the text and bring the website up/down to view the results

New System Center Certifications

rslaten — Sat, 13 Oct 2007 00:35:00 GMT

There will be a Live Meeting on October 24^th going over the new certifications for SCCM and SCOM. I will be adding these new certs to my list of things to do. See Trika's blog on the subject and for details on the Live Meeting.

The Mystery of the Lost Hardware Inventory

rslaten — Tue, 18 Sep 2007 20:37:00 GMT

In large environments, with tens of thousands of SMS 2003 clients, inventory may get "lost" for various reasons. An example of a lost inventory is below:

An SMS 2003 Advanced Client reports a full inventory upon install
The client continues to report delta inventory of changes
The client reports a delta inventory but that inventory gets "lost"
The client continues to report delta inventory but the changes in the "lost" inventory never appear in the database
The client gets resync'd, sending up a full inventory, and the changes that were "lost" now appear

Clients don't get resync'd under normal conditions therefore that inventory may not ever appear in the database. This can become an issue for inventory items such as security updates (staying applicable in the database but not applicable on the client). System Center Configuration Manager 2007 contains architectural changes which prevent this scenario from occurring. Specifically SCCM 2007 serializes software and hardware inventory reporting so that missing reports, which are based on serial numbers instead of timestamps, trigger a re-synchronization of inventory at the client.

So if you don't plan on upgrading to SCCM 2007 in the near future how might you prevent inventory from being "lost"? I documented some steps that should assist with detecting, and possibly resolving, the rare scenario when clients send up inventory that somehow disappears before getting loaded into the database. These steps are simply an example of how one might do this and the scripts are not tested and do not contain any type of error handling, logging, or reporting.

Step # 1 – Modify the SMS_DEF.MOF

Modify the Win32_OperatingSystem section of the MOF to report "LocalDateTime". This is required because the "Workstation Status" section shows "Last Hardware Scan" but there isn't a way to keep the history of this data therefore we need a consistent time we can key in on that writes to a history table.
Create a new class in the SMS_DEF.MOF to pick up a new WMI class which we'll create a populate later:

//SMSInventoryHistory
[SMS_Report(TRUE),
SMS_Group_Name("SMS Inventory"),
SMS_Class_ID("MICROSOFT|SMS_INVENTORY|1.0"),
Namespace("\\\\\\\\.\\\\root\\\\SMSINVENTORY")]
class InvHistory: SMS_Class_Template
{
[SMS_Report(TRUE),key]
datetime   ScanTime;
[SMS_Report(TRUE)]
uint32   KeyIncrement;
[SMS_Report(TRUE)]
string   ScanType;
};
Notice the KeyIncrement field. This is there because we want to report every instance of this class for every inventory. It is for this reason that we don't need history enabled for this class so it can be disabled by using KB836432 (This still applies to SMS 2003 and likely SCCM 2007).

Step # 2 – Create a recurring advertisement, which should probably run at least twice as often as hardware inventory (if not more), which runs client.vbs

Creates a new "SMSInventory" class under "root" if it doesn't exist
Gets the last hardware scan from root\ccm\invagt:InventoryActionStatus
Creates a new instance of InvHistory in the SMSInventory namespace if the hardware scan is new
If the inventory is already recorded then update the "KeyIncrement" field so SMS reports this in every delta
Cleans up any instances older than 14 days by default to keep the delta inventory from growing too large

'client.vbs
'rslaten
'08/15/2007

'Determines when to clean out old inventory instances
iDays = 14

'Connect to WMI locally
Set oLocator = CreateObject("WbemScripting.sWbemLocator")
Set oSvc = oLocator.ConnectServer(".", "root")

'Create a custom namespace to hold data if it doesn't already exist
Set oNamespaces = oSvc.InstancesOf("__namespace")
bFound = False
For each oNamespace in oNamespaces
    If oNamespace.Name = "SMSInventory" Then
        bFound = True
        Exit For
    End If
Next

If Not bFound Then
    Set oNamespace = oSvc.Get("__namespace")
    Set oCustomNameSpace = oNamespace.SpawnInstance_
    oCustomNamespace.name = "SMSInventory"
    oCustomNamespace.Put_()

    'Create a custom class to hold data
    wbemCimtypeUint32 = 19
    wbemCimtypeDatetime = 101
    wbemCimtypeString = 8
    Set oWMI = GetObject("winmgmts:root\SMSInventory")
    Set oClass = oWMI.Get()
    oClass.Path_.Class = "InvHistory"
    oClass.Properties_.add "ScanTime", wbemCimtypeDatetime
    oClass.Properties_("ScanTime").Qualifiers_.add "key", true
    oClass.Properties_.add "KeyIncrement", wbemCimtypeUint32
    oClass.Properties_.add "ScanType", wbemCimtypeString
    oClass.Put_()
End If

'Get timezone
Set oSvc = oLocator.ConnectServer(".", "root\cimv2")
Set collTimeZone = oSvc.ExecQuery("select Bias, DaylightBias from Win32_TimeZone")
For Each oTimeZone in collTimeZone
    iTimeZone = oTimeZone.Bias - oTimeZone.DaylightBias
Next

'Get last Hinv report date and convert to local time
Set oSvc = oLocator.ConnectServer(".", "root\ccm\invagt")
HActionID = """{00000000-0000-0000-0000-000000000001}"""
Set oHinv = oSvc.Get("InventoryActionStatus.InventoryActionID=" & _
    HActionID)
Const CONVERT_TO_LOCAL_TIME = true
'dLastReportDate = ConvertToGMT(oHinv.LastReportDate)
dLastReportDate = oHinv.LastReportDate
dLastReportDate = WMIDateStringToDate(dLastReportDate)
Set dNewTime = CreateObject("WbemScripting.SWbemDateTime")
dNewTime.SetVarDate dLastReportDate, CONVERT_TO_LOCAL_TIME
dLastReportDate = dNewTime.GetVarDate(CONVERT_TO_LOCAL_TIME)
dLastReportDate = DateAdd("n", iTimeZone, dLastReportDate)
dLastReportDate = GetWMIDate(dLastReportDate, "+000")

'Set last Hinv report date
Set oSvc = oLocator.ConnectServer(".", "root\SMSInventory")
Set oHinvHistory = oSvc.InstancesOf("InvHistory")

'First instance in class
If oHinvHistory.Count = 0 Then
    Set oHinv = oSvc.Get("InvHistory").SpawnInstance_
    oHinv.ScanTime = dLastReportDate
    oHinv.KeyIncrement = 0
    oHinv.ScanType = "Hardware"
    oHinv.Put_()
Else
    bFound = False
    
    'Calculate date to clean up old instances
    Set dExpire = CreateObject("WbemScripting.SWbemDateTime")
    dExpire.SetVarDate Now, CONVERT_TO_LOCAL_TIME
    dRegular = dExpire.GetVarDate(CONVERT_TO_LOCAL_TIME)
    dExpireDate = DateAdd("d", -iDays, dRegular)
    dExpire.SetVarDate dExpireDate, CONVERT_TO_LOCAL_TIME
    
    'Updating all instances
    For each oInstance in oHinvHistory
        If dLastReportDate = oInstance.ScanTime Then
            bFound = True
        End If
        
        'Clean up old instances
        If oInstance.ScanTime < dExpire Then
            oInstance.Delete_
        Else
            'Increment the KeyIncrement property to force SMS to pick
            'up the instance during inventory
            oInstance.KeyIncrement = oInstance.KeyIncrement + 1
            oInstance.Put_
        End If
    Next
    
    'Create a new instance if needed
    If Not bFound Then
        Set oHinv = oSvc.Get("InvHistory").SpawnInstance_
        oHinv.ScanTime = dLastReportDate
        oHinv.KeyIncrement = 0
        oHinv.ScanType = "Hardware"
        oHinv.Put_()    
    End If
End If

Function ConvertToGMT(dDate)
    ConvertToGMT = Replace(dDate, "-000", iTimeZone)
End Function

Function WMIDateStringToDate(dtmInstallDate)
     WMIDateStringToDate = CDate(Mid(dtmInstallDate, 5, 2) & "/" & _
     Mid(dtmInstallDate, 7, 2) & "/" & Left(dtmInstallDate, 4) _
     & " " & Mid (dtmInstallDate, 9, 2) & ":" & _
     Mid(dtmInstallDate, 11, 2) & ":" & Mid(dtmInstallDate, _
     13, 2))
End Function

Function GetWMIDate(vd,strOffset)
    On Error Resume Next
    GetWMIDate = Year(vd) & AddZero(Month(vd)) & AddZero(Day(vd)) & _
            AddZero(Hour(vd)) & AddZero(Minute(vd)) & _
            AddZero(Second(vd)) & ".000000" & strOffset
End Function

Function AddZero(pNum)
    On Error Resume Next
    If pNum <=9 then
        AddZero = "0" & pNum
    Else AddZero = pNum
    End If
End Function

Step # 3 – Periodically check to see if there is any missing hardware inventory by running server.vbs

Queries the SMS database for OS LocalDateTime and the custom SMSInventory ScanTime
Compares the ScanTime to LocalDateTime, searching for missing LocalDateTime entries. To find a match the LocalDateTime must be within 5 minutes from the ScanTime. If any clients take longer than 5 minutes to complete a hardware inventory cycle then this may need to be increased.
For any ScanTime without a matching LocalDateTime, we display the fact that there is a missing inventory.

'server.vbs
'rslaten
'08/15/2007

'Globals
Dim aClients(), iClients, iInstanceCount, iTotal

'Get SMS Namespace
sSMSNameSpace = GetSMSNameSpace()

'Connect to site server
Set oWMI = GetObject("winMgmts:" & sSMSNameSpace)

'Query clients and create and array of client objects
sQuery = "select distinct SMS_R_System.Name, SMS_R_System.SMSUniqueIdentifier," & _
    " SMS_G_System_SMS_INVENTORY.ScanTime, SMS_GH_System_OPERATING_SYSTEM.LocalDateTime" & _
    " from  SMS_R_System inner join SMS_G_System_SMS_INVENTORY on " & _
    "SMS_G_System_SMS_INVENTORY.ResourceID = SMS_R_System.ResourceId inner" & _
    " join SMS_GH_System_OPERATING_SYSTEM on " & _
    "SMS_GH_System_OPERATING_SYSTEM.ResourceID = SMS_R_System.ResourceId " & _
    "where SMS_G_System_SMS_INVENTORY.ScanTime is not NULL and " & _
    "SMS_GH_System_OPERATING_SYSTEM.LocalDateTime is not NULL"

Set aCollection = oWMI.ExecQuery(sQuery)
iTotal = aCollection.count
For Each oInstance In aCollection
    If sLastName <> oInstance.SMS_R_System.Name Then
        'Make sure this isn't the first iteration in the array
        If iInstanceCount <> 0 Then
            'Add to master array
            ReDim Preserve aClients(iClients)
            Set aClients(iClients) = oClient
            iClients = iClients + 1
        End If
        
        'Now create new client object
        Set oClient = New Client
        oClient.sClient = oInstance.SMS_R_System.Name
        oClient.sGUID = oInstance.SMS_R_System.SMSUniqueIdentifier
    End If
    
    If sLastScan <> oInstance.SMS_G_System_SMS_Inventory.ScanTime Then
        oClient.AddSMSInventoryScanDate(oInstance.SMS_G_System_SMS_Inventory.ScanTime)
    End If
    
    If sLastOSScan <> oInstance.SMS_GH_System_OPERATING_SYSTEM.LocalDateTime Then
        oClient.AddOSScanDate(oInstance.SMS_GH_System_OPERATING_SYSTEM.LocalDateTime)
    End If 

    sLastName = oInstance.SMS_R_System.Name
    sLastScan = oInstance.SMS_G_System_SMS_Inventory.ScanTime
    sLastOSScan = oInstance.SMS_GH_System_OPERATING_SYSTEM.LocalDateTime
    iInstanceCount = iInstanceCount + 1
    
    'If this is the last instance add it to the array
    If iInstanceCount = iTotal Then
        'Add to master array
        ReDim Preserve aClients(iClients)
        Set aClients(iClients) = oClient
        iClients = iClients + 1    
    End If
Next

'Loop through clients in memory to check for missing inventory
For each oClient in aClients
    aScans = oClient.aScans
    WScript.Echo "******************************************"
    WScript.Echo "Client = " & oClient.sClient
    WScript.Echo "GUID = " & oClient.sGUID
    For each oScan in oClient.aScans
        If not oScan.isOS Then
            Set oMatch = new Match
            oMatch.dScan = WMIDateStringToDate(oScan.dDate)
            oMatch.bMatch = false
            
            'Find a matching OS time (look for match within 5 minutes)
            For each oTime in aScans
                If oTime.isOS Then
                    If DateDiff("n", oMatch.dScan, _
                        WMIDateStringToDate(oTime.dDate)) < 5 and _
                        DateDiff ("n", oMatch.dScan, _
                        WMIDateStringToDate(oTime.dDate)) > -5 Then             
                        oMatch.dOS = WMIDateStringToDate(oTime.dDate)
                        oMatch.bMatch = true
                    End If
                End If
            Next
            
            'Make sure we found a match
            If not oMatch.bMatch Then
                WScript.Echo "*PROBLEM FOUND*"
                WScript.Echo "SMS Inventory Report = " & oMatch.dScan
                WScript.Echo "*NO MATCHING OS DATE FOUND*"
            Else
                WScript.Echo oMatch.dScan & " = " & oMatch.dOS
            End If
        End If
    Next
    WScript.Echo "******************************************"
Next

'Get regular date
Function WMIDateStringToDate(dtmInstallDate)
    WMIDateStringToDate = CDate(Mid(dtmInstallDate, 5, 2) & "/" & _
        Mid(dtmInstallDate, 7, 2) & "/" & Left(dtmInstallDate, 4) _
            & " " & Mid (dtmInstallDate, 9, 2) & ":" & _
                Mid(dtmInstallDate, 11, 2) & ":" & Mid(dtmInstallDate, _
                    13, 2))
End Function


'Gets SMS Namespace from local site server
Function GetSMSNameSpace()
    Set refWMI = GetObject("winMgmts:\root\sms")    
    Set colNameSpaceQuery = refWMI.ExecQuery("select * from SMS_ProviderLocation")
    For Each refitem in colNameSpaceQuery
        GetSMSNameSpace = refitem.NamespacePath
    Next
End Function

Class Client
    Public aScans()
    Private i
    Public sClient
    Public sGUID
    
    Public Sub AddOSScanDate(dScanDate)
        'Check for duplicates
        bFound = false
        For Each oScan in aScans
            If dScanDate = oScan.dDate And _
                oScan.isOS = true Then
                bFound = true
                Exit For
            End If
        Next
        
        If Not bFound Then
            Set oScan = New Scan
            oScan.dDate = dScanDate
            oScan.isOS = true
            ReDim Preserve aScans(i)
            Set aScans(i) = oScan
            i = i + 1
        End If
    End Sub
    
    Public Sub AddSMSInventoryScanDate(dScanDate)
        'Check for duplicates
        bFound = false
        For Each oScan in aScans
            If dScanDate = oScan.dDate And _
                oScan.isOS = false Then
                bFound = true
                Exit For
            End If
        Next
        
        If Not bFound Then        
            Set oScan = New Scan
            oScan.dDate = dScanDate
            oScan.isOS = false
            ReDim Preserve aScans(i)
            Set aScans(i) = oScan
            i = i + 1    
        End If
    End Sub
End Class

Class Scan
    Public dDate
    Public isOS
End Class

Class Match
    Public dScan
    Public dOS
    Public bMatch
End Class

Step # 4 – Force a resync on the clients that have "lost" inventory

The method to delete the local inventory cache is the known and documented method of doing this.
There may be a way to use the MP API to create a server side policy, but this would require significant work and a non-script (C++) application. Internally, as you can see with SQL profiler, we execute the sp_RC_InvResync stored procedure but as you know executing these directly isn't recommended or supported.

These examples do not contain error checking and need to be optimized for performance. The examples have also only been tested in a very small test environment using only the scenario documented at the top of this post. If this method is to be used in a production environment more code work and testing needs to be done, particularly on the server end, to improve the detection logic, add history to which clients resyncs have been sent to, and add reporting capabilities.

**Steps 3 and 4 might not be required. During testing I found a potentially unforeseen benefit of running the client.vbs script on clients with "lost" inventory. Since every instance of the InvHistory class is reported during every inventory, when an instance is listed as "Update" but isn't in the database, dataloader will automatically request a resync for that client. More testing of this finding needs to be done to verify that this will work, and if it does it should simplify detecting which clients have lost inventory**