Notes on System Center Operations Manager : Availability

Last contacted, better SQL query

MSutara — Mon, 28 Jul 2008 12:57:00 GMT

My post about how to receive approximate last contacted time spawn good feedback and thanks to Robin Drake for rewriting my SQL script without use of cursors while producing single grid. Following is his version:

use OperationsManager

declare @substract float

declare @numberOfMissing float

declare @interval float

-- Get the number of missing heartbeats

select @numberOfMissing = SettingValue from GlobalSettings GS

join ManagedTypeProperty MTP with(nolock) on GS.ManagedTypePropertyId = MTP.ManagedTypePropertyId

where MTP.ManagedTypePropertyName = 'NumberOfMissingHeartBeatsToMarkMachineDown'

-- Get the heartbeat interval

select @interval = SettingValue from GlobalSettings GS

join ManagedTypeProperty MTP with(nolock) on GS.ManagedTypePropertyId = MTP.ManagedTypePropertyId

where MTP.ManagedTypePropertyName = 'HeartbeatInterval'

-- Calculate the amount of lapsed time before a system is marked as non contactable

select @substract = (@numberOfMissing * @interval)/100000

select B.DisplayName, AH.TimeStarted, (cast((cast(tmp.MaxTimeStarted as float)- @substract) as datetime)) as [ApproxLastContactedTime (UTC)],

dateadd(hh, +9, (cast((cast(tmp.MaxTimeStarted as float)- @substract) as datetime))) as 'ApproxLastContactedTime (Pacific)' from Availability A

join BaseManagedEntity B with(nolock) on B.BaseManagedEntityId = A.BaseManagedEntityId

join AvailabilityHistory AH with(nolock) on AH.BaseManagedEntityId = A.BaseManagedEntityId

join

(

select MAX(AHTMP.TimeStarted) AS MaxTimeStarted, BME.BaseManagedEntityId from AvailabilityHistory AHTMP

join BaseManagedEntity BME with(nolock) on BME.BaseManagedEntityId = AHTMP.BaseManagedEntityId

where BME.IsDeleted = 0

group by BME.BaseManagedEntityId

)

TMP on AH.TimeStarted = MaxTimeStarted

where A.IsAvailable = 0 and B.IsDeleted = 0

Robin, Thanks again!

“Net view” as diagnostic used when heartbeat is missing

MSutara — Fri, 25 Jul 2008 01:25:00 GMT

This all started as question about how “Computer Not Reachable” works. It has been asked in newsgroup and I thought I will help and provide some inside. First, I would like to repeat, that in my opinion regarding the fact that computer not reachable detection is based on the fact that heart beat is missing while targeting “Health Service Watcher” is at least questionable and not very fortunate (it is almost like we try to find causality without root causality engine). Due to performance limitations (Main reasons lying in the fact we did not want to end up having as many workflows doing ICM Ping (which is from OpsMgr perspective a blocking operation) as many computers are present within topology, or not having efficient targeting story for such computer not reachable recognition) we now have this “nasty beast” where we execute diagnostic on heartbeat monitor state change which is later supposed to (thru some set of recoveries) make a monitor state change for “Computer Not Reachable”. This all design was not very reliable in RTM and I had to introduce some mitigations and reliability improvements in SP1 (which off course was not too positive because now two alerts are raised instead of one which was raised in RTM, but believe me, you want reliability J).

Regardless of my runt, I decided to show how to add custom diagnostics for Heartbeat monitor and eventually answer a question and provide management pack which will set the state of some custom monitor based on recovery output. This all could be advance talk, please comment and/or ask question thru this blog if something requires further explanation (this I will not discuss on newsgroup, all this provided info is about undocumented functionality which may break in any future release!).

So to adding custom diagnostic is really easy. In this case all we need (monitor and target) is public and accessible outside of management pack where such elements are defined. At the end it is just adding following raw XML into custom management pack (assuming you have all MP references set properly).

<Diagnostic ID="Microsoft.SystemCenter.Community.Diagnostic.NetViewDiagnostic" Comment="In response to heartbeat failure, net view machine" Accessibility="Internal" Enabled="true" Target="SCLibrary!Microsoft.SystemCenter.HealthServiceWatcher" Monitor="SC2007!Microsoft.SystemCenter.HealthService.Heartbeat" ExecuteOnState="Error" Remotable="true" Timeout="300">

<Category>Maintenance</Category>

<ProbeAction ID="Command" TypeID="System!System.CommandExecuterProbe">

<ApplicationName><![CDATA[%windir%\System32\net.exe]]></ApplicationName>

<WorkingDirectory />

<CommandLine>view $Target/Property[Type="SCLibrary!Microsoft.SystemCenter.HealthServiceWatcher"]/HealthServiceName$</CommandLine>

<TimeoutSeconds>30</TimeoutSeconds>

<RequireOutput>true</RequireOutput>

</ProbeAction>

</Diagnostic>

Now acting based on the result is where real fun starts! First I need to mention that “Computer Not Reachable” is defined as internal, so is not accessible and new monitor must be defined rather then disable original thru override included with management pack. I will not provide much information about why it is “aggregate” monitor, the only thing I will say is that in OpsMgr, aggregate monitor is only monitor without a workflow and runtime is magically making all the state changes. In our case we are setting the state thru recovery, so we do not want to “WASTE” a workflow (and there would be as many of it as many computers within enterprise are monitored) if we never expect that workflow to set the state anyway. Also, there are some “magic” modules I WILL NOT DISCUSS (maybe ever, but we will see in next release), where those modules will set the state of the monitor. (There might be some of you willing to do some reverse engineering and you may get an idea how set state critical when result of recovery provides info about command “net view” failing, and how we set state when command succeeded though.) So here is the recap thru screenshots:

1. After import, monitor state is NEVER set until net view command is executed

2. When RMS recognizes that heart beat is missing, we execute “net view” command inside of diagnostic. When net view succeeds, we set state of monitor to “Healthy”, we set to “Critical” otherwise

Attached is management pack that provides this functionality. It may be used AS IS and confers no rights and support. Enjoy!

Last contacted

MSutara — Thu, 24 Jul 2008 20:13:00 GMT

Some customers not only want to know that health service hosted on the computer was not heart beating (remember, this is way different from our approach to recognize computer down, which BTW solely based on my own opinion, is rather unfortunate attempt to recognize something for what health service watcher was not originally designed), but they also would like to get at least some information about the last time such health service contacted its management server. This post is one possible solution that can be used.

This information is already present in health explorer (in some sort of form as we will see later), but is not as friendly to locate and requires “big” TCO. I will mention how one can do that In SP1 version of OpsMgr2007 anyway:

1. One need to open “Health Explorer” form health service watcher that was marked critical. (Health service watcher views are located inside of the folder “Operations Manager” subfolder “Agent” for those of you who never had to wonder there.)

2. Locating “Health Service Heartbeat Failure” monitor and exploring “State Change Events” tab is next.

3. Context of the top state change to critical carries data type that caused state change and its “Date and Time” carries value when runtime recognized that heart beat is missing

Following is screenshot from next version of OpsMgr. We did some improvements in unavailability recognition and changed internal plumbing for some of the “Health Service Watcher” monitors (that is outside of the scope of this post and I may do another one describing the changes once release date approaches). It display the fact that data type used still contains same information about when runtime recognized that health service was not heart beating and that such information is present inside of “Date and Time” within context of the state change:

So I just proved that this is highly ineffective to do when multiple health services are not heart beating and one wants to have a quick view with information when heartbeat miss was recognized and what was possibly last time given health service contacted its server.

Before we do this, I need to explain how availability is stored in our Operational Database a little bit. There is a table “Availability”. One of the columns for this table is “LastModified”. Value is equal to the time when runtime notified SDK service about availability changes. That is not the time when runtime was last contacted by health service though. Last contacted time can be calculated based on heart beat interval and how many heartbeats should be missed prior notifying SDK about the fact that heartbeat was missing. Values for interval and number of missies are stored within global settings. And that gives us opportunity to create following SQL script:

use OperationsManager

declare @substract float

declare @numberOfMissing float

declare @interval float

select @numberOfMissing = SettingValue from GlobalSettings GS

join ManagedTypeProperty MTP with(nolock) on GS.ManagedTypePropertyId = MTP.ManagedTypePropertyId

where MTP.ManagedTypePropertyName = 'NumberOfMissingHeartBeatsToMarkMachineDown'

select @interval = SettingValue from GlobalSettings GS

join ManagedTypeProperty MTP with(nolock) on GS.ManagedTypePropertyId = MTP.ManagedTypePropertyId

where MTP.ManagedTypePropertyName = 'HeartbeatInterval'

select @substract = (@numberOfMissing * @interval)/100000

declare availCursor cursor

for

select B.DisplayName, AH.TimeStarted from Availability A

join BaseManagedEntity B with(nolock) on B.BaseManagedEntityId = A.BaseManagedEntityId

join AvailabilityHistory AH with(nolock) on AH.BaseManagedEntityId = A.BaseManagedEntityId

join (

select MAX(AHTMP.TimeStarted) AS MaxTimeStarted, BME.BaseManagedEntityId

from AvailabilityHistory AHTMP

join BaseManagedEntity BME with(nolock) on BME.BaseManagedEntityId = AHTMP.BaseManagedEntityId

where BME.IsDeleted = 0

group by BME.BaseManagedEntityId

) TMP on AH.TimeStarted = MaxTimeStarted

where A.IsAvailable = 0 and B.IsDeleted = 0

open availCursor

declare @name nvarchar(255)

declare @time datetime

fetch next from availCursor into @name, @time

while @@FETCH_STATUS = 0

begin

declare @approxTime datetime

select @approxTime = (cast((cast(@time as float)- @substract) as datetime))

select @name as Name, @time as Recognized, @approxTime as 'ApproxLastContactedTime (UTC)', dateadd(hh, -7, @approxTime) as 'ApproxLastContactedTime (Pacific)'

fetch next from availCursor into @name, @time

end

close availCursor

deallocate availCursor

Based on the result and comparing with health explorer data, we can see that recognized is “equal” and approximate last contacted is calculated (by default it should be around 3 minutes before recognition). Maybe I will create a report in the future which will try to display this information in more unified manner, but that is not my intent right now ...