This posting is provided "AS IS" with no warranties, and confers no rights. Use of included script samples are subject to the terms specified at http://www.microsoft.com/info/cpyright.htm

This walkthrough will concentrate on mitigating some of the security limitations of MOM 2005 when managing machines, which are part of a workgroup environment, or to be more specific which are not part of an Active Directory Forest.

We will look at how to utilize PKI infrastructure in conjunction with IPSec capabilities of the Windows platform to perform mutual authentication based on X509 certificates.

Please, note that this walkthrough is only applicable to the MOM 2005 environment, since SCOM 2007 has a built-in mechanism to utilize X509 certificates to provide mutual authentication in a workgroup environment.

For details on how to configure SCOM 2007 to perform mutual authentication using X509 certificates see my blog on Configuring SCOM 2007 to perform mutual authentication with non-domain joined machines using X509 certificates.

When deployed in an Active Directory environment MOM2005 server and MOM2005 clients will mutually authenticate each other by using Kerberos Protocol. This is the default behavior of MOM 2005 which is controlled by the Mutual Authentication Required Setting of MOM 2005 server. This mutual authentication provides the assurance to the server that the alert and event information received from the clients is coming from the trusted source (in other words is not spoofed). At the same time the client is assured that the information it is sending is going to the trusted destination i.e. MOM 2005 server and not some imposter. Hence the built-in mutual authentication mechanism provides the foundation for secure operation of MOM 2005.

In a workgroup environment Kerberos authentication cannot be performed, therefore in order to accommodate the management of non-domain joined machines we are forced to disable the mutual authentication option on MOM 2005. Since this setting is global it consequently affects both domain joined machines and non-domain-joined machines, therefore significantly reducing the level of security within the MOM 2005 environment.

To mitigate this limitation of MOM2005 we can utilize IPSec to perform mutual authentication via X509 certificates. The basic idea of this solution is to leverage the fact that the IPSec channel has to be establish prior to the MOM specific traffic ever being exchanged, so by utilizing the mutual authentication capabilities of IPSec we can regain that high level of assurance that the data is being exchanged between the trusted peers.

To see the walkthrough, please, follow this link.