Since we began working with MIIS back in 2003, we have taken many different approaches to group management and member population. Perhaps the most familiar solution is the Group Populator program that is provided with MIIS 2003/ILM 2007. Later we were introduced to a SQL-based solution by Microsoft that morphed into a Microsoft Identity and Access Management Series application. At the same time, we branched off and created multiple solutions for our customers that are based on Extensible Connectivity Management Agents (XMAs). One such solution was the management of Active Directory security groups that have members in foreign domains. Since the ILM Active Directory Management Agent (AD MA) does not support this out-of-the-box, we had to be inventive to find a solution that works without writing a replacement for the AD MA.
Figure 1: Sample AVP Text File
In this text file, there is a person record for each user and a group record for each group. At a minimum, these records define a unique identifier attribute, such as an employeeID or uid that can be used to join the record to a MetaVerse object. In addition, each group that needs to be managed also defines a multi-valued attribute for member. The values of the member attribute are references that join to objects in the Group Populator MA Connector Space by the unique identifier (automatically), and ultimately to objects in the MetaVerse by virtue of the joins that the Connector Space have with the MetaVerse. A standard AVP MA, referred to as the “Group Populator MA”, imports the text file and manages the groups stored within the MetaVerse.
Figure 2: Reference Attributes & MetaVerse Connectors
At this point, the Group Populator solution has a direct flow from the member reference attribute in the Group Populator MA Connector Space to the member reference attribute in the MetaVerse. The next step is to export that group membership out to Active Directory. Fortunately, ILM has the built-in capability to convert the member reference attribute in the MetaVerse to the member reference attribute for a group in AD, so long as both have correlating entries for each of the members. Therefore, the solution has the out-of-the-box capability to directly flow the member reference attribute from the Group Populator MA, through the MetaVerse and back out to AD without writing a single line of code (other than customizing the Group Populator program to suit your needs).
An inherent limitation of the Group Populator solution is the reliance on an outside process that must execute successfully during an ILM schedule. Since I tend to be a purist, I wanted to encapsulate the Group Populator into an XMA. This way the entire process could be driven from within ILM without the need for any external processes. The XMA worked like a charm and is still in use today, but it eventually gave way to the new SQL-based solution for group management.
We encountered a serious performance issue with how ILM processes the multi-valued attributes coming from SQL tables. This is a problem with the MIIS engine (2003 SP1) and not the solution and wouldn’t inevitably be a problem for limited-sized deployments. However, we were managing hundreds of thousands of security groups. I let it run on the initial import for around 48 hours before I finally stopped it… and it was not even two-thirds of the way through! Luckily, we were able to retrofit it using an XMA to dump the data from SQL to AVP text files. In fact, the end result was not much different than the XMA we created for the original Group Populator. The AVP text file import takes only a fraction of the amount of time as compared to going directly to SQL Server for the data. In addition, we were able to apply a snapshot design in order to create a delta. Importing a delta of the group changes takes only a fraction of time as compared to always having to do full imports (a requirement of the Group Populator solutions). I don’t want to stray in that direction because it is enough subject matter for another article by itself. I do want to point out that MIIS 2003 SP2 and ILM 2007 both have substantial performance improvements over MIIS 2003 SP1, but still cannot compare to using AVP files, especially with deltas.
Fortunately, there is a solution that can leverage each of the known group management applications. Thus, you can retrofit your existing Group Populator implementation to work with groups that have members in foreign domains. This is done by creating an Advanced Export Attribute Flow for the member attribute in the AD MA. Unfortunately, ILM will not allow you to create an Advanced Export Attribute Flow from one reference attribute to another, so the design has to be altered slightly to accommodate. Instead of storing the member attribute in the MetaVerse as a reference attribute, the solution calls for using a custom multi-valued String attribute. The added benefit is substantial performance improvements by eliminating the referenced attribute cross-reference that ILM performs for each member. This eliminates a substantial number of SQL queries by the ILM engine when managing these objects.
Next, I will provide a solution in two parts. The first part is a MetaVerse Provisioning solution to ensure that Foreign Security Principals are created for all members. The second part is how to flow the member attribute into AD by manually writing out the reference attribute for member.
Active Directory allows a principal from a foreign domain by creating a pointer to it in the group’s domain. This pointer is referred to as a Foreign Security Principal (FSP). FSP’s are principals with valid SIDs (Security IDs) just like users, groups and computers. The group’s domain can utilize the SID for an FSP just like a first-class principal in the same AD Forest.
Provisioning FSPs is a two step process. First, an AD MA needs to be configured for each of the AD Forests in question.
Figure 3: Multiple AD MAs with MetaVerse Connectors
For the purposes of this article, let’s assume there are two AD MAs; one AD MA imports and manages groups from the “GroupDomain” and another imports users from the “UserDomain”. The AD MA for the group domain joins and/or projects groups in accordance to the Group Populator solutions. Typically, the AD MA would only join groups and leave it up to the Group Populator MA to do the projecting. The AD MA for the user domain needs to project persons into the MetaVerse, or follow the design for your solution. The end result accomplished by the solution is to ensure that all users and groups that are managed have connectors to objects in the MetaVerse. The Group Populator solutions have the same requirement to facilitate the automatic references.
The AD MAs need to flow two attributes into the MetaVerse. The first attribute is the distinguishedName (DN). You will need to create a new Indexable String attribute (not indexed) in the MetaVerse named “distinguishedName” and add it to the person and group object types. You can setup a direct import attribute flow from <dn> to the new distinguishedName attribute. The second attribute is the securityID (SID). Same as distinguishedName, you will need to create a new Indexable String attribute in the MetaVerse named “securityID” and add it to the person and group object types. This will require an advanced import attribute flow from objectSid to securityID. Since an objectSid is in a binary octet format, the flow will need to convert it to a readable string format referred to as an SDDL (Security Descriptor Definition Language). Here is a code snippet that illustrates two ways to perform this conversion…
// Advanced Import Attribute Flow with .NET 1.1 byte[] objectSID = csentry[ "objectSID" ].BinaryValue; mventry[ "securityID" ].Value = ConvertSidOctetToSDDL( objectSID ); // Advanced Import Attribute Flow with .NET 2.0+ using System.Security.Principal; byte[] objectSID = csentry[ "objectSID" ].BinaryValue; mventry[ "securityID" ].Value = new SecurityIdentifier( objectSID, 0 ).ToString(); using System; using System.Runtime.InteropServices; // ConvertSidOctetToSDDL for .NET 1.1 implementation [DllImport( "advapi32.dll", CharSet = CharSet.Auto, SetLastError = true )] private static extern int ConvertSidToStringSid(IntPtr pSID, ref IntPtr pSidString); public static string ConvertSidOctetToSDDL (byte[] octetArray) { IntPtr sidPtr = Marshal.AllocHGlobal( octetArray.Length ); Marshal.Copy( octetArray, 0, sidPtr, octetArray.Length ); IntPtr sidStringPtr = IntPtr.Zero; int result = ConvertSidToStringSid( sidPtr, ref sidStringPtr ); if ( result == 0 ) { throw ( new System.ComponentModel.Win32Exception( Marshal.GetLastWin32Error() ) ); } string sidString = Marshal.PtrToStringAuto( sidStringPtr ); Marshal.FreeHGlobal( sidPtr ); Marshal.FreeHGlobal( sidStringPtr ); return sidString; }
Once you have established the import attribute flows, you can then proceed with the provisioning logic. This will require a MetaVerse rules extension. In the Provision method for the MV extension, you will need to check for and create the FSP for any principals that may be members of the groups. This would include persons or groups that have valid distinguishedName and securityID attributes. To check for the existence of the FSP, you will need to use DirectoryServices and calculate the expected distinguishedName of the FSP in AD. If the FSP does not exist, DirectoryServices may also be used to create the FSP. There is a clever trick you can use to do this; if you modify the member attribute of a decoy group, AD will automatically generate the FSP for you. Here is a code snippet for the Provision logic...
using System; using System.DirectoryServices; // Provision Logic, could add additional object type restriction if ( ( mventry[ "distinguishedName" ].IsPresent ) && ( !mventry[ "distinguishedName" ].Value.Equals( string.Empty ) ) && ( mventry[ "securityID" ].IsPresent ) && ( !mventry[ "securityID" ].Value.Equals( string.Empty ) ) ) { string securityID = mventry[ "securityID" ].Value; string principalDn = mventry[ "distinguishedName" ].Value; // Insert Proper Values for the following variables... string ldapPrefix = "LDAP://GroupDomainDCName.GroupDomain.local/"; string forestDomainDn = "DC=ForestDomain,DC=local"; string groupDomainDn = "DC=GroupDomain," + forestDomainDn; string provisioningGroupPath = groupDomainLDAPPrefix + "CN=FSP Provisioner," + groupDomainDn; string fspPath = ldapPrefix + "CN=" + securityID + ",CN=ForeignSecurityPrincipals," + groupDomainDn; // This is only necessary for principals in different forests if ( ! principalDn.ToLower().EndsWith( forestDomainDn.ToLower() ) ) { // Check to see if the FSP has already been created bool fspExists = false; DirectoryEntry searchRoot = null; SearchResultCollection resultSet = null; try { searchRoot = new DirectoryEntry( fspPath ); string[] props = new string[]{"distinguishedName"}; DirectorySearcher searcher = new DirectorySearcher( searchRoot, "", props, SearchScope.Base ); resultSet = searcher.FindAll(); fspExists = ( resultSet.Count > 0 ); } catch {} finally { if (resultSet != null) try { resultSet.Dispose(); } catch {} if (searchRoot != null) try { searchRoot.Dispose(); } catch {} } // Create the FSP by adding the principal as a member to a decoy security group if ( !fspExists ) { DirectoryEntry groupEntry = null; try { try { groupEntry = new DirectoryEntry( provisioningGroupPath ); groupEntry.RefreshCache( new string[]{ "member" } ); } catch { throw new UnexpectedDataException( "The FSP Group could not be opened." ); } groupEntry.Properties[ "member" ].Add( "<SID=" + securityID + ">" ); groupEntry.CommitChanges(); groupEntry.Properties[ "member" ].Clear(); groupEntry.CommitChanges(); } finally { if (groupEntry != null) try { groupEntry.Dispose(); } catch {} } } } }
Once the FSPs have been created in the group domain, you are free to add members to groups using that FSP. The second part of the solution is to manually flow the member attribute into AD.
Figure 4: Elimination of Reference Attributes
For purposes of this solution, let’s call this new MetaVerse attribute “memberUID”. You would likely want to rename this to the proper attribute name of your member unique identifier, such as “memberEmployeeID” or “memberNumber”. Next, modify your Group Populator MA by removing the current import attribute flow from member to member. Now you can switch to the Configure Attributes page of the Group Populator MA Properties and change the member attribute to be of type String from the original value of Reference (DN). Next, reconfigure the attribute flow by creating a direct import attribute flow from member to memberUID in the MetaVerse.
At this point we have effectively removed the reference attributes from both the Connector Space and the MetaVerse. This eliminates the need for ILM to perform any cross-referencing between objects within the Connector Space or objects within the MetaVerse (mutually exclusive, then within the attribute flow). As you can imagine, this amounts to substantial performance improvements and considerably less chatter between ILM and SQL Server to maintain those references. There is another benefit; the Group Populator MA no longer needs the matching records for the member attribute! You can now either remove all records for objectType equal to person from the import source or simply add a Connector Filter to eliminate them from the Connector Space import. Since the Connector Space no longer maintains those objects, we eliminate the overhead required to add those objects to the Connector Space and join those objects to persons in the MetaVerse.
using System; using System.Text; using System.Collections.Specialized; using System.Data; using System.Data.SqlClient; /// <summary> /// Overloaded function to return a single MetaVerse Entry /// </summary> /// <param name="lookupAttributeValue">nVarChar value for the SQL Where Clause</param> /// <param name="lookupAttributeName">Column name for the SQL Where Clause</param> /// <param name="selectAttributeNames">Column names for the SQL Select Clause</param> /// <returns>Pipe-delimited list of attribute values</returns> public static string LookupMetaVerse(string lookupAttributeValue, string lookupAttributeName, string selectAttributeNames) { StringCollection lookupAttributeValues = new StringCollection(); lookupAttributeValues.Add( lookupAttributeValue ); StringDictionary returnValue = LookupMetaVerse( lookupAttributeValues, lookupAttributeName, selectAttributeNames ); if ( returnValue.ContainsKey( lookupAttributeValue ) ) { return returnValue[ lookupAttributeValue ]; } else { return string.Empty; } } /// <summary> /// Overloaded function to return multiple MetaVerse Entries /// </summary> /// <param name="lookupAttributeValues">nVarChar values for the SQL Where Clause</param> /// <param name="lookupAttributeName">Column name for the SQL Where Clause</param> /// <param name="selectAttributeNames">Column names for the SQL Select Clause</param> /// <returns>Dictionary of pipe-delimited list of attribute values</returns> public static StringDictionary LookupMetaVerse(StringCollection lookupAttributeValues, string lookupAttributeName, string selectAttributeNames) { StringDictionary returnValue = new StringDictionary(); StringDictionary processingIds = new StringDictionary(); // This connection string for the ILM database // *should* work for most deployments string miisCS = "Data Source=(local);" + "Initial Catalog=MicrosoftIdentityintegrationServer;" + "integrated security=SSPI;persist security info=true"; SqlConnection cn = new SqlConnection( miisCS ); cn.Open(); try { int index = 0; foreach (string lookupAttributeValue in lookupAttributeValues) { if (index == 100) { ProcessLookupMetaVerse( cn, processingIds, returnValue, lookupAttributeName, selectAttributeNames ); index = 1; } else { index++; } processingIds.Add( lookupAttributeValue.ToLower(), lookupAttributeValue ); } if (processingIds.Count > 0) { ProcessLookupMetaVerse( cn, processingIds, returnValue, lookupAttributeName, selectAttributeNames ); } } finally { cn.Close(); } return returnValue; } private static void ProcessLookupMetaVerse(SqlConnection cn, StringDictionary processingIds, StringDictionary returnValue, string lookupAttributeName, string selectAttributeNames) { string[] selectAttributes = selectAttributeNames.Split( ',' ); if ( selectAttributeNames.ToLower().IndexOf( lookupAttributeName.ToLower() ) == -1 ) { selectAttributeNames = lookupAttributeName + ", " + selectAttributeNames; } StringBuilder sb = new StringBuilder(); foreach (string lookupAttributeValue in processingIds.Values) { if (sb.Length > 0) { sb.Append( "," ); } sb.Append( "N'" ).Append( lookupAttributeValue ).Append( "'" ); } string sql = "SELECT " + selectAttributeNames + " FROM MicrosoftIdentityintegrationServer.dbo.mms_metaverse WITH (nolock)" + " WHERE " + lookupAttributeName + " IN (" + sb.ToString() + ")"; SqlCommand sqlProc = new SqlCommand( sql, cn ); sqlProc.CommandType = CommandType.Text; sqlProc.CommandTimeout = 1200; SqlDataReader sqlReader = null; try { sqlReader = sqlProc.ExecuteReader(); while (sqlReader.Read()) { string lookupAttributeValue = sqlReader[ lookupAttributeName ].ToString(); string returnAttributeValues = string.Empty; foreach (string selectAttributeName in selectAttributes) { string returnAttributeValue = string.Empty; try { returnAttributeValue = sqlReader[ selectAttributeName.Trim() ].ToString(); } catch {} if ( !returnAttributeValues.Equals( string.Empty ) ) { returnAttributeValues += "|"; //Delimitor } returnAttributeValues += returnAttributeValue; } string properCaseValue = processingIds[ lookupAttributeValue.ToLower() ]; returnValue.Add( properCaseValue, returnAttributeValues ); processingIds.Remove( lookupAttributeValue.ToLower() ); } } catch (Exception ex) { System.Diagnostics.Debug.WriteLine( ex.Message ); throw ex; } finally { try { if (sqlReader != null) { sqlReader.Close(); } } catch {} } sqlProc.Connection = null; }
The LookupMetaVerse function accepts a list of lookup values to search the MetaVerse. In this case the lookup values are the individual UID’s for the group members. The function returns a list of pipe-delimited results for each of the attributes specified in the selectAttributeNames argument. For example, to return the distinguishedName and securityID for a MetaVerse entry with the uid of “UserDomain\User1”, you would use the following syntax…
string attribValue = @"UserDomain\User1"; string attribName = "uid"; string selectAttribs = "distinguishedName, securityID"; string results = LookupMetaVerse( attribValue, attribName, selectAttribs ); string[] attributeValues = results.Split( '|' ); string distinguishedName = attributeValues[ 0 ]; string securityID = attributeValues[ 1 ];
I should probably add a disclaimer regarding the LookupMetaVerse function. Microsoft does not support you going directly to the SQL Server tables to find information. They take this stance because it gives them the capability to change the database schema in future versions. While I understand the reasoning behind this, it is incredibly faster and more efficient to go directly to the database in the proper circumstances. Besides, you will only have a single point-of-failure in your codebase, and that would be this reusable LookupMetaVerse function. It could easily be retrofitted to use the Utils.FindMVEntry function. However, the FindMVEntry function does have limitations and has a considerable amount of additional overhead, such as instantiating a new MVEntry object and populating it will all data for every user you need to look-up. The LookupMetaVerse function is extremely efficient by using a forward-only, read-only, non-locking DataReader and consolidating multiple look-ups into a single query. It’s the difference between one as opposed to thousands of queries that ILM might require to achieve the same results.
if ( ( !mventry[ "memberUID" ].IsPresent ) || (mventry[ "memberUID" ].Values.Count == 0) ) { if (csentry[ "member" ].IsPresent) { csentry[ "member" ].Values.Clear(); } } else { // Insert Proper Values for the following variables... string forestDomainDn = "DC=ForestDomain,DC=local"; string groupDomainDn = "DC=GroupDomain," + forestDomainDn; // Perform a cross-reference of the memberUID to MetaVerse Entries StringCollection uids = new StringCollection(); foreach (Value mvValue in mventry[ "memberUID" ].Values) { uids.Add( mvValue.ToString() ); } string selectAttribs = "distinguishedName, securityID"; StringDictionary memberAttributes = LookupMetaVerse( uids, "uid", selectAttribs ); StringDictionary mvValues = new StringDictionary(); StringCollection csValues = new StringCollection(); StringCollection removeValues = new StringCollection(); foreach (string uid in memberAttributes.Keys) { string[] attributeValues = memberAttributes[ uid ].Split( '|' ); string principalDn = attributeValues[ 0 ]; string securityID = attributeValues[ 1 ]; if ( ( !principalDn.Equals( string.Empty ) ) && ( !securityID.Equals( string.Empty ) ) ) { // If the member is in the same forest, use the DN of the principal // Otherwise, use the Foreign Security Principal DN if ( principalDn.ToLower().EndsWith( forestDomainDn.ToLower() ) ) { mvValues.Add( principalDn.ToLower(), principalDn ); } else { string fspDn = "CN=" + securityID + ",CN=ForeignSecurityPrincipals," + groupDomainDn; mvValues.Add( securityID.ToLower(), fspDn ); } } } if (csentry[ "member" ].IsPresent) { foreach (Value csValue in csentry[ "member" ].Values) { string key = csValue.ToString().ToLower(); if (key.IndexOf( "foreignsecurityprincipals" ) > -1) { // Parse out the securityID key = key.Substring( 0, key.IndexOf( "," ) ); key = key.Substring( 3 ).ToLower(); } if ( mvValues.ContainsKey( key ) ) { csValues.Add( key ); } else { removeValues.Add( csValue.ToString() ); } } } foreach (string csValue in removeValues) { csentry[ "member" ].Values.Remove( csValue ); } foreach (string mvValue in mvValues.Keys) { if ( !csValues.Contains( mvValue ) ) { csentry[ "member" ].Values.Add( mvValues[ mvValue ] ); } } }
The preceding code performs a manual reconciliation of one multi-valued attribute to another. You can reuse the logic for other multi-valued attribute flows that need special logic as well. The overall scheme is that you merge one list into another and keep track of what items are in the target list, but not the source list. These remnants are the items that need to be removed from the target list. In this case, the source list is the MVEntry memberUID attribute and the target list is the CSEntry member attribute.
In this article, we have provisioned Foreign Security Principals into an Active Directory Domain for group members that are outside the group’s Active Directory Forest. We also learned how to leverage the existing Group Populator solutions to manage these members, while improving performance. This included reusable code that is handy in everyday ILM deployments and a better understanding of how ILM processes reference attributes as opposed to standard multi-valued String attributes.