Before talking about SharePoint, it is necessary to talk about the Windows operating system. Security authorization is based on Access Control Lists, or ACLs. An ACL is a list of access control entries (ACE). Each ACE identifies a security principal; such as a user or AD group, and the access rights allowed, denied, or audited for that security principal. The original designers of the Windows OS set a maximum size of 64K bytes for ACLs, which at the time seemed much more that would be needed.

Operating System ACLs

An obvious question is how many users or AD groups can an ACL contain? The answer is, “It depends”. Since an ACL is composed of ACEs, the question becomes “How many ACEs can an ACL contain”? An ACE is a variable length structure. The variable part is the user or group’s security ID, or SID. The security identifier (SID) structure is a variable-length structure used to uniquely identify users or groups. The variability is driven by the domain topology in which the SID was created and assigned. A deeply nested forest domain structure will result in longer SIDs than a flat domain structure for example. Given this variability, the commonly heard numbers of ACEs in an ACL range from 1,000 to 2,000.

SharePoint ACLs

Up to this point, we have been talking about operating system ACLs. These types of ACLs can impact search crawling (discussed later), but not authorization within SharePoint itself.

SharePoint has its own type of ACL it uses for authorization within SharePoint. SharePoint uses its ACLs to make access decisions; as well as, UI trimming decisions. SharePoint ACEs are optimized for SharePoint’s use cases; hence the format is simpler than operation system ACEs. In SharePoint, every user or AD group explicitly added to a site collection gets an entry in the site collection’s UserInfo table, is assigned a principal ID (PID), and gets an entry in the Perms table.

The Perms table contains SharePoint’s ACLs, which are composed of SharePoint ACEs. A SharePoint ACE consists of the PID, and an 8 byte permissions mask. So if inheritance is NOT broken, every user or NT group given access to a site collection has one row in the UserInfo table and one row in the Perms table. If a user which receives access indirectly through a NT group accesses the site, a row is added to the UserInfo table to contain that user’s display name, email address, and depending upon the operations that user performs on the site, a copy of their operating system SID; but no new row is added to the Perms table since the original row for the user’s group is sufficient to define the SharePoint permissions for that user.

The Crawl ACL Translation Problem

The SharePoint search crawler keeps a copy of an object’s ACL in the search database to enable security trimming at search query time. To keep things simple and efficient for query processing, the search crawler translates SharePoint ACLs into operating system ACLs when building the index. Herein lays a problem. If the translated SharePoint ACL generates an operating system ACL larger than 64K, the search crawl throws an error, “parameter is incorrect”. If this error occurs at the site collection level, none of that site collection’s content will be indexed. This results in a situation in which the site collection itself works fine, but none of the site collection content can be searched.

How Can ACLs > 64K Happen?

One way is to explicitly add enough individual users or AD groups to a site collection’s membership so that the mapping of the SharePoint ACL to operating system ACL during search crawling results in an operating system ACL greater than 64K. The first section of this post says this will happen between 1,000 and 2,000 unique users or AD groups, depending upon the size of each user or group’s SID. While unlikely, this is a possibility.

The other, more likely way, is through broken permission inheritance within a site collection.

What’s Wrong with Broken Inheritance?

Let’s walk through a common scenario. A site collection is created for a large community of users; for example, a sales and marketing group with 5,000 members. Using best practice, we create a Sales and Marketing AD group, add the 5,000 users either directly or by nesting already existing AD groups. We then grant this AD group access to the site collection. The end result is one row in the UserInfo table and one row in the Perms table. Both the SharePoint ACL and mapped operating system ACL are less than 100 bytes long. Life is good; search indexing is happy and efficient.

We then start loading sales and marketing documents into an array of subsites and document libraries. It becomes apparent one subsite is for marketing research. We only want 25 people to have access. What do we do? We break inheritance with the site collection and give those 25 people explicit access to the marketing research subsite. We now have at least 25 rows in the UserInfo table, but more importantly, 25 additional rows in the Perms table.

Let’s say this scenario repeats itself for subsites divided by product line and country/region; since it is important to keep product strategy restricted. Maybe we break inheritance for 10 other subsites of 25 users each.  The net result is an additional 250 rows in the Perms table.

Now comes the killer scenario. We add another AD group to site collection member ship which includes all full time employees (FTEs); containing a total of 30,000 members, either directly or through nested AD group membership. This group has read access to all sales and marketing content except the previously mentioned subsites with broken inheritance. This action by itself only adds one more row to the Perms table, which is not a problem; however, the plot thickens…

It is determined that certain FTEs in manufacturing and other support organizations need contributor access to specific documents scattered throughout the sales and marketing site collection. One-by-one, inheritance is broken on specific documents, and individual FTE users are granted contributor access to just those documents.  Each time this happens, rows are added to the Perms table. Over time, perhaps 5 unique FTE users are granted broken inheritance access to 300 different individual documents, adding 1,500 rows to the Perms table. Broken inheritance has become a growing cancer.

Eventually the Perms table reaches 1,800 rows. The Perm row for the site collection itself now contains 1,800+ ACEs. The incremental crawl that night starts throwing “parameter is incorrect” errors while crawling the site collection. Why? Because the operating system ACL the search crawling is trying to generate from the 2,000+ ACEs in the site collection Perms table exceeds the 64K ACL max size.

What Can You Do To Protect Yourself?

Plan ahead when designing the information architecture (IA). Besides the number of AD groups and users explicitly granted membership in your site collections, consider what would have if you need to start breaking inheritance for granular access control at the list, library, or individual document level. Several customers have already hit problem.

I know of one major manufacturer whose Extranet supply chain site is based on one site collection. As more-and-more broken inheritance was applied to individual documents to keep each supplier’s information private to that supplier, the site collection has been unsearchable.

The only after-the-fact solution is painful refactoring of the site collection structure to more granularly designed site collections, whose usage is carefully aligned with the intended user communities.