Keith Richie : Understanding SharePoint

Upgrading a SharePoint Portal Server 2003 Shared Services Farm with NLB and Forced SSL to MOSS 2007

krichie — Wed, 20 Dec 2006 01:46:00 GMT

How's that for a blog posting title!

Oh, and I'll add IIS redirect virtual servers into the mix.

I've been working with a customer on detailing all the steps necessary to make this a reality, and I've seen quite a few posts out on some internet searches where a ton of folks are asking this exact same question.

"How and what is the procedure for upgrading a SharePoint Portal Server 2003 Shared Services Farm with Network Load Balancing and Forced SSL to MOSS 2007"

This is certainly a loaded question, and there are important aspects you really need to understand about your current deployment and take into consideration before you just install the bits and start running. Also, this doesn't even consider the state of your sites, i.e. how many custom site definitions, site templates, or web parts, etc, etc, etc.

As well, I discovered one important piece about your current environment you need to be aware of that boils down to what URL and name you bound your portal/wss sites on to begin with. My team has been harping on me to document and explain this specific “What’s in a name” piece for a very, very, very long time, and I don’t see how I can get by with explaining this whole upgrade process without starting with it.

Unfortunately, you’re not going to get all your answers in “This” blog post, so consider this a teaser trailer for a series of blog postings on this subject coming to an aggregator near you!

More to come

- Keith Richie

Understanding SharePoint - Part 6 - Duplicate Content Sources bound to multiple indexes causes duplicate alerts to be fired

krichie — Tue, 28 Nov 2006 04:58:00 GMT

I had a theory that this caused problems and validated it today while working with a customer who has a large number of content sources spread across multiple indexes.

Note: First of all, poking around through the UI to look at all the details of indexes, content sources, etc is a real pain. Thus (You guessed it :)) I wrote a tool to gather and export this information so that I could look at it holistically. It was actually a modification to the SPSAddCS tool in the SharePoint utility suite, just modified to use XML and added a "Read" operation to it to dump the data in XML.

Once I've had a chance to test it out in more detail, I'll post the code for it as well.

Since this customer had 4+ portal indexes with approximately 10 content sources per index, it made it allot easier to look at everything and spot the problem quickly.

Once I saw that they had two content sources pointing to the exact same content yet on two different indexes, it was a simple matter of setting this up on a plain Jane standalone SharePoint Portal Server 2003 box to validate.

I validated this using the following setup on my local test server.

Created a plain IIS web site and placed a default web page with the following contents:

<html>

<body>

<P>Hello!</P>

<P>Keith Richie was here</p>

</body>

</html>

Issued a search in the portal for “Keith Richie”, then selected “Alert Me” to create a portal alert, with its’ settings set to immediate
Added a content source to this plain web site did a full crawl and confirmed that the notification was sent
Created a new index

Created a duplicate content source to the same plain web site bound to this index
Made an update to the default web page
Did a full crawl on both indexes, and I received two notifications about the same change

Created a third new index

Created a third duplicate content source to the same plain web site bound to this index.
Made an update to the default web page
Did a full crawl on all three indexes, and I received three notifications about the same change

Depending on your structure, and Exclude/Include rules created, this can occur for SharePoint sites as well

You ARE however, blocked from creating a duplicate content source on the same index, but not on a completely different index. When you attempt to create a duplicate source address on the same index, you'll get the following error message:

“The object you are trying to create already exists. Try again using a different name. “

But if you are creating this on a different index, you will not get a warning.

In MOSS 2007, you are not allowed to even have multiple indexes unless you create multiple SSPs. I'm not 100% up to snuff on the WSS 3.0/MOSS 2007 alert architecture, but I am very curious and would suspect that you would encounter the same problem, and could confuse users.

- Keith

Previous Posts on this series:

Understanding SharePoint Series

Understanding SharePoint - Part 5 - Keep those webs small if you must use SMIGRATE

krichie — Thu, 27 Jul 2006 18:46:00 GMT

Understanding SharePoint - Part 5 - Keep those webs small if you must use SMIGRATE

This is the fifth post in a series in regards to "Not being mislead by what your seeing :)"

DISCLAIMER: This post may include steps that guide you through issuing statements against your SharePoint content databases. By no means does this mean that you should change any thing in the database. This is simply for "READING" values, and even this should be done during Off-Peak hours.

The Scenario

You have a sub web of a site, and it's a whopper. Let's say it's about 25GB :) Perhaps you want to relocate it to it's own site collection, or you need to move it around in the hierarchy. The only way to do this is via SMIGRATE as my post titled Moving a web and preserving permissions using SPUserUtil describes.

You kick off a SMIGRATE to export the web, and it eventual fails with possibly an access violation.

The Reason

Unlike STSADM, which chunks the individual records of a site collection to the backup file, SMIGRATE allocates memory for caching the overall manifest of the export. SMIGRATE could require up to 3 times the amount of memory needed to generate the manifest data. The data is not chunked as in STSADM, but rather generated in memory and then persisted to the export file.

It generates the manifest file in UTF-8 format for use later to read it back in. It also generates in another format for use with XML compilation internally, which could lead to needing generally 2x the amount of memory as the first allocation, thus making the necessary memory requirement around 3x.

As you can see, trying to SMIGRATE our 25GB scenario sub web could possibly require around 75GB of virtual memory, and most systems would never have this. Besides, you would still be limited by the amount of addressable memory on a 32 bit system, as is Windows 2003. Thus, at most, there is a 2-3 GB address space for user mode applications depending on your servers setup. The entire web could never be exported if the overall size of the memory requirement during the export exceeded 2-3 GB, and that's if you happened to be fortunate to not have anything else running on the server that day and the planets aligned just right.

In Conclusion

If you have to use SMIGRATE routinely to move webs, then keep them small. Otherwise, you may need to use something to export your files out to reduce the webs size, to then move them, and then import the items back in, thus losing original time stamps, user associations to document and list items, etc.

Hope this helps!!

- Keith

Previous Posts on this series:

Understanding SharePoint - Part 1 - Understanding the SharePoint Portal Server Indexer
http://blogs.msdn.com/krichie/archive/2006/07/20/672755.aspx

Understanding SharePoint - Part 2 - The Infamous Query Plan Bug and The Origins of SPSiteManager
http://blogs.msdn.com/krichie/archive/2006/07/20/672755.aspx

Understanding SharePoint - Part 3 - Just because your database is larger doesn't necessarily mean that your index size will be
http://blogs.msdn.com/krichie/archive/2006/07/20/673339.aspx

Understanding SharePoint - Part 5 - Lock those sites before performing backups to prevent orphaned entries! -
http://blogs.msdn.com/krichie/archive/2006/07/25/678040.aspx

Additional reference material:

Administrator's Guide for Windows SharePoint Services - Migrating and Upgrading Web Sites
http://www.microsoft.com/resources/documentation/wss/2/all/adminguide/en-us/stsf12.mspx?mfr=true

Disaster Recovery in SharePoint Products and Technologies - Recovering Individual SharePoint Sites
http://www.microsoft.com/technet/prodtechnol/sppt/reskit/c2861881x.mspx

Backing Up and Restoring Web Sites
http://office.microsoft.com/en-us/assistance/HA011608261033.aspx

SPSiteManager is contained in the The SharePoint Utility Suite at:
http://www.microsoft.com/sharepoint/downloads/components/detail.asp?a1=724

Understanding SharePoint - Part 4 - Lock those sites before performing backups to prevent orphaned entries!

krichie — Tue, 25 Jul 2006 20:08:00 GMT

Understanding SharePoint - Part 4 - Lock those sites before performing backups to prevent orphaned entries!

This is the fourth post in a series in regards to "Not being mislead by what your seeing :)"

Isn't this really Orphaned Sites - Part 4?

I almost made this part 4 of my "Orphaned Sites" series, but decided not to, because I want to stress the fact of the importance to read/write locking your sites before performing ANY site collection level type backups.

The Scenario

For whatever reason, you need to restore a specific site collection, or web, etc. It could be that important information was deleted, or you're in full disaster recovery mode.

You kick off a STSADM -o restore of the site collection and you even include -overwrite switch, and it fails with an error message that indicates:

Exception from HRESULT: 0x80040E2F

This HRESULT simply means, "Violation of PRIMARY KEY constraint ", which generally means one thing :) You have entries in the database for this site previously, possibly orphaned, that is preventing the restore to continue, or...what this post is about :)

But I used the overwrite switch, should that not have taken care of cleaning up the site first?

You would think, since all the overwrite switch does is force a delete of the site collection on the database before performing the restore. It's the same process if you were to do the following manual steps:

STSADM -o deletesite -url http://server/sites/thesite
STSADM -o restore -url http://server/sites/thesite -filename thesite.bak

(Of course, if you have orphans on the site :) It may be difficult to actually delete the site, thus refer to my "Orphaned Sites" series, :))

But the problem isn't that there was orphaned data to begin with, but rather "New" data related to the site added outside of the restore, that then causes the restore to fail.

The Reason

The reason in this case, may be due to the fact that while the restore is progressing, the site isn't locked and therefore users accessing the site is causing the duplicate entries.

For the most part, everything stored along with a site has a unique GUID, so this isn't "Duplicate Named" documents or duplicate list items etc causing this, as those would have a unique GUID associated with their entries. Instead, what it COULD be, is entries in the UserInfo table being created for the site collection by means of user visits, and thus, when they restore gets to the point where it starts attempting to insert it's copy of that same UserInfo data, the exception occurs.

How this can occur, is actually quite simple.

When we detect a user hitting a site collection for the first time, we cache information in the UserInfo table for that user specifically associated with that site collection (See http://blogs.msdn.com/krichie/archive/2006/02/18/534752.aspx for a a more technical description of this) .
During a restore, we may not be inserting the Site Collection level UserData information yet,
Enough information however, IS currently restored (Such as a webs ACL, containing the ACL's of Users and Groups), to allow the authentication to the webs.
The Web was not locked during backup, therefore the lock state of the site is not set during the restore, thus users are able to access the webs in the site.
Because we see no information currently in the UserInfo table for the user, a record is created for the user.
The restore progresses along, and the backed up UserInfo data is inserted into the table, yet because there is already an entry for a user it's trying to restore in the database, the exception is thrown and the restore fails.

If the site collection was read/write locked before the backup was performed, you would not run into this problem, because the lock state of the site would prevent any access to it until the restore was complete, in which you would then go and unlock the site.

Read/Write Locking sites is actually a recommended best practice from the product group if you are going to run site collection level backups of your data on production sites If this is true, they why doesn't STSADM automatically lock the site for you?

Please refer to the official Backing Up and Restoring Web Sites - http://office.microsoft.com/en-us/assistance/HA011608261033.aspx for an explanation :) But I'll call out the specific reasons why here:

Site backup and restore are not designed to be used when the server is under active load.

If a site is in use when the backup operation is run, the data in that site may continue to change throughout the operation. The resulting backup file may be inconsistent with the actual state of the site and, if you restore this file, the restored site or database will be inconsistent as well. To avoid possible inconsistencies, lock the site collection prior to backing up a site in an active farm, using the "No Access" lock. After the backup is complete, set the lock back to "Not locked". For information about locking and unlocking a site collection, see the Managing Locks section of the Configuring Site Collection Quotas and Locks topic.

In Conclusion

It's extremely important to lock your sites before backing them up using STSADM. You could encounter problems later due to the inconsistencies that can be introduced as noted in the aforementioned official documentation, but not limited to just what I'm noting in this post.

If you use SPSiteManager from the SharePoint Utility Suite, it automatically sets the site to the No Access state before performing a backup operation. You can then use the additional -o unlock site operation to unlock your restores after complete. The SPSiteManger repartition operation performs the lock, and unlock automatically when moving a site.

Hope this helps!!

- Keith

Previous Posts on this series:

Understanding SharePoint - Part 1 - Understanding the SharePoint Portal Server Indexer
http://blogs.msdn.com/krichie/archive/2006/07/20/672755.aspx

Understanding SharePoint - Part 2 - The Infamous Query Plan Bug and The Origins of SPSiteManager
http://blogs.msdn.com/krichie/archive/2006/07/20/672755.aspx

Understanding SharePoint - Part 3 - Just because your database is larger doesn't necessarily mean that your index size will be
http://blogs.msdn.com/krichie/archive/2006/07/20/673339.aspx

Additional reference material:

Backing Up and Restoring Web Sites
http://office.microsoft.com/en-us/assistance/HA011608261033.aspx

SPSiteManager is contained in the The SharePoint Utility Suite at:
http://www.microsoft.com/sharepoint/downloads/components/detail.asp?a1=724

Understanding SharePoint - Part 3 - Just because your database is larger doesn't necessarily mean that your index size will be

krichie — Fri, 21 Jul 2006 04:25:00 GMT

Understanding SharePoint - Part 3 - Just because your database is larger doesn't necessarily mean that your index size will be

This is the third post in a series in regards to "Not being mislead by what your seeing :)"

DISCLAIMER: This post shows using Query Analyzer to issue statements against your SharePoint content databases. By no means does this mean that you should change any thing in the database. This is simply for "READING" values. And even this should be done during Off-Peak hours.

Uh, but my site has some really large documents in it, and the database is 3 times the size as a smaller site with thousands of files! What do you mean my index isn't going to be larger??

This discussion came up a few days ago, and I felt it was worth posting externally as well. I hope this posting helps you understand how to properly estimate your capacity planning as well as hardware requirements for your SharePoint environments.

The Scenario

You have a content database that has grown very large in size, or a content database with thousands of documents spread throughout, will the size of the full text indexes be equally larger compared to a small database and a small number of documents?

The answer is, no...Not necessarily. You cannot determine the size of your full text indexes based on content database size nor document items alone, and I'll show you why.

The Test

Consider the following example data:

Note: Your results if you perform this test, might be slightly different.

I've created a second Windows SharePoint Services site called and located at simply http://www.tailspintoys.com/sites/smalfilesite

This site was placed into a content database simply named small_site_DB.
I created a unique index in Portal server, and a content source that crawls this site and places the crawled content in said index. (We'll call it SmallFileFile_INDEX)
As a control value, I crawl this site now, and notice that it is ~1 MB in size, and indexed roughly 6 items (These are the default folders, pages for the site)
I uploaded exactly 50 word documents that have a size of roughly 1 MB each to a document library. This equates to ~ 50 MB of content
From SQL Enterprise Manager, I can see that the total size of data in the database is ~80 MB (This is due to data+SQL index sizes)
I reset, and did another full crawl on this site and it jumps to roughly ~2MB in size, and shows 57 documents in the index.
I also have SQL full text indexing enabled for this database, as they were enabled via the Windows SharePoint Services "Configure full-text search" in Windows SharePoint Services central Admin.
By viewing the the Full-Text properties for this database in SQL Server, I see that there is 204 items and the catalog size itself is ~2 MB, with a unique key count of 1387. The index size to database size ratio is 1:40

I've created 1 Windows SharePoint Services site called and located at simply http://www.tailspintoys.com/sites/largefilesite.

This site was placed into a content database simply named large_site_DB.
I created a unique index in Portal server, and a content source that crawls this site and places the crawled content in said index. (We'll call it LargeFile_INDEX)
As a control value, I crawl this site now, and notice that it is ~1 MB in size, and indexed roughly 6 items (These are the default folders, pages for the site)
I uploaded exactly 10 word documents that have a size of roughly 20 MB each to a document library. This equates to ~200 MB of content
From SQL Enterprise Manager, I can see that the total size of data in the database is ~286 MB (This is due to data+SQL index sizes)
I reset, and did another full crawl on this site and stays at ~1MB in size, and only shows 7 documents in the index.
I also have SQL full text indexing enabled for this database, as they were enabled via the Windows SharePoint Services "Configure full-text search" in Windows SharePoint Services central Admin.
By viewing the the Full-Text properties for this database in SQL Server, I see that there is 124 items and the catalog size itself is ~6 MB, with a unique key count of 1307. The index size to database size ratio is 1:47, roughly the same as the small database in size comparison.

The Portal full text index for the larger content database with large items, didn't grow at all, while the full text index for the smaller sized database with many smaller items, did.

The SQL full text index for the larger content database, grew to 6MB for only 124 items, while the SQL full text index for the smaller content database grew to 2MB with 204 items.

The Reason

For the portal full text, indexes, Portal Search products, two very important settings that determine how content is indexed, based on size. These settings are called MaxDownloadSize and MaxGrowFactor, and are best explained by this article:

Managing Search Settings for the Portal Site

http://office.microsoft.com/en-us/assistance/HA011648411033.aspx

By default, SharePoint Portal Server will only crawl and filter a file with a size of up to 16 megabytes (MB) because of MaxDownloadSize. In our test above, only the files in http://www.tailspintoys.com/sites/smallfilesite are indexed, because their initial size does not exceed 16MB.

Another factor to consider from the above article is MaxGrowFactor. If the file is downloaded for indexing because it does not exceed MaxDownloadSize, yet when extracted and expanded into raw text for filtering, it exceeds MaxDownloadSizexMaxGrowFactor, only the first 64K of the file is actually indexed (Assuming the values are at their default)

Thus only the files in http://www.tailspintoys.com/sites/smallfilesite are indexed. So now consider where you have a very large content database with hundreds of sites, webs, document libraries, and many varying document sizes, it's quite possibly that your total index size will actually be smaller than a small content database with an equal number of documents, but all of a size below this threshold.

In Conclusion

Don't rely on database size to have any predictability on index size. A follow up post will contain the reasons for the SQL full text indexes. As well, we'll delve into the effect of SQL table indexes on the data based on size, counts, etc.

Hope this helps!!

- Keith

Previous Posts on this series:

Understanding SharePoint - Part 1 - Understanding the SharePoint Portal Server Indexer
http://blogs.msdn.com/krichie/archive/2006/07/20/672755.aspx

Understanding SharePoint - Part 2 - The Infamous Query Plan Bug and The Origins of SPSiteManager
http://blogs.msdn.com/krichie/archive/2006/07/20/672755.aspx

Additional References:

Managing Search Settings for the Portal Site

http://office.microsoft.com/en-us/assistance/HA011648411033.aspx

Understanding SharePoint - Part 2 - The Infamous Query Plan Bug and The Origins of SPSiteManager

krichie — Fri, 21 Jul 2006 01:22:00 GMT

Understanding SharePoint - Part 2 - The Infamous Query Plan Bug and The Origins of SPSiteManager

This is the second in a series of posts in regards to "Not being mislead by what your seeing :)"

In the next post in this series, I'll talk about SharePoint index sizes vs the size of your databases, as you can't assume that with a larger database, you have an equally larger index. I'll show you how it can actually be the complete opposite, where your index can actually be smaller for a larger database vs an index of content on a much smaller database

Holy SQL Spikes Batman!

A long time ago, in a cube far, far away (Ok, across the hall from where I'm at now, so maybe not so far away)....I worked on an issue with a customer that:

On one hand, literally sucked the life right out of me :)
On the other, give me the opportunity to come up with SPSiteManager (See Codin' till the fingers go numb)

Due to a recent occurrence of another customer hitting this, and the fact that I discovered I really didn't share this before, I felt this posting was warranted.

First off, thanks to Denzil Ribeiro, a SQL Escalation Engineer who originally worked with me internally to help narrow this down when I was an Escalation Engineer for SharePoint.

The Symptoms and Problem

You have a SharePoint site or Portal where you really don't have a lot of users permissions, but you do provide access via possibly a security group.
You don't have a lot of documents or list items, and are well within capacity planning guidelines.
99% of your sites have a relatively low number of users associated to it,

There have never been more than say a few hundred unique users every hit the site, or perhaps they are mainly My Sites, with one or two users (The MySite owner, and a friend or two), or perhaps they are many, many team sites with low activity.

1% of your sites, or rather perhaps just one single Site collection, or Portal Site Collection, has > 10,000 users associated with it.

You've provided access to the site/portal with domain groups, or even a single domain group such as NT Authority\Authenticated Users

Yet, sometimes, your performance or even accessibility to your SharePoint environment is completely blocked.
The Processors on your SQL server spike and sustain a high CPU utilization %, thus bringing everything else to a grinding halt.

In the case where we first hit this, the customer had plenty of web front ends, and was running on an active/active cluster with 16 processors per node, and all 32 procs would spike to 100%, they had one content database with a Portal site collection, and > 9,000 team and My Sites.

The Reason

When any SQL statement is executed in SQL Server, the relational engine first looks through the procedure cache to verify that an existing query execution plan for the same SQL statement exists. SQL Server reuses any existing plan it finds, saving the overhead of recompiling the SQL statement. If no existing execution plan exists, SQL Server generates a new execution plan for the query. (It determines based on the data, the indexes, amount of joins, etc the best approach for retrieving the data to handle the request and presenting the data back to the caller). This execution plan is then cached for use later. This is especially important, if the statement is very parameterized, such as the queries we use to generate result sets for document library and list views.

In this case, the initial statement is compiled with one set of parameters, and for 99% of the time, the plan that the SQL optimizer produced, was a good query plan based on that set of data, and the statement is executed in negligible time. When this same query is executed again with a different set of parameters, since it was a parameterized query, we look up the cached plan and find it, thus use it. The cached plan is good for the first set of parameters, but BAD for the second set, and thus can cause your SQL server to consume the CPU trying to process the results based on the plan it generated previously.

The reason, in this case, why the plan is bad for the second set of parameters was because of the data distribution in the UserInfo/UserData tables. (For more information on how users are cached in the UserInfo and UserData tables, see my posting here: http://blogs.msdn.com/krichie/archive/2006/02/18/534767.aspx ) Because this single site has many users recorded against it via the domain group, it increases the likelihood of this problem. The majority of the sites have very few rows in the UserInfo table for a given site, while the minority has thousands of rows in it for their sites. Hence, the execution plan chosen for a site with very few rows was optimal for any site that had few rows, but when that query plan is used with a site that has a TON of rows it is not optimal for that at all.

I want you to realize, parameterized queries and query execution plans ARE NOT BAD in general. In fact they are good for most cases where there is even data distribution, yet can cause problems if the data distribution is uneven OR if typical parameters are passed initially when forming the plan that is cached.

Oh dear, whaddya gonna do....

I filed a bug with the SharePoint product group team on this, as it wasn't a problem with SQL, it was just how SQL works. The SharePoint product group and sustaining engineering teams both agreed we needed to fix this, it's just the proper fix needed to be thoroughly investigated, regression tested, stamped with approval, etc. We all knew this was going to take a really long time to get the best fix created. (Surprisingly, this didn't take as long as the orphan cleanup fixes I worked with many folks on to push forward :)) Due to timing, we didn't make SP2, so this was a post SP2 hotfix for Windows SharePoint Services, which I HIGHLY encourage and recommend that you deploy. You can get the fix here: http://support.microsoft.com/?id=900929

In the mean time, I had to help this customer. The solution was to move the single site with the large amount of users associated with it to a separate content database, what I call repartitioning or site leveling. Query Execution plans are database specific, so this would prevent the re-use of the plan on large sites.

Ah, that's a snap right? Well, in the case where the site in question may have been a single Windows SharePoint Services team site, I could have used STSADM -o backup/delete/restore to the target database to take care of this one site, and be done with it all..but guess what....It wasn't a team site that was the problem, but rather the Portals site collection. That was the one with the large user dataset. Guess what, you can't use STSADM for that :) Also, you can't use SPSBackup for this either, because SPSBackup backs up the entire content database, not just the portals site collection.

What's the only other way to do this? Repartition the other 9,999 team sites out of this content database, and into another.

So just how would you do that? Ah shucks that's easy...Here's the steps:

In order to repartition Site Collections in and out of content databases you have to use the following steps:

If you have not done so already:
1. Create a new content database to place the backed up site into.
Read/Write lock the site so that it cannot be accessed during the backup operation
1. Navigate to Windows SharePoint Services Central Admin page
2. Select Manage Quotas and Locks
3. Select Manage Site Collection Quotas and Locks
4. Type in the URL of the site you need to lock and click the “View Data” button.
5. Select the “No Access” radio button in the “Site Lock Information” section and choose OK.
STSADM –o backup (to backup the site collection to disk)
STSADM –o deletesite (To delete the site from the current content database)
Visit the Content Database management page from the Windows SharePoint Services Central Admin pages.
1. Select the First content database and either:

i. Set it’s status to Offline or

ii. Set it’s max-sites and warning-sites counts to that equal to the number of current sites in the content database (Be sure to have a notebook handy to write down all of the original values so that you can re-set them in a later step.)

Repeat step 4.a for each content database EXCEPT the one you are restoring to (This prevents the WSS Logic from load balancing the existing content databases, to determine which content database to restore the site in)

STSADM –o restore (restore the site into the new targeted database)
Read/Write unlock the site to allow access to the site again
1. Navigate to Windows SharePoint Services Central Admin page
2. Select Manage Quotas and Locks
3. Select Manage Site Collection Quotas and Locks
4. Type in the URL of the site you need to lock and click the “View Data” button.
5. Select the “Not Locked” or “Adding content prevented” radio button in the “Site Lock Information” section depending on the sites previous state, and choose OK
Visit the Content Database management page from the Windows SharePoint Services Central Admin pages.
1. Select the First content database and either:

i. Set it’s status back to Online (If it was online to begin with) or

ii. Set it’s max-sites and warning-sites counts to their previous values (refer to the notes taken in Step 4.a.ii)

Repeat step 6.a for each content database

Repeat steps 2-8 (unless you need to create more content databases, in which case you restart at step 1).

In other words, do these steps ~10,000 times. Let's see, would YOU want to do those steps manually? Ah shucks that's easy...NOT!

As you can see this is a very labor intensive and time consuming process. If you had to do this for many sites it becomes impractical to do all of this manually.

The Solution, and birth of SPSiteManager

I had to come up with a solution for my customer. Otherwise, I would not be able to sleep....It was my issue, my customer, and they were in some serious pain, and besides...my job at the time required me to work the issue to closure. They were experiencing the issue all the time, and the outage was a sustaining 3 minutes for every user who hit this one tiny little document library, on this VERY popular portal area :). So what would happen, is users would notice the problem, and then keep trying to click on this document library...As soon as it would render, another user would come along and hit it, thus...down again. They were quite literally down, completely down, until they prevented access to this document library. I knew that we could perform the same steps needed as in the manual steps above, via the SharePoint Object Model, so off I went, Codin' till the fingers go numb.

While I was writing the repartitioning logic, I knew we would have other customers eventually hit this before the fix, and it would also be a great tool to have around, in case you needed to move sites around regardless of the bug or not. I also wrote a simple detection algorithm, to analyze and detect where content databases may be in this state, thus the reason you see the lowusercount= and highusercount= attributes in the <database> element in SPSiteManagers Site Distribution Document. I worked with MSIT to get it run on our own Microsoft internal SharePoint servers, to look for the occurrence of the problem. The product group even gave me a mirror of their internal server (A really big heavy portal server) to work against to ensure we got the logic flawless for solving this type of problem. (And if I haven't told you enough PG, MSIT, etc.....Thanks!!!!)

Heck, since I had to go examine the sites, webs, etc..I thought what the heck, I'll even go add in the checks against the capacity planning guidelines we have for SharePoint. The result, was yet another little tool that went "Boom"

In the end, when the customer ran SPSiteManager (in batches) to repartition their sites out, they completed it in about 10 man hours. That's ~10,000 site collections moved in 10 man hours...Now, imagine the amount of time it would have taken to do the manual steps above.

The ultimate fix

Now before you go off running, and telling everyone that they must repartition their large user associate sites immediately, STOP NOW. You do NOT have to do this. As I noted above, a post SP2 hotfix is available to remedy this problem. See http://support.microsoft.com/?id=900929 .

I will tell you right now, though, that:

The KB does not necessarily call this out, and in fact is worded improperly

It's currently worded as:

"When you run an ad hoc query on a Windows SharePoint Services site, the CPU usage on the server that is running Microsoft SQL Server consumes 100 percent of the CPU time. Therefore, users cannot access the server."

Which is VERY misleading. This in by NO means whatsoever implies that you should go run ad hoc queries on your database.
We are working to get the official KB updated with the proper wording.

It IS RECOMMENDED however, that you keep a portals site collection in it's own database, as with a portal, it is very common that you would have a large set of users associated with it, and it can get quite large, and if you need to move it later, guess what you can't. You'll have to repartition all those team sites out of it, rather than re-locate the Portal site collection to a new database. What I would recommend, is that when you create a new Portal, go set it's "Max Number of Sites" to 1 in the Manage Content Databases page, then create a separate content database for your team sites.

In Conclusion

If you feel you may be experiencing this problem, and you do not have the Post SP2 hotfix installed as noted above, a couple of simple tests can be executed

While experiencing the problem, issue the following statement against the content database in question.

DBCC FREEPROCCACHE

If your hang is cleared up by this, chances are you are being impacted by this issue.

Use SPSiteManager to perform a simply analysis (Without the need to do a deep level analysis) and look at your user associates on your content databases:

SPSiteManager -o analyze -allvs
- If you see a lowusercount <= 1000 users and a highusercount >= 9,000 to 10,000 users, chances are you are being impacted by this issue.

To put it bluntly, install the fix now (or the most recent rollup hotfix which will include this.) :)

Hope this helps!!

- Keith

Previous Posts on this series:

Understanding SharePoint - Part 1 - Understanding the SharePoint Portal Server Indexer
http://blogs.msdn.com/krichie/archive/2006/07/20/672755.aspx

Additional References:

Description of the Windows SharePoint Services post-Service Pack 2 hotfix package: November 15, 2005
http://support.microsoft.com/?id=900929

SQL Server Architecture: Execution Plan Caching and Reuse
http://msdn.microsoft.com/library/default.asp?url=/library/en-us/architec/8_ar_sa_4azp.asp

SQL Server Books Online: DBCC FREEPROCCACHE
http://msdn2.microsoft.com/en-us/library/ms174283.aspx

Understanding SharePoint - Part 1 - Understanding the SharePoint Portal Server Indexer

krichie — Thu, 20 Jul 2006 17:44:00 GMT

Understanding SharePoint - Part 1 - Understanding the SharePoint Portal Server Indexer

This is the first in a series of posts in regards to "Not being mislead by what your seeing :)"

In the next post in this series, I'll talk about The Infamous Query Plan Bug, and the origins of SPSiteManager

Be careful not to misinterpret what your seeing in regards to the SharePoint Portal Server Indexer.

I recently worked with a customer who was experiencing "well appeared to be experiencing" poor crawl performance from SharePoint Portal Server 2003. I'll be following up to this posting with results from the MOSS 2007 implementation of Search, but I'm sure they results will be pretty much the same.

The key here is to ensure that you are not misinterpreting the data you are being presented with. Keep the following two items in mind when examining your own portal

Item 1: Certain IFilters can lead you to think that your crawler is not working effectively, yet some can actually have a dramatic impact on performance

Crawl rates can easily be misinterpreted when you have 3rd party iFilters installed. This section will show you some guidance we give in relation to the impact that certain iFilters can have on your environment. I am "NOT" saying that these iFilters are "BAD", and you should not misinterpret me saying that :) I'm just noting the impact that they can have on your environment that you need to be aware of :).

Tests were run using a standalone Dell Precision 470 Intel XEON Server with CPU running at 3.0 GHZ. Total physical memory on the Server was 3.0 GB. SharePoint Portal Server and Windows SharePoint Services were both installed at Service Pack 2. This is a single Server deployment of SharePoint with SQL Server 2000 at SP4 loaded on one Server.

Performance Counters Used

SearchGathererProjects\Processed Documents Rate (Hence known as PDR in this document)

This counter identifies “The number of documents processed per second”

A standard set of test files was uploaded to both the default document library at the root Portal Area level and a test Windows SharePoint Services site document library at such as http://server/sites/testsite
The set of seed files used for testing consisted of 18 Zip files (which resulted in about 800 individual files unzipped, and a variation of .txt, .doc, .ppt, .xls, etc), 16 PDF files, and 8 standalone Word documents.
Initial testing with no 3rd party iFilters installed resulted in a PDR of around 22 documents per second processed with a maximum of around 116.
The addition of the Adobe iFilter at version 6.0 resulted in an approximate 40 to 50 percent performance reduction. The average PDR dropped to around 9 with a maximum of 72.
A Zip iFilter at version 1.2 from the I-Filter Shop was loaded and the Adobe iFilter was removed. This resulted in about the same performance hit as the Adobe iFilter caused.
The Zip iFilter only added a total of 18 items to the index, so the performance hit is misleading. In reality it added 800 unzipped items to the index.
With both the Zip iFilter and Adobe PDF iFilter installed and added to the list of file types to be crawled performance dropped again by another 10 to 15 percent resulting in a total performance hit of around 65 percent. Again, this is a bit misleading as the Zip iFilter actually added an additional 800 items to the index that only appear as a single item in the total for the Search catalog.

Performance Conclusion

Installation of the PDF iFilter will have the most adverse impact on indexing performance due to the fact that it is a single-threaded iFilter.
Although we appear to take a large performance hit with the Zip iFilter, performance may not be hampered due to the fact that we are actually adding numerous items to the index instead of only the one that appears in the index item count.
The Zip iFilter provided by the I-Filter Shop does indeed run as a multi-threaded filter.

Item 2: Don't misinterpret your gatherer logs.

You may find many "Delete" entries in your gatherer log for URLs that no longer exist, but the delete entries happen all the time, and it appears to never remove the entry from the index.

By default, SharePoint will not automatically remove an index entry until 3 consecutive crawls have occurred (whether it be full, or incremental). The reason for this, is we don't want to just remove the entry, as there may have been a network issue temporarily, and if we did this for every crawl, it could mislead users. For instance, if you had an alert on a document library, the user would constantly see "New content found" alerts for the same document between crawls because of a flaky network connection. Thus, if we can't connect to the target source after 3 attempts, then we consider it a dead link.

What you may actually be seeing is hits to sites from the portals "Sites" content source. Regardless if there is an entry in the index or not, if you have a site listed in the site directory, it will "ALWAYS" be checked from the crawler. In this case, the site is gone, so every attempt is going to fail. Thus, after 3 times, it sends a Delete transaction to the index to remove it from the index. The same would be true to a content source that has a target url to a site that no longer exists. Until you get rid of that site reference, or the content source, you will continue to see those entries in the gatherer log.

I've added some new features into SPSiteManager just for this reason, which allow you to clean up dead entries from your site directory and list of sites to be crawled.

Hope this helps!

- Keith