Tony McIntyre [MSFT]

SharePoint Portal Server 2003 Crawl Performance Part 8

IFilters

IFilters do a lot of the heavy lifting for the daemons during a crawl. The IFilter specification documents how to write an IFilter. Once an IFilter is written and deployed then it can be used by various crawling engines, including SharePoint.

In this post we will discuss how to determine if an IFilter is marked for use with a single-threaded daemon or a multi-threaded daemon.

First we need to understand why single-thread .vs. multi-threaded is important. It all comes down to how many of a certain type of document can be crawled by the gatherer at one time. When the gatherer first starts up daemons to do the crawling of items, it will start up what is called a multi-threaded (MT) daemon. This type of daemon can crawl up to 32 items at the same time. This means that many documents can be processed in a short amount of time. The other type of daemon is the single-threaded (ST) daemon. This type of daemon can crawl 1 document at a time.

When dealing with files, the extension will determine the specific IFilter that will be used. The IFilter's registration will determine the daemon that is chosen to execute it. If an IFilter is thread safe and can operate in a MT daemon and is registered appropriately then it will be loaded in a MT daemon. Otherwise it will need to be loaded in the ST daemon.

How to determine the threading model of an IFilter

When an IFilter is registered on a system, the threading model is specified so that it is published to all consumers if the IFilter is thread safe or not. You can inspect your own registry to determine the IFilter for a specific file type and if it is thread safe or not.

1. Start out on the Indexer machine and open the registry. You will need to select the HKEY_CLASSES_ROOT to begin. Since we are working with .DOC files, we need to navigate to the HKCR\.doc\PersistentHandler path. This path shows a (default) key that is of type REG_SZ. There is a GUID listed here. For the purpose of this discussion, I will simply refer to this a GUID1. You will need to write down this GUID for the next step
2. Next, you will need to navigate to HKCR\CLSID\{guid1} path. Once at this location you will see a node below this level called PersistentAddinsRegistered. Beneath that level you will see another node that is a GUID. We will call this GUID2.
3. Click on GUID2. This path shows a (default) key that is of type REG_SZ. There is yet another GUID listed here. We will call this GUID3. You will need to write this GUID down for the next step.
4. Armed with GUID3, you will need to navigate to the path HKCR\CLSID\{guid3}\InprocServer32. Here you can see the following
1. (default) which is the DLL that contains the IFilter implementation.
2. ThreadingModel which is the thread safety designation
1. Both indicates that this DLL can be used by both the ST and MT daemons
2. Apartment indicates that this DLL can only be used by the ST daemon
3. Free indicates that this DLL can be used by both the ST and MT daemons

Example using the .DOC extension

For this example, I will look at the Microsoft Word document type(.DOC) but this method will work for all document types.

1. I navigated to HKCR\.doc\PersistentHandler and found that the (default) key had a value of {98DE59A0-D175-11CD-A7BD-00006B827D94}. This is GUID1.
2. I navigated to HKCR\CLSID\{98DE59A0-D175-11CD-A7BD-00006B827D94}. I saw the PersistentAddinsRegistered and the GUID beneath it. The GUID beneath was {89BCB740-6119-101A-BCB7-00DD010655AF} and this is GUID2.
3. I navigated to HKCR\CLSID\{98DE59A0-D175-11CD-A7BD-00006B827D94}\PersistentAddinsRegistered\{89BCB740-6119-101A-BCB7-00DD010655AF} and found that there was a (default) key that had a value of {F07F3920-7B8C-11CF-9BE8-00AA004B9986}. This is GUID3.
4. I navigated to HKCR\CLSID\{F07F3920-7B8C-11CF-9BE8-00AA004B9986}\InprocServer32 and found two entries at this path. I found that the offfilt.dll is the DLL that has the IFilter implementation in it. I also found that the ThreadingModel is Both.

Real life

You will find that there are some IFilter DLLs that are marked as Both but should be marked as Apartment. The result of this incorrect assignment is unpredictable. In SharePoint, you may see the MT daemons crash frequently producing an event in the Application Event log with an ID of 1000. This is the first sign that there is a thread safety issue. I have seen other IFilters that simply produce different results each time a crawl runs. This typically will present itself when a user has an alert for a document change when no change was made to the document. This symptom does not always point to a thread safety issue with an IFilter but it can.

You should contact your IFilter vendor to determine if their IFilter is thread safe or not. You really want to have IFilters that are truly thread safe so that they can be used by the MT daemons. This in itself will help with your overall crawl performance. But, if an IFilter is incorrectly marked it can cause many performance problems due to crashes.

In a later post we will more items on tuning.

SharePoint Portal Server 2003 Crawl Performance Part 7

Site Hit Frequency Rules

The Site Hit Frequency rule (SHF) is a very powerful tool for performance. As I have come to learn, many people do not really understand it or how to use it to help them out on performance tuning.

When tuning the gatherer it is important to understand that it is not always about speed. We would all like to have the crawl to complete in less than 1 hour but depending on the size of your corpus this is not always something that is achievable.

The SHF rule is something that will help with underpowered target servers. For example, one of the things that we crawl internally is a very underpowered machine that physically resides in Germany. This is a web server that is under someone's desk and it's probably running a 500-600 MHz processor. It does fine for what it is doing but it can't withstand the full impact of the gatherer crawling it. So a SHF rule was created to ease the impact on this machine.

The Request Frequency has three choices on the screen and each have their own specific use. The three choices are

1. Request documents simultaneously
2. Limit the number of documents requested simultaneously
3. Wait a specified amount of time after each request

We will talk about these in reverse order. The option to Wait a specified amount of time after each request does just that. If you put 30 seconds in there, the gatherer will pull one document and then after it completes, the gatherer will wait for 30 seconds before getting the next document. This is the SHF rule that was created for the machine in Germany. It works out well because the machine eventually gets all of its content crawled and the machine is not over taxed.

The option to Limit the number of documents requested simultaneously again, does just that :> . If you set the number of documents to 3 then the gatherer will request 3 documents at the same time and no more. This limits the concurrent connections to this specific server. This is an important concept especially as we move to the 3rd and final option. You could use this to throttle down the number of connections to a server or remove the throttle altogether causing the gatherer to hit the server as hard and fast as possible. This is where the power starts to come into play as you will see shortly.

The option to Request documents simultaneously is the default and it can overpower some servers so use it sparingly. This item will pull as many items from the target server as it possibly can at any one time. Using this option will instantly increase the number of open connections to the server depending on the available filtering threads.

If you read part 4, then you remember the discussion on the Search Gatherer\Server Objects. The Sever object is the current governor. When a server object is created it will set a default maximum number of connections to the server itself. This number is calculated as 4 + the number of procs on your indexer. So in my case, I have a dual proc machine so if I am crawling my machine it will be 4+2=6. The server object will limit the number of concurrent connections to 6 at any one time. This was done for very valid reasons. The crawler wants to be a good corporate citizen and not overpower target machines during a crawl. This is a reasonable default to get the data quickly and not to overpower the target.

If you think about this, it can cause some interesting behaviors in the perfmon counters. For example, if I am only crawling my one server and  I have 32 filtering threads. Assuming all of my content was self contained on that machine, I would see 6 threads that were active and the others would be idle. As with all performance tuning, there is always a bottleneck, this time it is the default maximum connections in the server object.

The SHF rule can be used to override this maximum connection limit. Here is where the tuning becomes crucial. If your indexer is already peaking out on the CPU or disk utilization using the Request documents simultaneously option would be bad for you. The reason for that is that if you set this option, it will increase the stress on your indexer machine. This may or may not be what you want to do. I would suggest that you experiment and document what you do to understand what is best in your environment.

The way that I tested this was with Limit the number of documents requested simultaneously option. I chose to put a 10 in that SHF rule. Then I tested. I noticed that my machine could handle it just fine. So I increased it slowly up to about 32 without any difficulty. So, once I hit that number, I just converted it to the Request documents simultaneously option. When crawling a SharePoint site that is on a larger "beefier" machine it is ok to do this.

If you set the Request documents simultaneously option for a file server, you will notice that the impact on the file server and the indexer will be very noticeable. Again, you have to tune these in your environment. I was working with one customer that had very large machines that could handle the load but they were restricted by the default maximum number of connections that is imposed by the server object. They had recently migrated several different SharePoint farms into one farm. This meant that the connection limit imposed by the server object was too restrictive for their environment. We added a SHF rule and found that the gatherer started putting the idle filter threads to work. We also found that the gatherer was picking up the speed in which it crawled items. The machine was not greatly impacted at all but we needed to ensure that the other machines were still doing ok.

When the gatherer is crawling a SharePoint site, it is using web services on the Web Front End machine so you will need to watch that machine to ensure that the additional traffic is not too much for that machine to handle. Also, you are increasing the work that is being pushed to SQL server so you will need to monitor that as well.

If used properly the site frequency hit rule can be a great asset in performance tuning but it can also be a great enemy, so be aware of the impact to other machines when using this rule.

In a later post we will discuss additional items for use with tuning.

SharePoint Portal Server 2003 Crawl Performance Part 6

SQL performance

I think it goes without saying that if performance of the SQL server is not running at its highest then it will definitely slow things down in SharePoint. However, there are times that even if SQL is running at its highest that actions that occur in SharePoint can cause SQL not to run as fast as it can when we are doing a crawl. I noticed this while testing a theory that I had.

I had just recently added quite a bit of data to a SharePoint site for the purpose of performance testing and found that things were running a bit slow for my tastes. I was reading an outstanding book called Microsoft SharePoint Products and Technologies Resource Kit If you don't have it, I highly recommend it as it will really fill out some of the things that have been covered in these postings. I read in this book in Chapter 10 around page 263 that several SQL commands could be issued to help with performance.

Before showing these commands it is very important to understand that updating the SQL database that is used for SharePoint is not supported. These commands are performing SQL maintenance and these specific commands have been allowed by our development team. Before doing anything in our database, it is very important that you have read kb 841057 and understand it before you proceed.

The SQL updates that are documented in the Resource kit are as follows. I have taken the liberty to add a few extra commands as well.

• use sps2003p1_site -- your site database
  • UPDATE STATISTICS Personalization
  • UPDATE STATISTICS UserInfo
  • UPDATE STATISTICS WebMembers
  • UPDATE STATISTICS Sites
  • UPDATE STATISTICS Webs
  • UPDATE STATISTICS Lists
  • UPDATE STATISTICS WebParts
  • UPDATE STATISTICS Docs
  • DBCC DBREINDEX (Personalization,'',0)
  • DBCC DBREINDEX (UserInfo,'',0)
  • DBCC DBREINDEX (WebMembers,'',0)
  • DBCC DBREINDEX (Sites,'',0)
  • DBCC DBREINDEX (Webs,'',0)
  • DBCC DBREINDEX (Lists,'',0)
  • DBCC DBREINDEX (WebParts,'',0)
  • DBCC DBREINDEX (Docs,'',0)
• use sps2003p1_serv -- your serv database
• UPDATE STATISTICS srch_gathererlog_1
• UPDATE STATISTICS srch_gathererlog_2
• UPDATE STATISTICS srch_gathererlog_3
• DBCC DBREINDEX (srch_gathererlog_1,'',0)
• DBCC DBREINDEX (srch_gathererlog_2,'',0)
• DBCC DBREINDEX (srch_gathererlog_3,'',0)

The commands above that are for your _serv database have to do with the gatherer log tables. You may have a different number than what I have shown here, you will need to look at your database to determine if you have more than the 3 that I have listed here and add or remove commands as appropriate for your database.

When running these commands and then doing another full crawl my crawl time reduced over 1 hour in run time, your mileage may vary. I would recommend that these be run frequently, perhaps weekly or monthly depending on the amount of data that you are adding/deleting to the farm.

In a later post we will discuss additional items for use with tuning.

SharePoint Portal Server 2003 Crawl Performance Part 5

Now that we have discussed some of the main counters. Let's take a closer look to see how to use this information to improve our crawl performance.

Resource Usage/Performance Level

The value seen in the Search Gatherer\Performance Level directly correlates back to the Resource Usage setting found in the Central Administration pages for Search. This selector ranges from Background - Dedicated and it shows up in this counter with the range of 1-5. The default is the center 'dot' in this setting, which equates to 3.

This setting configures the gatherer to better use the resources you have made available. The specific resources I am referring to are the memory, CPU, etc. How "beefy" is the machine. The bigger it is the higher you will want to make this setting. It is possible however that if you are running on a less than optimal box that you will want to reduce this number as not to punish your index server too much. That is the fun of tuning.

Assuming that you have a large "beefy" machine you will want to move the Resource usage to the 4th selection at a minimum. The biggest change that occurs here is that you allow all of the filtering threads to be started at a normal thread priority rather than below normal thread priority. Moving to the 5th selection or dedicated does not increase this priority setting any more.

By increasing the Resource usage to a 4 or 5 will also serve to increase the maximum number of Filter threads that are allowed on the indexer machine. A setting of 4 will allow for up to 48 filtering threads to be created on the machine and a setting of 5 will max out the machine at 64 possible filtering threads. Having more threads potentially means that more items will be crawled at one time but does not guarantee that you will always use 48 or 64 threads at all times. More on this later in a later posting.

Document, Document, Document...

Before making any of these changes you will need to document your current crawl and how it is running so that when you make changes you can determine what impact these changes have in your environment. There is no silver bullet to performance tuning with the gatherer but if you take methodical steps you can increase it to get the greatest performance in your environment.

When I start working on a machine trying to tune it I work to document many things in the environment and then I track what I have changed and how it effects the overall crawl performance and time to run.

The things that I collect include the following

• Resource Usage/Performance Level
• Site Hit Frequency Rules
• Timeout settings - Connection
• Timeout settings - Request Ack
• Current crawl time from the gatherer log summary (Site Settings->Configure Search and Indexing->Manage Content Indexes use View gatherer log menu option on the specific index that you are working with)
• Documents in Index (Site Settings->Configure Search and Indexing->Manage Content Indexes)
• Content Index size (Site Settings->Configure Search and Indexing->Manage Content Indexes)
• # of documents marked for Retry (Site Settings->Configure Search and Indexing->Manage Content Indexes use View gatherer log menu option on the specific index that you are working with)
• Unique errors in the gatherer log summary and # of each occurrence (Site Settings->Configure Search and Indexing->Manage Content Indexes use View gatherer log menu option on the specific index that you are working with)

I like to document each of these things so that I know if a change will increase or decrease the time it takes to crawl, or changes the number of errors, etc.

In a later post we will discuss items for use with tuning.

SharePoint Portal Server 2003 Crawl Performance Part 4

To continue the discussion of Crawl Performance, we will look at some additional counters that will help with further analysis of state of the crawler.

Here we will expand on the counters we have reviewed before and see what information can be collected from them.

What are the counters trying to convey?

In the Search Gatherer object some additional counters offer a glimpse into the bigger picture. We will also look at few counters in the Search Gatherer Projects performance object. This particular performance object is associated with the individual content index projects that you have created. For example, Portal_Content and Non_Portal_Content. For the purposes of this discussion I have chosen Portal_Content.

• Search Gatherer\Performance Level
  • This is a very important indicator of the number of resources that can/will be consumed by the crawler. The range of values is between 1 and 5, and the default value is 3.
  • This number impacts the number of single-threaded and multi-threaded daemons (mssdmn.exe) that will be started. If the number is set to 1 or 2 then the number of daemons that will be started is reduced greatly. This way the crawling does not have a huge impact on your server. Another important thing to point out here is that this number also has an effect on the thread priority of the filtering threads. When a setting of 1-3 is used the filtering threads are started with a below normal thread priority. This means that the filtering threads are not getting as many cycles as you may want or desire.
  • It can also have an impact on the multi-threaded daemon by adjusting the number of threads that are available for calling IFilters. As the Performance Level goes down the fewer the threads available for calling IFilters.
• Search Gatherer\Server Objects 
  • This counter reflects the number of servers that the gatherer is accessing at one time during a crawl. What this means is that when a server is accessed for the first time a server object is created. A server object would be created for each unique host server that you are configured to crawl. For example, if you were configured to crawl http://mysharepoint and http://www.microsoft.com, you would have at least 2 server objects created. More would be created if there were links to other servers.
  • That server object holds many pieces of information about the server that is be contacted. One of the things that this object holds is the number of concurrent connections that are allowed. In a later post we will discuss the Site Hit Frequency rule and how that impacts this connection limit.
• Search Gatherer\Threads Accessing Network
• This counter shows the number of available Filtering threads that are currently accessing the network. When a filtering thread receives the next item to crawl that thread is responsible for loading the protocol handler responsible for connecting to the target. If the target of the item to be crawled is on a file share, then we will load the protocol handler that is responsible for connecting to file servers.
• Depending on the item type, it is possible that the item will be copied from the target machine to the local machine in the farm, specifically the indexer that is requesting the item.
• The item will be stored in the temporary directory on the indexer machine. This directory is typically located at <drive>:\Program Files\SharePoint Portal Server\Data\temp. This is configured in the Central Admin service within the Manage Server Settings-> Search Server Settings page. The File Locations section allows you to set the temporary path. It is VERY important that you do not have anti-virus scanning enabled for this directory. Doing so will only serve to slow down the overall crawl. You need to ensure that you don't have any infected files contained within SharePoint but this can be done on upload and using a virus product that can integrate with SharePoint. Many of these are available and you should find one that fits your needs.
• During the time that the item is being copied from the target, this counter is incremented. When the copy has completed and we are no longer hitting the network this counter is decremented.
• After the item is on the local drive in the temp directory, this thread will load the IFilter and begin calling the GetChunk method to extract the contents of the item. This work is performed by the daemon and as the data is fed back to the MSSearch process for final processing, this counter is incremented as well. So what this means is that as long as data is coming back from the daemon and being fed back upstream to the MSSearch process, this counter will reflect that activity. So it in essence will show that the thread is still accessing the network when it is really not impacting the overall network. This is due to the design of the counter.
• Search Gatherer\Threads In Plug-ins 
  • This counter shows the number of threads that are in Plug-ins at any one time. The designers of Search made two very cool choices, 1) make the daemons sacrificial to account for misbehaving IFilters and 2) make MSSearch extensible by designing plug-ins to handle various functions.
  • One of these plug-ins is the Subscription Plug-in (SUBPI). When data is fed back upstream from the daemon certain actions must be taken to complete the overall work for the gatherer. One of these functions is to create SPS alerts, not WSS alerts as these are done with a very different mechanism. That is the function of this plug-in.
  • Too much time spent in this counter may indicate a slow connection to the back-end SQL server. It is expected that this counter will increment/decrement regularly during a crawl.
• Search Gatherer Projects\Crawls in Progress
  • This counter indicates that a crawl is in progress. It does not tell you what type of crawl it is but it at least clues you into the fact that there is a crawl running.
  • When looking at perform data from a historical perspective, you are able to determine the start and stop times of the crawl by using this counter.
  • Many times when customers call in they know that a crawl is running but they don't know which index is crawling because they have so many of them. This counter helps us to focus in on which specific one is running.
  • I use this counter to restrict the view in the perform capture when looking at historical information. For example when I have 3-4 crawls in one perfmon log, I can move the start and end time bars in Perfmon to include just the data for a specific crawl.
• Search Gatherer Projects\Incremental Crawls 
  • This counter indicates that the crawl that is currently running for this Project (Index) is an Incremental crawl. When the counter is 1 then it is an incremental when it is a 0 then it is a full crawl or perhaps an adaptive crawl. I have not seen many people run the adaptive crawl so you won't see much of that here.
  • Again this counter is used to identify the crawl in question when looking at historical information. It is also a good sanity check when looking at live data to determine if it is a incremental crawl or full crawl that is currently running, if you don't already know.

In a later post we will discuss additional counters and what they indicate.

SharePoint Portal Server 2003 Crawl Performance Part 3

To continue the discussion of Crawl Performance, we will look at some additional counters that help with further diagnosing the state of the crawler.

Here we will expand our subset a little bit and see what information can be collected from them.

What are the counters trying to convey?

In the Search Gatherer object some additional counters offer a glimpse into the bigger picture.

• Search Gatherer\Filter Process Created
  • This counter will show you the number of filtering processes or daemons (MSSDmn.exe) that have been created since the last time MSSearch.exe has been recycled or the machine has been rebooted. This is an interesting number to watch because it can be an indicator of the health of the IFilter's on your system. For example, if you have an IFilter that is prone to crashing, it will take the filter process down and MSSearch will need to start a new one. This is bad when you have a lot of content that needs to be crawled and the daemons keep crashing. When I say high for this counter, I am referring to over 300-400 have been created in a 24 hour period. It is possible that you have a lot of different crawl schedules that will cause daemons to start and stop and if this is the case then you don't necessarily have a problem. If you feel the number is too high, check the application event log. If the daemons are crashing, they will typically leave an event id 1000 in the event log. If you have the Dr. Watson reporting features enabled, you will also see a corresponding event id 1001 logged after each event id 1000. These application event IDs 1000 and/or 1001 are helpful in determining the IFilter that is crashing.
• Search Gatherer\Filter Processes 
  • This counter will show you the number of filtering processes or daemons (MSSDmn.exe) that are currently running on your machine. This is helpful to reference when you are trying to get a feel for the amount of effort the MSSearch process believes that your machine should exert to perform the crawl. This number is somewhat tied to the Search Gatherer\Performance Level counter that we will discuss later in some depth.
• Search Gatherer\Filter Processes Max
  • This counter will show you the maximum number of filtering processes or daemons (MSSDmn.exe) that have been started since the last reboot or the last recycle of the MSSearch process. I really only use this counter as a reference.
• Search Gatherer\Filtering Threads 
  • This counter combined with the next counter will directly impact the performance of your crawl. Each thread listed here is the equivalent to one IFilter and one main document to be crawled. If this number is low (2-5) then you will have a longer crawl than if this number were higher(14-30+). Think of each of these threads as a line at the grocery store. If there are thousands of people in the store and only 2 people that can exchange money at the check out station, then all of the people will stack up and wait their turn. In there are 30+ people in the store that can exchange money at the check out station, then the lines will get cleared out more quickly.
• Search Gatherer\Idle Threads
  • This counter indicates the number of filtering threads that are idle. So if you have 32 Filtering Threads and 0-2 of them are idle, then you are still filtering 30-32 documents at one time. If all 32 are idle and you have documents to be crawled, it is possible that you have a problem that needs to be investigated further. This investigation would start in the application event log and then move to the diagnostic logging provided by the product. These diagnostic logs are accessed via the Central Administration page and are under the Diagnostic Logging link at the bottom.

In a later post we will discuss additional counters and what they indicate.

SharePoint Portal Server 2003 Crawl Performance Part 2

Where to Start

I prefer to work with real time data but the same thing can be done with perf logs. In part 1 (see SharePoint Portal Server 2003 Crawl performance part 1 I listed each of the perf counters that should be logged and the ones that we are focusing on for this part is a subset of those same counters. In this discussion, I am making the assumption that the environment (network, disk drive, etc) is working without issue and that the suspicion is with the crawler.

I like to start with this subset and then move forward from there based on what is presented in these counters.

What are the counters trying to convey?

In the Search Gatherer object there are a couple of key counters that I always review.

• Search Gatherer\Document Entries
  • This counter represents the current number of items (documents, lists, events,etc) that are in the queue to be crawled.
  • When this counter is > 0 the daemon(s) recognizes that there is work to be done and crawling starts.

• Search Gatherer\Delayed Documents
  • This counter represents the number of Search Gatherer\Document Entries that exceed the Site Hit Frequency rule and need wait their turn to be crawled. This is a normal function of crawling.
  • This is more of an informational counter and initially I don't really take any action on it but it is good to know the number during the crawl. In a future post we will dive into this counter a little deeper to really understand what is represented here.

• Search Gatherer\Documents Filtered and \Documents Filtered Rate
  • These two counters work together. The Documents filtered counter indicates the number of documents that were filtered by a daemon. This does not indicate success it only indicates that the document was marked as completed.
  • When an item is crawled it is typically processed by some sort of IFilter. Not all items have an IFilter but are processed by something like an IFilter. For example, when you import users from Active Directory, there is not a people IFilter so another piece of code handles this specific item. The bottom line is that when an item is completely crawled, this counter is incremented.
  • The Documents Filtered Rate is the number of Documents Filtered per second. This number is interesting when estimatating the duration of a crawl. For example, one could estimate the remaining crawl runtime by doing the following
    • Estimated Seconds Remaining = Search Gatherer:Document Entries / Search Gatherer:Documents Filtered Rate
    • Estimated Minutes Remaining = Estimated Seconds Remaining / 60
  • Using this estimation technique one could get a feel for the remaining time. For example using the data in the above graphic you would get the following
    • 3249 (Document Entries) / 4 (Documents Filtered Rate) = 812.25 (est. seconds remaining)
    • 812.25 (est. seconds remaining) / 60 (seconds in a minute) = 13.53 (est. minutes remaining)
  • I can't stress enough that this is an estimation. The reason is that the number of documents that the crawler can filter per second can vary wildly. I have seen some customers get 140-150 per second and then drop down to 2 per second. Another reason that this could change is if the cralwer encounters a document that has many links in it, this will cause the Search Gatherer\Document Entries counter to increase. So it is more of a guesstimation of the time it will take than a reliable time.

• Search Gatherer\Documents Successfully Filtered and \Documents Successfully Filtered Rate
• These two counters also work together. The Documents Successfully filtered counter indicates the number of documents that were filtered by a daemon and done so without an error being returned from the IFilter.
• The bottom line here is that when an item is completely crawled and no error was returned, the Documents Successfully Filtered counter is incremented.
• The Search Gatherer\Documents Successfully Filtered Rate is the number of Documents Successfully Filtered per second.
• Search Gatherer\Heartbeats
• This counter is the basic heartbeat of the search process. It is incremented every 10 seconds. I normally use this counter to determine how long search has been up and running as all of the counters are cumulative since the last time search was started

In a later post we will discuss additional counters and what they indicate.

SharePoint Portal Server 2003 Crawl Performance Part 1

From time to time, I get asked about how to determine why the SPS 2003 crawler/gatherer take a long time to crawl data and build the index. There are many reasons for this but it takes some digging to really determine the exact cause. In this series I will endeavor to explain steps to isolate and improve the performance of the gatherer.

Hot Fixes 

We need to start with the low hanging fruit first. We need to ensure that your farm is in a good state prior to performance tuning the gatherer. The first part of this is to ensure that you are running a current hotfix level for SPS and WSS 2003. While these hotfixes will not fix all problems they will help the overall health of the gatherer in your farm. I recommend that you at least apply the following hotfix packages:

1. Windows SharePoint Services Service Pack 2
2. SharePoint Portal Server Service Pack 2
3. Post SP2 Hotfix Roll-up Package KB900929
• This fix corrects a problem where the CPU usage on the server that is running Microsoft SQL Server consumes 100 percent of the CPU time, therefore users and the gatherer cannot access the server.
4. Post SP2 Hotfix Roll-up Package KB925380 
• This fix corrects a problem where pages can take 30 to 50 seconds to display. This was primarily noticed by users reporting a long time to render but it also impacted the gatherer.
5. Post SP2 Hotfix Roll-up Package KB924934
• This fix corrects a problem that was found in the tquery.dll that was causing mssearch.exe to crash. This is localized and the hotfix that is referenced here is for English. If you are not running English, you can contact Microsoft to request the corresponding hotfix for your desired language.

Registry Key Changes

There are some registry key changes that you may want to include as a part of the environmental configuration. These are not required but they make it easier for you to review the data. If you choose to perform these registry key changes you will need to do them on the server(s) that is/are acting as the Index server(s) in the SPS Farm. Since we are talking about registry keys, it is important to reiterate the words from Microsoft regarding updates to the registry.

Warning Serious problems might occur if you modify the registry incorrectly by using Registry Editor or by using another method. These problems might require that you reinstall your operating system. Microsoft cannot guarantee that these problems can be solved. Modify the registry at your own risk.

1. This registry key provides the default thousands separator in the data where appropriate.
• HKEY_CURRENT_USER\Software\Microsoft\SystemMonitor
• Add a new DWORD value named DisplayThousandsSeparator and give it a value of 1.
• Refer to KB300884 for more detailed information on this procedure
2. This registry key displays the Process ID with the Process Names where appropriate.
• HKEY_LOCAL_MACHINE\System\CurrentControlSet\Services\PerfProc\Performance
• Add a new DWORD value named ProcessNameFormat and give it a value of 2.
• Refer to KB281884 for more detailed information on this procedure
3. This registry key displays the Thread ID associated with Thread counters where appropriate.
• HKEY_LOCAL_MACHINE\System\CurrentControlSet\Services\PerfProc\Performance
• Add a new DWORD value named ThreadNameFormat and give it a value of 2.
• Refer to MFartura's blog for more detailed information

Setting up Perfmon

When troubleshooting performance problems with a crawl, it is essential that you collect the right performance counters on the Index server(s). Here is a list of counters that you should collect for these types of problems.

• \ASP.NET\*
• \ASP.NET Applications(*)\*
• \Cache\*
• \LogicalDisk(*)\*
• \Memory\*
• \NBT Connection(*)\*
• \Network Interface(*)\*
• \Objects\*
• \Paging File(*)\*
• \PhysicalDisk(*)\*
• \Process(*)\*
• \Processor(*)\*
• \Redirector\*
• \Server\*
• \Server Work Queues(*)\*
• \Search\*
• \Search Archival Plugin(*)\*
• \Search Catalogs(*)\*
• \Search Gatherer\*
• \Search Gatherer Projects(*)\*
• \Search Indexer Catalogs(*)\*
• \Search Schema Plugin(*)\*
• \Search Topic Assistant(*)\*
• \SharePoint Portal Alerts Notification Service(*)\*
• \SharePoint Portal Server Alert Manager(*)\*
• \SharePoint Portal Server Alerts Plug In(*)\*
• \System\*
• \Thread(*)\*
• \Web Service(*)\*
• \Web Service Cache\*

If you are not familiar with the (*)\* and the \* designations they mean the following

• \*   all counters
• (*)  all instances
• (*)\*  all counters and all instances

These counters are specific to SharePoint Portal Server 2003. These counters do not include any perf counters for SQL. The sample interval for these counters will depend on how long it takes to reproduce the problem. If it is a long running crawl you might want to set it to 15 secs per sample. It could be argued higher or lower but this should give you a reasonable picture of what is going on at the time of the crawl.

In a later post we will discuss the counters and what they indicate.

About Me

I am currently a Senior Escalation Engineer for the SharePoint Portal Server and Windows SharePoint Services team. I started with Microsoft in Feb. 2000 as an Escalation Engineer (EE). An EE is typically the last line of support for most products within Microsoft. EE's are tasked with the most demanding and technically difficult problems within the product. We spend time isolating code problems and working with our development team(s) to correct these problems.

In addition to the normal EE responsibilities, I also work with my development team(s) and the Supportability teams to find ways to improve the overall support experience with SharePoint. I also spend a good deal of time mentoring and coaching others.

From a personal perspective, I am blessed to be married to wonderful woman named Toni. I balance my free time between Toni, Church, woodworking and XBOX.

Tony McIntyre