Steve Sheppard's Blog : Documentation

How do I tune the MOSS Object Cache for performance and economy?

steveshe — Thu, 12 Mar 2009 22:36:00 GMT

Tuning the size of the MOSS Object Cache is done via the Max Cache Size settings on the Site Settings > Object Cache Settings page. It is important to recognize that this maximum cache size setting is a limit and not a static value. For example; Just because the maximum cache size setting for the Object Cache is configured to 100MB does not mean that it will consume 100MB of memory at startup. It simply means that if the cache were to exceed 100MB it will be compacted to reduce its memory consumption to a level below that maximum value.

For purposes of this discussion we will categorize cache compaction rates into three categories, unacceptable, acceptable and optimal. A cache compaction rate of more than 6 per hour should be considered unacceptable. A rate of between 2 and 6 compactions per hour is acceptable and between 0 and 1 cache compactions per hour is optimal. You should only target the optimal level of cache compactions if you have sufficient amount physical memory installed on your servers to achieve this goal and still have sufficient free physical memory remaining on the server to support other critical system operations.

You should regularly monitor for cache flushes because they are extremely expensive in terms of cache performance impact. A cache flush results in the ejection of all cache contents. Cache flushes are triggered by the creation, deletion or moving of a web. They can be monitored via the "SharePoint Publishing Cache/Total number of cache flushes" counter in Performance Monitor. If you are seeing cache flushes throughout the production day you should reconsider how you manage webs during those hours.

Based on these considerations we have developed the following guidance for how to tune the Object Cache size for acceptable performance using the "SharePoint Publishing Cache / Total number of cache compactions" performance counter.

We feel this level of cache performance will meet an economical customers performance needs without unduly sacrificing their limited memory resources. The steps to achieve this are fairly straightforward and must be applied in an iterative fashion until the desired level of performance is achieved. The recommended steps are:

1. Start with the default cache settings of 100MB on the site collection.

2. Capture at least 8 hours worth of performance data from the WFEs while the system is under a typical load using the "SharePoint Publishing Cache/Total number of cache compactions" counter. Since we are interested in tracking compactions per hour it is acceptable to capture this data at 1 minute intervals.

3. After analysis of the data, if we are exceeding the threshold for acceptable cache compactions we will need to add an additional 50MB to the Maximum Cache Size value and run the test again.

4. We should continue this process until such time as we have achieved an acceptable cache compaction rate.

New Database Maintenance whitepaper released

steveshe — Mon, 03 Mar 2008 22:32:00 GMT

We have just realeased a whitepaper that describes the recommended maintenance strategies for the databases that host content and configuration settings for SharePoint Products and Technologies. You can get it here. It covers:

· Checking database integrity.

· Defragmenting indexes by either reorganizing them or rebuilding them.

· Setting the fill factor for a server.

· Shrinking databases to recover unused disk space.

Overlapped Recycling And SharePoint: What Are The 64-bit Settings?

steveshe — Sun, 23 Dec 2007 20:24:00 GMT

We strongly recommend that our customers move to 64-bit servers for MOSS and WSSv3 unless there is some significant reason that prevents it. It is also worth noting that according to our recent documentation this will definitely be the last version of SharePoint to run on 32-bit hardware. This brings up the question of whether or not you need to configure overlapped recycling on 64-bit servers.

While we have quite a few customers already on 64-bit servers I have not seen a lot of problems that would be addressed by these settings but I have seen a couple. This tells me that there is still value in configuring these settings on 64-bit servers. Unfortunately, I don’t have any solid data to make recommendations on what those settings should be.

That means the best I can offer is some thought on how I would figure it out. I would start by enabling a scheduled recycle each day that happens about 30 minutes before my first users start work. This makes good sense as I consider a daily recycle to be good housekeeping. As mentioned in a previous post, recycling the worker process cleans out fragmentation and results in a more orderly workspace.

Next, I would get Performance Monitor data for my servers for a period of two weeks. These two weeks would ideally contain a few days of my highest user loads. It is also important that during these two weeks that I’m not already experiencing problems like crashes or memory allocation related errors in the ULS or Application Event Logs.

While I can’t tell you exactly what days are in your business are the most intensive I can give an example to guide you. Many companies function on a monthly cycle where their peak loads occur at the end of each month. In this type of business I would gather my data for the last week of the current month and the first week on the next month. This would probably give me a solid representation of my heaviest loads and my lightest loads.

In my scenario I’m going to guess 30% growth over the max I see in my Performance Monitor data is about the most to I want to allow before I recycle my worker process. I will be using that number in the calculations below.

Once I have my data I would then determine the maximum size that the worker process ever reached over that two week period in terms of Process/Private Bytes and Process/Virtual Bytes. I would also gather the Memory/Available MBytes value from those same times where I saw the maximum values.

Once I had these values I would first determine if I have enough free physical memory to enable overlapped recycling. To determine this I would compare the maximum Private Bytes value to the available MBytes value and if Available MBytes was not at least equal to ((Private Bytes value * 1.3)+300MB) I would not enable overlapped recycling until I had added more physical memory to the system.

Assuming I have enough free physical memory I would then use those maximum observed values plus my 30% growth factor to come up with the settings for the “Maximum Memory Used” and “Maximum Virtual Bytes” recycle settings. Here is an example:

Maximum Observed Private Bytes	2000MB
Maximum Observed Virtual Bytes	4000MB

Using these numbers I would configure the IIS Application Pool using the following settings:

Maximum Memory Used Value	2600MB
Maximum Virtual Bytes Value	5200MB

UPDATE 6/19/2008

I thought it was worth coming back to this post and updating it. As of this date we have not seen any significant problem in this area with customers running on 64-bit platforms. Most of the customers I am aware of on 64-bit platforms are not using memory based recycling settings. I do want to say that I would still reccomend a scheduled nightly recycle because it will help reduce any possiblity of problems caused by fragmentation.

Overlapped Recycling And SharePoint: Tracking Recycle Events

steveshe — Sun, 23 Dec 2007 19:39:00 GMT

As part of your efforts to properly configure your server for overlapped recycling you will obviously need to know when a recycle takes place. It also makes sense that you would want to know about ALL recycle events, not just overlapped ones. To this end you will need to configure your IIS Application Pool to track all of these events in the Event Logs. Here is how you how do this:

1. Click Start, click Run, and then type cmd at the command prompt.

2. Change to the directory where Adsutil is located. The following is the default directory location: %SYSTEMROOT%\Inetpub\AdminScripts

3. Type the following command:

cscript adsutil.vbs Set w3svc/AppPools/[YourAppPoolName]/LogEventOnRecycle 255

In the command above, replace [YourAppPoolName] with the actual name of the application pool upon which you want to enable the events.

Note:

If your application pool name has a space in it, for example, “SharePoint- 80”, you must include double quotes around the metabase path in the command. Here is an example:

cscript adsutil.vbs Set "w3svc/AppPools/SharePoint - 80/LogEventOnRecycle" 255

For those among you who are curious as to the details of what this all means, here are some links to explain:

How to modify Application Pool Recycling events in IIS 6.0
LogEventOnRecycle Metabase Property (IIS 6.0)

Overlapped Recycling And SharePoint: A Hidden Benefit

steveshe — Thu, 20 Dec 2007 16:01:00 GMT

There is a hidden value in configuring the overlapped recycling values. I was lucky enough to gain the collaboration of Thomas Marquardt on a particularly nasty performance case and he let me in on this little secret. Well, it’s not really a secret it’s just so obscure that if you don’t know about it, and where to research it, then it might as well be.

The secret is this, the ASP.NET Cache references the value entered into the “memory used” setting to govern how aggressively it tries to reduce the size of the cache. In a nutshell, if you don’t set the “memory used” value, the ASP.NET Cache will assume that when your process reaches 800MB of Private Bytes that we are running out of memory and it will begin to induce garbage collections in the managed heap to free up some space. The further past 800MB you go the more aggressive it gets. This will show up in a Performance Monitor trace as a sharp increase in the .NET Memory/% time in GC and .NET Memory/# Induced GC counters for your worker process as memory pressure increases.

Note:

In his post Thomas mentions a very important code fix that was released as a hot fix prior to .NET 2.0 SP1 and included in .NET 2.0 SP1. It was created to adjust how aggressively the ASP.NET Cache responds to low memory conditions. Anyone running SharePoint should have the hot fix or the service pack installed. The kb for the hot fix is: KB93876.

I won’t try to restate the details on how this works here because it is complicated and I would probably get it wrong. I also won’t try to simplify it because Thomas has already done a very good job of mapping it out, you may just have to read it a few times to really understand it completely. You can find Thomas’ words here.

Overlapped Recycling And SharePoint: Configuring The Shutdown Timeout

steveshe — Wed, 19 Dec 2007 21:04:00 GMT

The previous post was so short, I though today I’d give you two posts. The shutdown timeout value is used to determine how long IIS will allow for the old worker process to finish the in-flight requests that were active at the time of the recycle event before it forcibly terminates the process and drops the request. To be very clear here, if the application pool competes all of its outstanding requests before the timeout expires then it will terminate at that time. If the timeout is hit before the requests are complete, IIS will forcibly terminate the worker process and the users will eventually receive a timeout error.

The default for the shut down timeout setting is 90 seconds. This seems like a long time for most web applications. However it may not be enough for a SharePoint Server. Consider this scenario; You have configured your server to allow file uploads of up to 50MB. You have users who telecommute and even some on other continents from time to time. These both represent an opportunity for very long running operations. When you throw these into the overlapped recycle scenario you will probably end up exceeding that 90 second timeout value by a wide margin.

This is why you need to carefully configure the shut down timeout when using overlapped recycling. Setting it too low results in intentionally dropping user requests which is very harmful the user experience. Setting this value too high results in a worker process that could potentially hang around too long.

How long is too long? Depending on how tight you are on free physical memory you could end up impacting overall server performance by keeping this large worker process around while other demands for server memory cause you to start paging. Assuming you have followed the memory guidance from the previous post, Overlapped Recycling And SharePoint: What To Watch Out For, then you are probably relatively safe from this problem. That really only leaves the user impact to be considered.

In our documentation we recommend that you set this value at 300 seconds. This is probably going to meet the needs of most environments. The way to know if you haven’t allowed enough time for all the longer running requests to complete is to monitor the Application Event Log. If you see warning event messages like the one below then you may want increase the timeout further:

Event Type: Warning

Event Source: W3SVC

Event Category: None

Event ID: 1013

Date: 12/1/2007

Time: 13:07:21 PM

User: N/A

Computer: <ComputerName>

Description: A process serving application pool 'SharePoint - 80' exceeded time limits during shut down. The process id was '<xxxx>'.

In the end, it is entirely up to you how long you are willing to have that extra worker process hanging around.

Stop back by tomorro when I will give you my thoughts on: Overlapped Recycling And SharePoint: What Are The 64-bit Settings?

Overlapped Recycling And SharePoint: What To Watch Out For

steveshe — Wed, 19 Dec 2007 20:26:00 GMT

The next section of the documentation on IIS Process Recycling that needs a little SharePoint context added is the section on “Considerations When Recycling Applications”. I believe this section may put a lot of people off of using overlapped recycling. Here is what it says:

When applications are recycled, it is possible for session state to be lost. During an overlapped recycle, the occurrence of multi-instancing is also a possibility.

Loss of session state: Many IIS applications depend on the ability to store state. IIS 6.0 can cause state to be lost if it automatically shuts down a worker process that has timed out due to idle processing, or if it restarts a worker process during recycling.

Occurrence of multi-instancing: In multi-instancing, two or more instances of a process run simultaneously. Depending on how the application pool is configured, it is possible for multiple instances of a worker process to run, each possibly loading and running the same application code. The occurrence of an overlapped recycle is an example of multi-instancing, as is a Web garden in which two or more processes serve the application pool regardless of the recycling settings.

If your application cannot run in a multi-instance environment, you must configure only one worker process for an application pool (which is the default value), and disable the overlapped recycling feature if application pool recycling is being used.

WOW! That’s is some scary sounding stuff, especially if you don’t even understand what “session state” and “multi-instancing” are.

If I didn’t know that SharePoint explicitly supports and recommends overlapped recycling I’d be a bit concerned about enabling this feature. Luckily for me, we recently published a set of recommended settings for configuring overlapped recycling on IIS Application Pools hosting SharePoint, you can find it here.

There is another thing you should be aware of when configuring overlapped recycling that they failed to mention in that documentation. You need to be absolutely sure you have enough free physical memory available to support multiple instances of the worker process.

For example if you are running SharePoint on a server with only 1GB of physical memory (the minimum) then you probably don’t want to enable overlapped recycling. The reason is that when a recycle event occurs, IIS is going to create a new worker process alongside your existing one. If your existing worker process has a working set of 650MB and you only have 180MB of free physical memory, then the creation of that second worker process is going to cause a significant amount of paging and greatly impact performance. You should use Performance Monitor to get a solid understanding of the memory requirements for your worker process. You can then use that information to determine if overlapped recycling is a viable choice for you. If you do not have enough resources with your current configuration I would suggest that it’s worth the effort and expense to upgrade you servers so you can take advantage of this great feature.

In the next post I will be talking about: Overlapped Recycling And SharePoint: Configuring The Shutdown Timeout

The documentation update has been completed

steveshe — Wed, 19 Dec 2007 20:18:00 GMT

The original version of Planning and Deploying Service Pack 1 for Microsoft Office SharePoint Server 2007 in a Multi-server Environment contained an error. It stated that the recommended value for the overlapped recycling setting for Virtual Memory Used was 1300MB. This was incorrect. The document has been updated to reflect the correct value: 1700MB.

Overlapped Recycling And SharePoint: Scheduled Recycling

steveshe — Wed, 19 Dec 2007 05:21:00 GMT

As I have mentioned in previous posts, scheduled recycles are a good thing. You should view them as basic process housekeeping task, kind of like washing dishes or doing laundry. Scheduled recycles are of benefit primarily because they ward off problems caused by both heap fragmentation and virtual memory fragmentation. For those of you who don’t know what fragmentation is, I’ll attempt to describe it here. First, I’ll give you an analogy:

At my restaurant I have a huge parking lot. It has 200 parking spaces. In the morning my parking lot is completely un-fragmented because each night all the customers take their cars to another lot or to their homes and then it’s empty.

Once I open my restaurant, people start parking in various parking spaces. This is not a problem because I have lots of spaces and most people are just picking up takeout food for breakfast so they don’t stay very long. This means most of the cars are parked near the front of the lot and there are a lot of free spaces in large clumps. My parking lot is slightly fragmented

As we near lunch time the lot starts to really fill up and cars are parking further out in the lot and some are staying a lot longer than others. This creates clumps of cars all over the lot with only a few single spaces in between the clumps. My parking lot is becoming heavily fragmented. This is still not a problem as long as I have at least one parking spot for each car. In other words, I can still fulfill all of the allocation requests.

Later in the afternoon a tour bus shows up with people who want to eat lunch at my restaurant. A tour bus needs at least 8 parking spaces side by side to park. My parking lot has 48 spaces free but there no groups of more than 6 spaces side by side. My parking lot is too fragmented to meet this request.

The bus driver is denied a parking space. Not because my parking lot is too small for a bus. I even have enough total parking spaces to park a bus, they are just not in a contiguous block.

Technically, we’re all done with the analogy. But, if you really want to have some fun, add in some abandoned cars (memory leaks) and people who scrape or bump the other cars while pulling in or out and then just quietly just drive away (memory corruption).

To make the mental connection back to memory fragmentation simply replace the parking lot with your worker process’s virtual address space and replace the cars and the bus with memory allocations.

Now that you understand conceptually how memory gets fragmented, the following statements should make perfect sense to you. Fragmentation is a term generally used to describe the condition when the amount of a given resource as a whole is sufficient for a request to be satisfied, but there is not a large enough block of that resource to accommodate the entire request in one contiguous block. This is true of memory and also disk drives. Lots of people are familiar with disk fragmentation so I’ll explain the difference.

What separates the memory problem from the disk problem is that when a large enough contiguous block of disk space cannot be allocated, the file system just breaks the data into chunks and spreads it around, linking each chunk to the next so they form a chain of chunks. This type of fragmentation has a performance impact as the heads are forced to fly back and forth across the disk to hunt down the chunks, but everything still works.

The same is not true for memory. As a general rule, most programming languages that support direct memory addressing assume that memory allocations are contiguous. This means that if I can’t get a single contiguous chunk of memory of the size I requested then the only valid response is to deny my request. In the .NET world that response comes in the form of an OutOfMemoryException.

SharePoint lets you know when this happens by logging memory allocation failures in the ULS logs, the Application Event Logs and even to the client browser on occasion. If you believe you are getting these error as a result of fragmentation you should use Performance Monitor to examine the amount of Virtual Bytes being consumed by your worker process at the time of the error. If you are experiencing memory allocation failures when there is a significant amount of free memory available in the virtual address space, say more than 300MB, you MAY be suffering from fragmentation and should engage Microsoft Support for assistance. For the do-it-yourselfer, check out Yun Jin’s blog for a jump start.

Note:

There are some operations in SharePoint that require vast amounts of memory to complete. These are things like using STSADM to backup large site collections where the manifest for the backup may require several hundred megabytes or using the Download a Copy feature with very large files. These operations may fail even when there is very little fragmentation present because they require such large allocations. If you experience these types of failures you may wish to wait until off peak hours, recycle the worker process and then attempt them again immediately to take advantage of the pristine address space created by the recycle.

By recycling the worker process periodically we clear out all of those little allocations that that are breaking up the address space and start fresh. While fragmentation will eventually creep into my address space again, if I recycle the worker process at regular intervals it will probably not get bad enough to cause me problems.

Check back later when we'll talk about: Overlapped Recycling And SharePoint: What To Watch Out For

Overlapped Recycling And SharePoint: Memory Based Recycling

steveshe — Tue, 18 Dec 2007 17:15:00 GMT

I have already mentioned the values for Maximum Memory used and Maximum Virtual Memory we recommended in our recent whitepaper in previous posts but I want to speak on them directly for a minute. Let’s start by saying that there is no lack of controversy on what the proper values are for these settings. There are teams within Microsoft that believe that the Maximum Memory Used setting should never exceed 800MB and that the Maximum Virtual Memory setting should never exceed 1500MB. These are great conservative numbers and if you can service your required user load without constantly recycling the worker process while using them, then I would say that you should use them. They represent a “safe harbor” setting. Using these settings you will probably never see SharePoint logging allocation failures in your ULS or Application Event Logs.

Unfortunately, many of our customers are trying to get the last bit of life out of their 32-bit servers while moving from the less resource intensive world of SPS2003 and WSSv2 to the more feature rich and resource intensive world of MOSS and WSSv3. This means that they are often willing to take a slightly more aggressive approach to memory management to buy time until their hardware leases expire or until they have completed testing on their new 64-bit hardware. So, while there may be some theoretical or anecdotal controversy around using the 1000MB/1700MB values, we have used them successfully in some very memory intensive environments and I feel comfortable that they will work for the vast majority of customers running SharePoint.

It is possible that in some environments you may even be able to push the “Maximum memory used” setting of 1000MB a little further but I would strongly recommend that you test it thoroughly and aggressively monitor your ULS logs and Application Event logs for memory related errors to ensure that you are not causing problems for your users. While I’m on the subject, with all of the setting levels mentioned above I would recommend that you regularly peruse your Application Event Logs and ULS logs to ensure that your system is healthy and stable, this is just good administrative practice.

There is a also a hidden value in configuring the Maximum Memory Used setting. I was lucky enough to gain the collaboration of Thomas Marquardt on a particularly nasty performance case and he let me in on this little secret. Well, it’s not really a secret it’s just so obscure that if you don’t know about it, and where to research it, then it might as well be.

The secret is this, the ASP.NET Cache references the value entered into the Maximum Memory Used setting to determine how aggressively it governs the cache. In a nutshell, if you don’t set the Maximum Memory Used value, the ASP.NET Cache will assume that when your process reaches 60% of physical memory or 800MB of Private Bytes that we are running low on memory. It will then begin to induce garbage collections in the managed heap to free some memory. The further past 800MB you go the more aggressive it gets. This will show up in a Performance Monitor trace as a sharp increase in the .NET Memory/% time in GC and .NET Memory/# Induced GC counters for your worker process as memory pressure increases.

In his post Thomas mentions a very important code change that was released as a hot fix prior to .NET 2.0 SP1 and included in .NET 2.0 SP1. It was created to adjust how aggressively the ASP.NET Cache responds to low memory conditions. Anyone running SharePoint should have the hot fix or the service pack installed. The kb for the hot fix is: KB93876. In the right scenarios, configuring the Maximum Memory Used setting and this hot fix can provide dramatic improvements in performance.

Overlapped Recycling And SharePoint: Why SharePoint Requires It

steveshe — Tue, 18 Dec 2007 04:59:00 GMT

This is one of the topics I wanted to inject into the discussion that is not in the generic overlapped recycling documentation. There are at least two very good reasons why I believe SharePoint requires the safety net provided by overlapped recycling.

Reason One: SharePoint Has No Throttle

This is causing us some major headaches because under the right circumstances it makes it look like SharePoint is leaking memory or caught in a 100% CPU loop. Really, all it’s doing is what you asked it to, which is to handle all incoming users requests. Every user request to a web application, SharePoint or some other, has a cost associated with it in terms of memory and CPU. As user load and features increase, the bill gets higher and higher, eventually causing the server to run at a deficit in one of these categories and fail.

To illustrate why administrators don't know when this is happening in SharePoint, let me ask a question; What is the error message that SharePoint throws to indicate user load has reached the maximum for the server? That’s right, there isn’t one. SharePoint will happily continue accepting user requests long after it has entered the danger zone with respect to resources. Without overlapped recycling being configured the server will eventually run out of resources and start to fail in odd ways. Here are some examples:

Problem Description	Common Symptoms	Comments	Likely Remedy
Running out of virtual address space	OutOfMemory exceptions in the ULS logs or browser, operations that would normally complete fail with weird or nonsense sounding errors. On 32-bit servers This problem usually manifests when the Private Bytes counter is greater than 1000MB and Virtual Bytes counter is over 1700MB.	This is why we recommend that you set the Maximum Used Memory value for overlapped recycling to 1000MB and the Maximum Virtual Bytes value to 1700MB. More conservative numbers that you will find mentioned sometimes are 800MB and 1500MB respectively. The numbers mentioned in what I write here and in our recent whitepaper are targeted at customers trying to get the most out of their 32-bit hardware. Your mileage may vary.	Move to 64-bit. If you can’t do that then you will probably need to add more Web Front Ends or possibly even do both. There is a point of diminishing returns on all of these and at some point you will simply have too much load and you need to consider splitting your users into multiple farms. Our current documentation states that you will get your best performance with 4-5 web front ends per SQL server.
Running out of physical memory	Users complain of poor performance, administrators notice high CPU utilization and disk activity.	The poor performance complaint is caused by the fact that the server is spending all of its CPU time paging memory to disk rather than serving user requests. This is also what drives the high disk utilization.	Add more physical memory. Once you have enough you may start running into some of the other problems on this list but at least this one is easily identifiable and resolvable.
Fragmentation of the Virtual Address Space	The outward symptoms are very similar to running out of virtual address space; OutOfMemory Exceptions in the ULS logs or browser, operations that would normally complete fail with weird or nonsense sounding errors. This problem is easily identifiable by the fact that the value for Private Bytes and Virtual Bytes for your worker process will be significantly less than the maximum for your architecture at the time of the error.	If you are experiencing these symptoms you can usually validate that your problem is fragmentation with a very simple test, recycle the worker process. If all the symptoms go away and then reappear later with similarly low values in Private Bytes and Virtual Bytes then you are probably suffering from fragmentation and should open a case with Microsoft Support to discover the root cause.	Because you usually will not hit the Maximum Memory Used limit or the Virtual memory Used limit prior to failing due to fragmentation you should be able to get immediate relief by implementing scheduled, time-based, or request based recycles of your worker process. This will reorder the address space and relieve the fragmentation temporarily. Long term you will have to discover the cause of the fragmentation and fix it.

Unfortunately, people not familiar with how a SharePoint Web Front End behaves when it is overloaded tend to think it’s leaking memory. This is not unreasonable when you think about how a memory leak in IIS typically manifests itself, by a steady increase in consumption of committed memory over time. Committed memory in a process is represented in Performance Monitor by the Process/Private Bytes counter.

Many people demonstrate that SharePoint is leaking by disabling all of the overlapped recycling settings and then using Performance Monitor to track the Private Bytes counter for the worker process until it crashes. They will then show a steady upward trend in the Private Bytes until the time when the server started to fail, declare that SharePoint has a memory leak and demand that we fix it immediately.

A more complete analysis would have included a look at user load and SQL performance over that same time. In most cases, if they did that, they would have seen that the increase in Private Bytes was tightly coupled to the increase in user requests or that the SQL server has 100% CPU utilization or is experiencing severe query blocking.

In some scenarios there is no evidence of increased user load because the administrator only discovered the problem after the load was already too high. The typical behavior in this case is that the worker process recycles or crashes and as soon as the new worker process comes online, it’s memory footprint rockets up again and the cycle repeats until the server is taken offline or the users give up and go away. In these scenarios, the server will be OK until some time the next day when the load hits a critical level again and then it will start behaving this same way.

In the overloaded server scenario people also get confused when they disable the overlapped recycle settings and the process runs for another hour before it starts crashing in bizarre and often random ways. Their reaction to this is to assume that SharePoint is broken, that the crash is inevitable and that the overlapped recycling is just getting in the way of their troubleshooting. This could not be further from the truth.

In reality ASP.NET uses the overlapped recycling settings to do some performance management tuning. By disabling these settings they are actually making the problem worse. I’ll cover exactly what that ASP.NET tuning does in a later post.

Reason Two: All applications have flaws, SharePoint is no exception

Virtually all applications that are under a constant update cycle at some point will suffer from problems like memory leaks, fragmentation, memory corruption, etc. As developers you generally guard against these things by implementing very strict change control processes, peer code review before check in and stringent patch testing. We do all of those things and more in SharePoint. So why does SharePoint suffer from some of these things from time to time? Because we don't own all of the code that runs in our worker process.

To begin with we leverage code from a variety of sources just to produce SharePoint. Here are a few of this bits of code we get from other sources:

· Shared Office components.

· MDAC – Microsoft Data Access Components

· .NET Framework

· ASP.NET

· SQL Server

· Windows

· Active Directory

I’m sure there are many more but I think that makes my point. These components are written by hundreds of developers, each with their own design specs and expectations about how that code will be implemented. If they make a single mistake in that code or we make a mistake in how we implemented it, that will eventually show up as a problem in our worker process. There is also one huge hole that allows us to suffer from these problems even when we don’t introduce them ourselves, extensibility.

SharePoint is designed as an extensible platform. The most common extensibility point that causes us to fall victim to these problems is custom web parts. You only need to read the content we have developed to help developers do a better job when writing code against the SharePoint object model to know that it’s really easy to get it wrong…and they still get it wrong… a lot! Here is some light reading on the subject for your reference:

Best Practices: Using Disposable Windows SharePoint Services Objects

Best Practices: Common Coding Issues When Using the SharePoint Object Model

One of the common custom web part patterns that is prone to high memory consumption problems and memory leaks when using the SharePoint object model is navigation controls. Due to the need for these controls to iterate through the entire structure of the sites in order to generate the navigation control, one tiny coding mistake can be amplified hundreds or thousands of times. This often results in huge numbers of leaked objects. Each SPSite or SPWeb object consumes ~2MB of memory. You can see how this could very quickly take down your server.

Let’s not forget anti-virus vendors and IFilter vendors. These too can contribute to bad behavior in our product and there is almost nothing we can do to prevent it.

The point of all of this is that no matter how good of a job the SharePoint development team does to avoid introducing these types of problems, they will probably creep into our code at some point. Even if we don't add them, someone else will.

A good analogy for my request that you run with overlapped recycling all the time is homeowners insurance. Do you go buy homeowners insurance when it looks like a storm is coming? No. You carry homeowners insurance all the time because you don’t want to be left holding the bag if a storm should unexpectedly come along and destroy your house. You should view overlapped recycling as a form of insurance against other peoples coding mistakes and run it all the time, not just when it looks like there is bad code on the horizon.

Since we’re on the subject of bad web parts, if you suspect that you have a web part that is not following the best practices referenced above, a simple way to test that theory is to uninstall the web part and see if your memory consumption goes down or the memory allocation errors go away. If this is not possible, you should open a service request with Microsoft support to assist you in troubleshooting this problem further. As I’ve said, these are not an uncommon problem and for the most part if you tell the engineer you suspect this is the problem they can usually confirm that with a memory dump of your process. This can also be accomplished with a code review of the web part, assuming you have the source code.

In closing, if you are attempting to troubleshoot a problem like those described in this post and you intend to engage Microsoft Support you will be doing yourself a huge favor if you gather the data we will need prior to making that call. Here is a brief description of what you need to do:

1. Gather Performance Monitor data for 1 full day. Starting before users begin to use the system. You should recycle the worker process right after you start the performance monitor trace so we can see things from a clean start. You should include all objects and all counters using a 15 second interval. You should capture data from all of the Web Front Ends and the SQL server simultaneously. You do not need to restart the SQL server.

2. Capture a user dump of the worker process when it is above 900MB of Private Bytes memory consumption.

If you do not know how to accomplish these things the support engineer will be able to assist you when you call Microsoft Support.

Overlapped Recycling And SharePoint: The Generic Value Of Overlapped Recycling

steveshe — Sun, 16 Dec 2007 16:55:00 GMT

Now that we have a good understanding of the two types of IIS Process Recycling I want to discuss the portion of the documentation that attempts to explain the value of process recycling. From my discussions with customers it appears that most people consider process recycling in general and overlapped recycling in particular only as a form of first aid. They also take this analogy one step further by assuming that it should only be enabled when the server is obviously sick and that a recycle event is evidence of some catastrophic failure that must be rooted out and repaired immediately. Let’s review the passage in the documentation that I believe sets this expectation:

Worker process isolation mode offers process recycling, in which IIS automatically refreshes Web applications by restarting their worker processes. Process recycling keeps problematic applications running smoothly, and is an especially effective solution in cases where it is not possible to modify the application code. [Emphasis mine]

This paragraph may lead you to believe that only problematic applications need process recycling. Unfortunately they don’t really define what problematic is. I suspect they meant applications suffering from prolific native memory leaks, managed code with poorly written object lifetime management, applications suffering from significant memory fragmentation, etc.

Am I suggesting that SharePoint is so riddled with flaws that we need to keep the virtual ambulance at the ready at all times? Not at all. As support people we should take every advantage to prevent the user from being exposed to the problems in the applications we support. This is the idea behind many common uptime oriented features, SQL clustering for example. I am saying that overlapped recycling represents one such opportunity and you should take advantage of it.

Overlapped recycling was created because applications sometimes have undesirable behaviors like memory leaks, fragmentation, etc. If you are running a client application like Word or Excel and it has these behaviors you can simply save your work and restart the application. With web based applications this is simply not a viable solution. There simply is no reasonable way to have everyone "save their work and restart."

Overlapped recycling provides an excellent compromise between the need to restart an application during production hours and the desire to shield users from the impact of doing it. In my opinion, if you are not taking advantage of this feature then you are just punishing your users and adminstrators needlessly.

Overlapped recycling in general is a good thing but too much of a good thing always ends up being a bad thing. If your worker process is recycling at a very rapid pace, say, several times a day or more, then you should be concerned and work to identify the root cause. If your worker process recycles once or twice a week due to the maximum memory used limit or the maximum virtual memory limit I would not be too concerned but I might start planning to add additional Web Front Ends to reduce memory pressure or start on the work to migrate to 64-bit servers.

In my next post I will be discussing: Overlapped Recycling And SharePoint: Why SharePoint Requires It

New must-read SharePoint documentation released!

steveshe — Thu, 13 Dec 2007 19:30:00 GMT

I am working on a series of posts regarding performance and troubleshooting on SharePoint which I had hoped to get posted before the end of the year, only time will tell if my workload cooperates on that. But, in the meantime we have released a ton of new documentation and cross referenced a lot of it to older docs that provided a wealth of information. The net result is that we have some new stuff and provided a lot of clarification or updates to old stuff. I have provided a quick overview of the things I liked in each of these docs below:

Planning and Deploying Service Pack 1 for Microsoft Office SharePoint Server 2007 in a Multi-server Environment

· Publicly state our position on IIS Application Pool recycling and provide a recommendation on how to configure it. (As of this moment there is an error on the virtual memory setting, it says 1300MB, it should say 1700MB…)

· Stronger recommendations to use 64-bit Web Front Ends

· Reiterated that you cannot use the /3Gb switch on Web Front Ends and explain why.

· Publicly commented on mixing 32-bit and 64-bit Web Front Ends.

· Clearly state that 4-5 Web Front Ends provides maximum throughput.

· Publicly stated our position on Web Gardens.

· Provided caveats around using explorer view.

· Publicly deprecated the use of “Download a Copy” with large files.

· Best practices on managing large lists and provided links to more detailed guidance including how to limit the creation of personal views.

Performance Recommendations for Storage Planning and Monitoring

· We publicly state that v3 is more resource intensive than v2.

· Provide clarification on content database size limit of 100GB.

· Publicly state that v3 is the last version of SharePoint to run on 32-bit hardware.

· Publicly state that gradual upgrade may require twice as much SQL resources.

· Publicly recommend SQL 2005 SP2 64-bit.

· Recommend adding more SQL servers when you have more than 4 Web Front Ends.

· Recommended 16GB of memory for SQL server in large deployments.

Some other stuff I would highly recommend would be:

Plan for software boundaries (Office SharePoint Server)

Manage lists and libraries with many items