Performance, Performance, Performance
We just got out of a lengthy performance review on our way to Beta 2. Things are in good shape and we hope to have preliminary guidelines around the same time as the beta. Performance work is one of the most fun parts of building software, and while a huge topic, I thought I would post about a few interesting areas impacting our next release. Our goal is most SharePoint customers don’t have to worry much about performance. Just get the farm topology we recommend for your usage scenario and you will have plenty of headroom. But for large organizations, enterprise portals and internet sites with demanding performance requirements, it is pretty critical you understand this and plan your deployment carefully. Regardless of which group you fall into, I hope you find this interesting.
Current Product Guidelines - Here are links to our existing capacity planning guidelines which should provide good background for the discussion below and as well as a starting point for the next release:
SPS 2003 Guidelines
http://office.microsoft.com/en-us/assistance/HA011647571033.aspx
The numbers in appendix have been recently updated for the latest hardware and SQL Server 2005 showing an average RPS (requests per second) of 97, 198, 1114 for small, medium and large arms.
CMS 2002 Guidelines
http://www.microsoft.com/downloads/details.aspx?FamilyID=e1f50824-c281-4fd6-966c-ac4c68106010&DisplayLang=en
While less current numbers, there’s a lot of interesting details applicable to large scale sites notably caching features of CMS 2002 that will be integrated and enhanced in Office SharePoint Server 2007.
Usage Scenarios – No two customers are the same but we’ve come up with a few standard scenarios for assessing performance and scalability. Here are the key ones:
• Enterprise Collaboration Infrastructure - Since one of the goals is to help organizations save money via delegated server consolidation of intranet, collaboration and file servers, the key question is how many sites can we support in a single server farm. The notable thing for the development team is unlike the next two scenarios caching really doesn’t help as you have lots of sites each with modest usage.
• Enterprise Portals – Here the biggest issue is typically how much throughput does a single site support at peak (e.g. 9 AM with portal as the browser default home page). Caching can help some here although often an enterprise portal will show different content for different users based on security and targeting (e.g. Marketing and Finance see different content). Probably the biggest factor in portals is the impact custom code such as web parts could have if not tuned.
• Internet Presence – Internet presences have similar issues to enterprise portals at much higher scale points. The advantage they have is often large parts of their content are static and unsecured so caching can be extremely valuable. CMS uses ASP.NET output caching to support thousands of request per second per farm and beyond.
Usage Models – We developed models of usage patterns in these scenarios based on what we see form a variety of customers and publish the formulas so you can tweak for your organization. For example, Enterprise Collaboration Infrastructure tends to be the most demanding traffic per user of the three scenarios above which results in roughly 1 request per second per 1000 users. Since most medium-large companies will deploy at least 2 front-ends for avoiding a single point of failure, that led us in SPS and WSS to have a minimum goal of supporting 100 (dynamic, authenticated and unique) requests per second for a 2 front-end farm because then we could support the portal and collaboration infrastructure (without custom code) of an organization with 100,000 users making our base performance a non-issue for most enterprises. While the increase in hardware performance makes this less of an issue (e.g. the 198 RPS for a medium farm), it was definitely a huge mantra for us during V2. This focus also helps small-medium businesses because it tunes how much we can do on a single machine or more commonly single front-end since we’d almost always recommend running SQL Server on a separate server except for demo and development scenarios.
Scale-Out vs. Scale-Up in WSS V3 and Office SharePoint Server 2007 – So there’s a key difference between supporting lots of little sites (e.g. collaboration infrastructure and internet communities) and a few huge sites (e.g. enterprise portals and internet presences). We provide the base facilities for the former in Windows SharePoint Services. In fact, WSS V2 was designed not just with enterprises in mind (e.g. Microsoft’s internal use with 300K sites) but with the design for hosted services including ISPs and Microsoft’s recently announced Office Live service. The big design features are support for stateless, load-balanced web front-ends with multiple databases, site collections and sites per farm. There’s relatively little change in this model for WSS V3 other than the light-up support which Office SharePoint Server supports via an enhanced Shared Service Provider model which re-uses databases and ASP.NET applicaiton pools to enable all the sites to have all the features. So base WSS and SPS are fast enough to handle most sites with 100% dynamic content but Office SharePoint Server adds advanced caching on top to support the most demanding sites.
Base Execution Throughput and Latency – Our goal is not to require changes in the farm topologies from V2 and continue to deliver sub one second latencies on page requests. There are definitely some additions to WSS V3 that can impact performance – the movement of the web part framework and execution of pages from the database from the WSS layer to the ASP.NET V2 layer, supporting native ASP.NET forms for extensibility, support for security trimming the UI (e.g. readers won’t see things like “Site Settings”), expanded browser and accessibility requirements as well as more work in the database to support features like content types, workflow and item level security. We are focused on these and there is a lot of tuning going into to Beta 2 so we will be in good shape so that few customers will require more hardware to support the new features. There are also new capabilities such as the InfoPath and Excel browser-based rendering that we are making sure get tuned and we have guidelines for adding these services to the farm as well.
Caching – I think most people would be amazed at how much time we spend talking about caching. Of course, it is much faster to fetch formatted HTML or data from memory or disk than to make multiple database or SOAP requests and do the formatting via a combination of ASP.NET and XML\XSLT. That said, caching poses many challenges. In highly distributed usage patterns like enterprise collaboration or internet communities, it is often not needed or does not help. Even when it could benefit a large site, it complicates security (the cache better not undermine the underlying security of the system!) and timeliness issues (how do update all the caches on the front-ends when data changes). Finally, caches and cache maintenance can take up memory and processor time that, if not careful, could actually make performance worse. So this is why we didn’t add caching to WSS (other than letting you specify cache web parts) and did focused caching in SPS 2003 (e.g. caching the navigation hierarchy with security trimming). However, because of the internet scale requirements (the most demanding latency requirements and many thousands of pages per second vs. hundreds), CMS 2002 had very elaborate caching support (writers spent a whole chapter on it in CMS books!) that we have evolved even further in the Office SharePoint Server 2007 code base. Leveraging the experience of some of the most senior developers from CMS, Office SharePoint Server will support caching pages, page elements, XSLT, XML, data from SharePoint lists, etc. Where it makes sense we will do the right thing without much work but you will be able to set caching policies appropriate for different types of sites and they strictly enforce security (which is feasible with a few buckets of content but obviously will not scale to caching content fragements for every user). The biggest thing carried forward from CMS 2002 to Office SharePoint Server 2007 is support for ASP.NET output caching where all or most of the executed HTML page is cached and IIS can send it via a kernel mode request to the client. We are doing work so you can still use this even if parts of the page aren’t cacheable (e.g. “Hello Jeff” in the top navigation), you still get the benefit of caching the other 95%. This will not only make sure largest internet presences are as fast as possible, but also enterprise or divisional portals upgrading from SPS 2003 can get a performance boost as well particularly for frequently used pages.
Custom Code – One of people’s favorite things about WSS V2 and SPS 2003 is the ability to write custom web parts and event handlers in .NET. We’re expanding this significantly in WSS V3 and Office SharePoint Server 2007 with native ASP.NET 2.0 web parts, support for optional in-line code and compiled ASP.NET pages, more workflow and event handlers, etc. As with the past release, we will publish guidelines on writing efficient code which we will strongly encourage everyone to read on creating and destroying SharePont objects, monitoring SOAP and database connections and round-trips, optimizing how much XML you use, etc. I also know some of the readers of this blog are some of the best experts on this with the current product and will point you to other blogs. We love the SharePoint developer community out there. As I mentioned in the Page Anatomy post, we are also going to give options to reduce the amount of code you need to write. For example, instead of writing 10 Web Parts each making hard-coded SQL or SOAP request to a database or web service, think about using the new Business Data Catalog to register those entity and methods in an abstracted way and using the new Business Data Web Parts which are built on top of a generic and flexible Data Form Web Part which can be customized using the Office SharePoint Designer 2007 XSLT editing capabilities. It won’t handle every scenario, but hopefully in many cases you can use it and leverage our coding and performance tuning so you don’t have to do this kind of work.
Scalability – There are many dimensions to scalability (sites, documents, users, etc.) and we are making big investments in increasing these for the next release and we’ll cover them all in the capacity planning guidelines. I’ll go into just example in this post – search. The 2nd generation search engine we use in SPS 2003 supports 4 catalogs of 5 million documents each for a total of 20 million items. We started on a more scalable 3rd generation engine we will be using in both WSS and Office SharePoint Server a few years ago with a version first shipped in SQL Server 2005 last year. One of the side benefits of improved scalability is we are simplifying search administration by eliminating the concept of multiple catalogs. We will support a single more scalable catalog (per Shared Service Provider) with many “search scopes” that give the other benefits of multiple catalogs with more flexibility and less complexity. We are also improving indexing efficiency which will help both the Search server and the WSS system. This is via 2 key investments: 1) using both the new WSS change log support (which is also used by Outook 2007 for offline documents) and 2) doing ACL only indexing (when a new member gets added to a parent WSS site it does not force a re-indexing of all the content in the V3 search engine, just an update of the index ACLs). While these don't speed up the first crawl, they will help a lot on subsequent crawls and help provide more cycles for more sophisticated relevance computations. We are also supporting what we call “continuous propagation” so pieces of the index are much more frequently sent to the search server so search results in large scale farms are very timely vs. having to wait for a crawl to complete before the full index is published. Finally, because of the "light-up" features, Office SharePoint Server can provide the native search features for all your sites so there is no need to double index the content which both takes up space and adds load to the WSS system and index time. There’s similar kinds of investments made in document and user profile scalability we’ll cover in the future.
64-Bit – I thought I would end with a little discussion on 64-bit as we often find the perception of the benefits are a little beyond reality (e.g. won’t it make everything twice as fast?). Often times you will see solid gains for 64-bit machine but there may be little difference in 64-bit mode on that machine vs. 32-bit mode. More often, the gain in the test is relative to an older 32-bit machine. The new AMD and Intel 64-bit chips will help a lot (and we also like dual core and are looking forward to quad core), but the benefit over 32-bit mode is because of the increased address space support for those parts of the applications that can benefit from more RAM (e.g. the SQL, Index and Search services in a SharePoint farm). It might also help with the caching I mentioned earlier although disk-based output caching in ASP.NET is already extremely fast. For other parts of the system, the performance can actually be 15-20% worse because in 64-bit mode you are passing around double the data. The net benefit is different for different applications and usage patterns so we’ll let you know how this all nets out for the various SharePoint scenarios. I think you will see some healthy scalability improvements when using the extra memory and address space for the SQL and Search process in sites under heavy load, but just don’t magically expect 64-bit mode on the same hardware to be twice as fast.
I know there is a lot of interest in more specifics and there are many areas I didn’t have space to touch on. The team is working hard to publish at first draft with beta 2 and we will forward to your feedback on that as we finalize it for RTM.
-- Jeff Teper
