Massive Scale Deployment - Modularity in SharePoint Farms
I was talking to a team recently about how to scale to the hundreds and thousands of servers to support the extreme large scale deployments. The question was... Let's say you were going to design Myspace or a Wikipedia on WSS or SharePoint Server, how would you do it and what would you do? Without getting into the dev design I broke it down like this....
For the Web Tier:
For WSS, I would have both the WFE and Search/Index components on all Web Front ends, I would require all solutions be packaged and keep solutions/packages to a minimum. Configurations themselves I would keep to a minimum. I would likely go with 4 to 1 ration of web front ends to SQL backends. The assumption for me is this would mostly be authenticated traffic, if it's anonymous I would increase the ratio and look seriously at reverse hosting and cache devices (Akami?) to reduce the requests to the Web & SQL tiers. From a hardware perspective, I'd look at blades and investigate virtualization techniques, but first off I'd likely go with the cheap hardware and build in redundancy and look at standardization.
For MOSS: I would run all components except Indexing and SQL on the WFEs. I would also figure out what caching techniques reduce the most load and transactions to SQL. I would investigate reverse hosting and reverse proxy type scenarios with ISA 2006 and other third party options to both reduce the attack surface and to offload anything that could be handled out of cache. I would seriously look at the core.js file and how I could make sure it is not loaded in any anonymous scenarios and ensure that any request was optimized when it was requested. With anonymous and caching scenarios I would look at 8 or 10 Web Tier to in the 1 SQL tier.
For the App Tier:
I would seriously look at eliminating any complexity at this tier. I would not offload Excel or Query for 99% of circumstances. For Indexing in 99% of scenarios I would look to offload and use this with the WFE components on it to reduce load to the web tier. WSS wouldn't have anything in this tier.
For MOSS, If I knew I was going to have more than 8 servers, I would look at building a dedicated farm for Indexing and run an SSP farm. So in this case, For every 20-25 servers I would have a dedicated SSP farm. This would allow me to essentially remove any dependency on an SSP within the farm.
For SQL the Data Tier:
I would optimize the way this is modularly used and scaled. A lot of this is simply cost. In my past life I loved the flexibility of SANs and Clusters and how easy it was to grow disks and change and move the databases. Now, in this generation I look at elminating clusters with Mirroring and DPM 2007 as the way to scale with DAS or (Cheap) blob or block storage. The Config, SSP, and Search databases would be on optimized disks and the data (content dbs) would be on the cheapest disks I could support.
The true key to scale would be modularizing this. So with the thoughts above, it would look like this. W = Web, S= SQL
Content Farm 1
W1W1W1W1W1W1W1W1W1W1, W2W2W2W2W2W2W2W2W2W2W2, W3W3W3W3W3W3W3W3
3 banks of 8 WFE each (8 Web Front Ends) + (8 Web Front Ends) + (8 Web Front Ends)
3 pairs of mirrored SQL Servers S1->S1 S2->S2 S3->S3
(Config db, Search, and SSP on optimized disks on S1, content on S1 and S2 on cheap RAID 5 storage (would look at using the new storage provider for blob or block storage))
(For Modularity in SQL, I would allocate no more than 2TB per SQL server, (For content dbs I'd go somewhere from 10-29 content dbs of ~100GB each) depending on cost and availability requirements I'd mirror the databases to a similar system and use DPM 2007 (once it's available) as a backup solution. I'd use Diffs in the mean time.) Largest volumes would be 300 GB formatted. From an anonymous perspective each server could host 500K daily sessions with upwards of 100,000 daily users. Supporting a million anonymous users in this mythical example could be supported by 10 web front ends so in this farm we could support 3 million anonymous users. From an authoring perspective we'd need to be much more conservative. We'd look at 10,000 to 25,000 per web front end, let's say 10,000 daily authenticated users to be conservative. The 32 servers could then support 320,000 daily authenticated authors. If we need to support 1 million or more authors, we'd look at trippling the number of servers, but we'd follow our modular approach and even before getting to 32 we'd likely want to start to look at inbound and outbound bandwidth considerations. We'd likely want to create VLANs for the back end and ensure appropriate bandwidth to the SQL servers. In this mythical example if it was Active Directory we'd likely want to look at multiple domains at a certain scale. We'd also look at dedicated DCs to support this, I don't have any # of users per DC numbers, but somewhere around 3-5 Front ends or maybe more users could be supported based on the session lengths and TTL. in the millions and with SQL auth we could look at millions of rows, but indexes in the structure would start to be very important for quick retrieval in a forms based environment.
SSP Farm (For indexing)
W1W1
I1
S1->S2
The web tier in this SSP environment would be litely accessed and would be used primarily for it's query services. No end users would access with web servers directly. The index would do continuous propegation and would as well have web components (no local query) and do indexing locally. You may find this SSP farm model scales out even further. What if you had 2 SSP farms that indexed the same content providing a *highly available indexing solution* that could change? You can associate and reassociate SSPs!
What about adding servers?
Unattended installs are great for laying down the binaries and connecting to existing config dbs. Imaging I haven't heard much success, but there may be some options. The gotchas are.... The server names are in the config db, so you shouldn't copy config dbs. You could automate creating the config db, SSP, and Content dbs with the appropriate STSADM commands. There may be more options that you don't think of in terms of aliasing and name abstraction through host files. Don't get in trouble here though. The config db makes a ton of assumptions if servers change without changing what's installed underneath it. i.e. you shouldn't replace a WFE with a new WFE using the same name, because it might think it already has necessary solutions installed and web apps pushed down, etc...
Speaking of databases, this is where you would want to limit the number of sites (Site Collections) to limit the storage consumed. Having limits and quotas would be necessary to get the scale happening in a modular way. I recommend reading my previous postings on db sizes, site collection sizing, etc...
Good reading on Farm SharePoint Scale and Architecture:
TechNet: Plan for Availability - Good info on modularly scaling out and what server roles can be offloaded and other considerations.
TechNet: Logical Architecture - This under rated doc has some of the best content around SSPs, why more than 1 farm, and App pool considerations
TechNet: Estimate Capacity Requirements. 8 web servers tested against 1 SQL server, hardware and numbers included. Includes test results from perf team on adding more and more web front end servers. Remember this was in a test environment and not reality. The ratios will vary based on requests, type of content, caching/offloading and anonymous vs. authenticated.
