Joel Oleson's Blog - SharePoint Land : Backup

Backup Throughput Benchmarks

joelo — Tue, 11 Mar 2008 22:13:00 GMT

I twisted Mike Watson's arm to let me blog some of his recent time tests on backup. I thought the data was an interesting reference. Emphasis is purposefully placed on reference. Your numbers will sure to vary.

Data Protection Manager 2007: 100MB/sec (over GigE) GigE, Network, and Network Interface is usually the constraint. Also highly depends on the disk speeds. Our 12 disks SATA backup arrays only give us about 75MB/sec.
Site Collection Backup: 3.25MB/sec (to disk) STSADM –o backup -url You can do multiple threads, but it really impacts your performance when you got a like 5 threads going. Going from one to two will nearly double, but going from 1 to 5 is not 5X, and blocking will result.
Native SQL Backup: 150MB/sec (to disk) Using compression gets you about 30% compression with 25% faster backups. Adding multiple threads this can increase, but you risk total system performance at more than about 3 or 4.
Farm Backup: 35MB/sec (over GigE) With Central Admin UI. If you use disks local to the SQL server (even if over UNC) you will see much much, better performance than remote shares. This is the number most likely to vary from one environment to another since the Index backup is the slowest part, so it really depends on how large your index is and what topology and network you're using for your index server and between your SQL environment.

This information flows well with the post on demistifying and navigating backup strategies.

If your experience is different than this, or you want to add what your third party vendor speeds are, feel free to add to comments for others to enjoy.

Navigating SharePoint Server Backup

joelo — Sun, 24 Feb 2008 19:19:00 GMT

I think one of the most common IT questions is around how to backup SharePoint Server and when to use SQL backup over STSADM backup. I've chosen 200GB below, but essentially anywhere between 200-500GB or as soon as you're ready to manage it outside of a scheduled task. Let me quickly demistify this for you. There's essentially 5 sets of configuration and data you should be concerned about.

Content Databases - this is where you data is stored
Configuration Databases and Admin and SSP Admin Databases - farm and web app configuration including server names, solutions and features you've deployed
SSP, Search Databases and the Index(es) catalogs - Essentially everything from Profiles, BDC, Search, best bets, properties, content sources
Server Configuration - IIS configuration including metabase information and other server configuration such as system state including registry
On disk files - anything in the 12 hive or program files directories you've added or customized, all deployment packages and features (have special copies of these for redeployment), back these up. I'd STRONGLY recommend that there never be anything on disk that isn't in a feature or solution deployment pack, but I realize there are some exceptions that must be documented in your recovery plan especially around search configuration or some changes to specific web.config files. This also includes deployment of any add-ons. Make sure you have copies of these web part packages, interop solutions, search add-ons or whatever they are. I'd say all disks on all servers for simplicity, but this oversimplification doesn't help for more complex environments. If you can do a windiff (file comparison) against a clean box, you'll essentially reveal what the is necessary here. I recommend creating a special directory with a copy of all of the things you deploy to your farm from ifilters, web parts, solutions, web config change history, AAM configuration, the setup files, QFEs, service packs, really keeping a change log that's easy to re apply is essential and this needs to be easy to recover. You may want to keep this in a handy directory or share as a "version" of what you're running from top to bottom. I'd keep this on a staging server or some server off your production environment so you don't have to go to tape if you simply lost a web front end for example.

To backup 1-3 for small environments under 200 GB, you could essentially use the out of the box stsadm -o backup -directory commands to backup the relevant databases. This will need to be scheduled, if you're electing to use Microsoft tools, you'll likely start with a daily scheduled task that should be monitored. I do emphasize monitoring it, there are a number of ways this task can fail with improper perms, failed access to a share, missing flags, already running backup job that hung, etc... You may elect to purchase DPM 2007 or a third party to make the backup easy to do while making retrieval easier and potentially more reliable than a simple scheduled task. Using the -o backup -directory would give you the granularity of content databases. You'd then use NTBackup or whatever normal server backup you use to backup all of the backup files to tape, disk or into your offbox/offsite recovery solution. At this size for 4-5 you can backup all your directories and system state. It's the easiest route to go and you'll be glad you did when you try to locate that ifilter you purchased. (Essentially get a clean copy of 1-5 out of the data center.)

For over 200 GB I recommend no longer using STSAdm to backup the content databases. What's essentially happening in STSAdm -o backup -directory is a SQL backup anyway ;). So as you get more data, you need to have more granularity around it so you can more quickly monitor the jobs for backup up the individual content databases in a quick and manageable way. Use SQL Server to backup 1-2, and if you need granular recovery use DPM 2007 or AvePoint or your favorite backup vendor that supports SharePoint Server 2007 (some backup vendors add special sauce to do brick backups, some do not, so be sure to ask all your questions around ability to get granular recovery and speed) to backup the content for quicker recovery. Some vendors focus on speed, like Quest's SQL litespeed with database backup compression like you'll get in SQL 2008. You should still use stsadm -o backup to backup 3, you essentially want to keep the time stamp as close to the same time for all of the search/index related components (configuration and data). Of all of what you backup, the search, ssp, and index will be the most tricky to get right. I highly recommend you test out your recovery plan on a recurring basis. The ops team must know you to get this right unless your plan is to recrawl.

The larger your databases get, the more creative you'll get. Backup takes a lot of disk IO and network throughput, don't dismiss it as a bottleneck while it's running. It can easily be overlooked as the most heavy usage your boxes may see. This is one reason you may consider a snapshot solution like DPM 2007 or using VLANs to move the Network IO off the NICs that is normally used by your user traffic. Once this isn't competiting for resources with your users you can start to multi thread the processes and backup multiple databases at once. That's one reason I recommend not putting all your data in one large database. I can see I'm moving from simple, but you get the idea.

TechNet has some good resouces on backup "Administering backup and recovery for SharePoint Server 2007". This page was updated January 24, 2008 with more recent information on backup tools and resources including a demo.

If you find yourself frequently recovering databases to restore site collections or sites, I highly recommend the site delete capture tool. It's a simple add on currently posted on codeplex that is used by MS IT and was created using the object model and is deployed as a feature. The source is available if you want to change what it's doing.

Backup / Restore and Data Protection Manager 2007 RTM

joelo — Thu, 25 Oct 2007 16:12:00 GMT

DPM 2007 has RTMed! This means there are now more ways to backup and protect your farm.

I was on an internal field call (Academy Live) and Live Meeting yesterday with the DPM Product Managers doing a presentation on using DPM 2007 to protect your SharePoint Farm. There's some fantastic work done by the DPM team to really simplify backups for WSS and MOSS.

I've seen the draft of their SharePoint focused whitepaper which will be coming out soon in addition to some info around the SSP and recovering your indexes.

For more on DPM check out their DPM blog and the DPM Microsoft.com site. Be sure to look at for the DPM 2007 version which is the one that supports backup for Exchange, SQL, and SharePoint.

The gist of DPM 2007 is faster backups using the VSS snapshot provider that SharePoint provides to backup the farm. The copy on write technology can backup your farm as frequently as you want and then allow you to do either farm restores, database restores, or granular restores on a recovery farm.

MS IT has been using DPM 2007 pre RTM to backup their Asia farm with great success, their plans are to slowly replace their current backup solutions with DPM 2007 and I expect Mike Watson and/or Bill Baer to be blogging about it when they do.

The other big thing around backup restore is the recent whitepaper just posted to TechNet. It is THE most comprehensive backup/restore paper out there. The backup chapter in the Administrator's companion is pretty good too, but this one really spends a lot of time on restore which has been the gotcha for some customers. I'm sure you've heard me say it, but let me say it again. Test, test, test your restores. You have to have some process that tells you that backups were successful, not just to disk, but to whatever media you're going to after that. Next you must regularly test your recovery whether you are doing datacenter failover with logshipping, mirroring, or some third party solution. Don't just expect it will work, cause I'd suggest that it nearly always takes longer than you expected, and many times the tapes are bad, the disks that you were backing up to were bad or failed along with the disks your data was on and the ops guys are scrambling for documentation.

So here it is. Data protection and recovery for Office SharePoint Server. Don't let the title undersell this paper. It is the most comprehensive restore paper/info out there. If you want to challenge this, please do so in the comments and let us know what's better. I shared some quick thoughts on the 40 page paper in a post on the SharePoint Team blog.

September SharePoint Beagle

joelo — Fri, 28 Sep 2007 21:01:00 GMT

Bob Mixon (SharePoint MVP) has put together a very interesting free newsletter called the SharePoint Beagle. You can read it online or sign up for it on Bob's site. This months articles feature a backup/restore article by Bob Fox (SharePoint MVP). I got a request recently asking me to provide a sample script for backup. Here you go. He actually gives you step by step with screenshots to show you how to do this in task scheduler. He doesn't just tell you what options there are... he creates it for you.

Creating a simple backup script

Scheduling backup script with Task Scheduler

Validating backups

Restoration

The other article on creating form web parts are pretty good. I was hoping for a little more on the end user training article :). What exactly was I looking for? I was hoping for tips and tricks. What does the end user need, what does the designer need, who should be trained on Infopath, what should the site administrator be trained on?

My recent best practice has been to create a site collection administrator training course within your corporation that trains them on how to use support, train them on corporate information policies, retention, handbook stuff, and then after they accept all that, put them through permissions management, features activation, managing a list, corporate policies around master pages, etc...

Mirroring, Log Shipping and High Availability resources

joelo — Mon, 30 Jul 2007 00:23:00 GMT

Last week at TechReady I attended Mike Watson's and James Petrosky's talk on HA and DR scenarios. Good talk.

It reminded me of a few resources:

The best resource is "Using database mirroring with Windows SharePoint Services. " This downloadable white paper has been updated July 9th with more complete technical information. The mirroring article "Using database mirroring (Office SharePoint Server)" was recently updated by Mark Shea. They were some syntax issues, and more was added on aliasing and the stsadm rename server command was clarified.

Mike Watson (Engineer in MS IT/MMS) does a comparative analysis on his post "Database mirroring and log shipping. Which is better?" in his next post he follows up explaining that it's true that WSS and MOSS 2007 don't support automatic failover, but can be a scripted failover. Would be great if he could post his deck and link to it.

Bill Baer, does a post which appears to be an article on Mirroring which if not mistaken is something that Mike worked on with him. SQL Server 2005 Database Mirroring and Windows SharePoint Services 3.0/Microsoft Office SharePoint Server 2007 - Part 1 (Introduction, Overview, and basics). I do recommend viewing the comments on these as well. Some interesting comments. Part 2 is on configuration.

Scot Hilier posts some thoughts on Database Mirroring in a recent post, his thoughts are similar to how I'd do it high level.

I did a post a while back on all things related to replication topics and SharePoint. As well as a post on SharePoint HA and DR. For that I recommend combing through the performance and scale topic as well as high availability.

The Sams Publishing book SharePoint Server Unleashed has a great chapter that talks high level to the topic of Planning Redundancy and Scaling the SharePoint Environment. A sample of this chapter is now online.

There is an interesting KB on log shipping that talks to connecting to the log shipped content database. "Using Microsoft Windows SharePoint Services with a content database that is configured as read-only in Microsoft SQL Server"

We still unfortunately don't have good documentation on log shipping and SharePoint. We still refer people to the 2003 Resource Kit. Look at this section "Hot Server Farm Swapping via SQL Log Shipping" in the backup chapter. There are going to be differences in the database names and we do still not recommend log shipping the config database....

For more information, see the white paper “INF: Microsoft SQL Server 2000: How to Set Up Log Shipping”, the Microsoft Knowledge Base article 314515, and the article “Log Shipping in SQL Server 2000” there are the applicable 2005 articles as well.

Let me give you some hints on the config db... It contains servernames, web apps, web parts, features, solutions, services on server, etc... are installed on each of the servers in the farm. So changing servers without the knowledge of the config db makes it fail.

One of the better recommendations I've heard lately is about abstraction and aliasing. If the SQL server is an alias then changing the hostname to the other hostname without changing the name referred to in the config db can be done either via 1) DNS aliasing or using 2) SQL's own tools to add aliases. I also like the using the 3) host file entries, but this does require everyone know that it's done this way. Changing host file entries programatically in a flat file could be problematic, but is still scriptable. What if every server in the farm had an alias in a host entry on every other server in the farm... Interesting, but man does this take us forward or backward?

Either way, in your designs I totally recommend planning, and verification, not just once, but on a recurring schedule. Executing something like this on a monthly or bi-monthly basis is not overkill. It will keep your staff up to date with the solution and everyone can be on the top of their game.

Best Practice on Backups

joelo — Tue, 10 Jul 2007 04:56:00 GMT

In a recent thread with Bill Baer and an external consultant we had an interesting conversation on Backups, the Config DB, and Alternate Access mappings. The conversation was fairly rich, so I thought I'd share. You may want to start at the bottom, since this is an email thread. (portions removed to create anonymity)

Summary from my side on best practice... With disk based backups... Use catastrophic backup with a scheduled task for the most simple environment, then monitor the events for failed jobs or failed backups. Use SQL backups with jobs and then use stsadm to backup your indexes for the environment where you have a separate SQL team who monitor failed SQL jobs. It will be faster and easier to add other SQL jobs like DBCCs and verifications. For granular recovery, it's mostly about setting expectations and SLAs. The recycle bin in the product is a must use, you may want to consider extending the retention time at the web application level. For site backups/restores, I don't recommend doing stsadm backups on a daily basis unless you are ok with the overhead for disk space, and you don't mind blocking in the databases. You will see locks and blocking during site backups on large sites especially. What I do recommend is using the site delete capture tool on codeplex. Other granular recovery options exist with partner solutions such as from Avepoint, Commvault and Quest.

I do like DPM 2007 (http://www.microsoft.com/dpm) and am already a big fan (with the current beta and builds), but in the mean time I like what I've seen from third parties like AvePoint and Commvault and do recommend people check them out for full backup solutions.

Joel

From Bill Baer:

I have found the following >> "stsadm -o enumalternatedomains" which produces a great little export >>

AlternateDomains>

<UrlZone>Default</UrlZone>

</AlternateDomain>

</Collection>

<IncomingUrl>http://portal1.local</IncomingUrl>

<UrlZone>Default</UrlZone>

<MappedUrl>http://portal1.local</MappedUrl>

</AlternateDomain>

<IncomingUrl>https://portal1.local</IncomingUrl>

<UrlZone>Extranet</UrlZone>

<MappedUrl>https://portal1.local</MappedUrl>

</AlternateDomain>

</Collection>

<UrlZone>Default</UrlZone>

</AlternateDomain>

</Collection>

</AlternateDomains>

-----Original Message-----

From: Bill Baer

Sent: Friday, 6 July 2007 8:10 AM

To: Consultant

Cc: Joel Oleson

Subject: RE: Questions to pass on.

--Snip-- Correction SharePoint backup will not truncate your logs.

We rely on a combination of SQL backups and handle search through the OOB backup utility. This provides us a relatively safe recovery structure with a reliable percentage of success in terms of recovery. As we move forward in our DPM v2 testing we will likely drop the native backups and OOB backup of the search and indexing components. AAM is not backed up using the catastrophic backup, this is where DPM comes into the picture, but since it's not available to every customer - you should consider handling this via scripting or otherwise. I've never tried recovery based solely on catastrophic backup so cannot provide a great deal of input on configdb recovery issues in that space.

Load "Bill",8,1

-----Original Message-----

From: Joel Oleson [mailto:joelo@microsoft.com]

Sent: Friday, 6 July 2007 1:50 AM

To: MS

Subject: RE: Questions to pass on.

I'd say best practice is to use catastrophic backup to backup the index and Sql jobs to backup the databases. You don't want to back up the dbs twice.

For people that aren't comfortable with Sql... The easy thing to do is the catastrophic backup daily. The incremental diffs are good practice but do add to the complexity.

The config db restore is ok if all databases are being restored from a sql perspective, but under nearly all other circumstances you need a new config db which does mean redeploying features, solutions, remapping web apps... Redoing search and content sources doesn't make sense since that shouldn't be in the config db. I'd need more info.

You should loop in Bill Baer from MS IT and Samer Sawaya for details.

Joelo

-----Original Message-----

From: Consultant

To: MS

Sent: 7/5/07 5:44 AM

Subject: Questions to pass on.

Hi Could you please pass these questions on to the SharePoint guys:

1. The STSADM catastrophic backup switch also truncates the SQL logs (stsadm.exe -o backup -directory \\unc<file:///\\unc> -backupmethod full) once the backup is complete.

My question relates to "Best Practice"

In my opinion, "MOSS DRP Best Practice" should really be about telling our customers to perform both a SQL backup of databases as well as STSADM catastrophic backup (as above) to cover all bases.

Due to the fact that the Stsadm utility will truncate the SQL logs, should we be telling our customers to schedule the catastrophic backup after the SQL Database backup has occurred?.

By doing so, it ensures the transaction logs has been backed-up to tape before being "dumped".

Please pass this scenario on as I would like to get some feedback. Perhaps I'm being paranoid.

2. I have read the following two statements in Technet regarding the built-in backup:

a. SharePoint Backup/Restore will not backup "Alternate Access Mappings"

b. SharePoint Backup/Restore will not backup "Global Search Settings".

I'm just curious how we should be dealing with this if the tool doesn't cover it. Should we find a way to backup these settings via script or is this a manually re-instated setting? (which means also ensure it is documented to begin with). I want to ensure that we cover all aspects of SharePoint DRP in this document.

3. The Stsadm catatrophic restore option is able to restore the Configuration database and Central Admin database and web site.

I'm curious to obtain feedback from other SharePoint consultants out there regarding how you can actually use this to perform a DR as I can only seem to successfully bring back an environment by re-creating the Configuration database.

I'm assuming this option is available in the event of a corrupted Configuration database or Central Admin Database or web site.

Thank You.

Kind Regards,

XXX

Sr. Consultant

SharePoint Backup Restore, High Availability, and Disaster Recovery

joelo — Fri, 09 Mar 2007 01:49:00 GMT

This post has been long comming. I recently spent some time with Samer Sawaya of SharePoint Recycle bin fame :). We have some good memories working through that one. You can thank me later for the 2nd stage not getting cut, I fought hard for that. We...(read more)