SharePoint Hosting and Development : Disaster Recovery

Improving SharePoint with SQL Server 2008

Michael Watson — Tue, 19 Aug 2008 22:00:00 GMT

You might have noticed that SharePoint SP1 now supports SQL Server 2008 :-) http://blogs.msdn.com/sharepoint/archive/2008/08/15/sql-server-2008-support-for-sharepoint-products-and-technologies.aspx http://blogs.msdn.com/mikewat/archive/2008/08/19/sql-server-2008-is-now-officially-supported.aspx...(read more)

I’m Speaking at the SharePoint Best Practices Conference

Michael Watson — Wed, 13 Aug 2008 20:19:00 GMT

It’s official. I’ll be at the SharePoint Best Practices Conference September 15 th – 17 th in Washington DC. This will be a great conference. There are a lot of great speakers and sessions and the focus on best practices will cut out all the fluff. If...(read more)

SQL Mirroring Setup Made Easy

Michael Watson — Thu, 31 Jul 2008 07:38:00 GMT

In an ongoing effort to simplify mirroring I've decided to post the scripts I use to setup mirroring for myself. There are 9 easy steps to execute. The trick with most of these commands is that they output a script that you will need to copy into the...(read more)

DPM Search Backup White Paper.

Michael Watson — Fri, 11 Apr 2008 07:44:00 GMT

The DPM team released a whitepaper detailing how to backup MOSS search. http://www.microsoft.com/downloads/details.aspx?FamilyID=0150d17d-9f28-4ef6-8dc8-8fbb5fed5cfc&DisplayLang=en This is another feather in the DPM hat....(read more)

More Clarification Needed? Geographic Separation of SharePoint Farm Components.

Michael Watson — Thu, 03 Apr 2008 20:03:00 GMT

I continue to hear questions and debate over how to build local or regional immunity into a single SharePoint farm. Enterprising SharePoint folks want to make sure that their SharePoint service remains online even if Dr. Evil fires the “laser” at their...(read more)

The Newest Version of the Database Mirroring Whitepaper for SharePoint has been released

Michael Watson — Mon, 03 Mar 2008 21:08:00 GMT

Using Database Mirroring with Office SharePoint Server and Windows SharePoint Services Wow! This one took a while to update. We added more prescriptive information on the type of supported topologies. It also recommends SQL connection aliasing for failover...(read more)

Mirroring and Bandwidth

Michael Watson — Sat, 17 Nov 2007 03:15:00 GMT

I just completed some testing to see how much network bandwidth between SQL nodes affects synchronous mirroring performance. The SQL mirroring white paper indicates that bandwidth does not matter much above 10Mbps, but that may lead you to the wrong conclusion...(read more)

SQL Aliasing Pros and Cons

Michael Watson — Fri, 26 Oct 2007 22:14:00 GMT

I wanted to write a quick note on the good and bad of using SQL aliasing for redirecting SharePoint between mirrored SQL nodes. Good - It's a very simple solution. Each application server in the farm simply needs to be configured with the alias information....(read more)

Restoring SQL Server. Model DB Restore gotcha

Michael Watson — Fri, 26 Oct 2007 04:43:00 GMT

If somebody would have mentioned this, I would have 3 hours back. When recovering SQL server from a situation where you have lost the model database, make sure to remove the data and log files (if any exist) from the original directories before attempting...(read more)

DPM, SharePoint, and Database Mirroring

Michael Watson — Wed, 24 Oct 2007 05:10:55 GMT

Lately I've been working on System Center Data Protection Manager 2007 for protecting our SharePoint and SQL servers. If you are not familiar with DPM, DPM is all about protecting your servers by replicating block level differences to a disk based store....(read more)

Are we planning for the wrong disasters?

Michael Watson — Mon, 22 Oct 2007 09:13:27 GMT

I've been building a DR solution to protect against some major disasters. Earthquakes, tsunamis, nuclear war, global destruction, you know the fun stuff, but maybe I should be planning for more immediate disasters. http://blogs.msdn.com/briankel/archive/2007/10/21/my-building-was-flooded.asp...(read more)

Installing a SharePoint DR farm.

Michael Watson — Sun, 21 Oct 2007 09:09:42 GMT

I wanted to post a quick overview of how to install a DR farm to protect your main farm by log shipping your content databases to the second farm. The trick with the secondary farm is that the log shipped databases will be attached to the web application from the get-go. This allows you to browse the content at anytime using the secondary farm's local url and as bonus you get to search the attached sites so that search is functional all the time as well. See my earlier post for more information.

Here are the high level steps:

Install SQL
Configure log shipping for each content database that needs to be log shipped. Make sure to use the option to leave the database in standby mode and not to disconnect users when restoring logs. This particular configuration will not work without these settings.
Install SharePoint configuring the DR SQL instance as the default SQL server.
Patch to primary farm’s version.
Configure/Start all services.
Create SSP instance. Alternatively, you could restore the latest SSP backup to the farm. This would ensure all relevant settings are synchronized. You’ll need to disable any content source crawl schedules that should not be crawled while the farm is inactive.
Configure profile import.
Create a new web application for each web application that exists on the primary farm. Specify any database name for the content database. It will be deleted later.
Once the web application is created, DO NOT create a site collection. Instead, navigate to content databases and remove the default content database associated with each new web app.
Add each log shipped content database to the corresponding web application. Ensure the correct number of sites are displayed.
Export the primary farm’s SSL certificate (if applicable) and install it on the DR web servers.
Navigate to the web application AAM and add an AAM that corresponds to the primary site(s) url. Ensure SSL sites are referenced by “https://”
Open %windir%\system32\drivers\etc\hosts with notepad and add an entry for the AAM pointing at the local loopback address as follows:

127.0.01 WebServerUrl

Navigate to the SSP and click on search settings > Choose content sources and crawl schedules.
Locate the “local office server SharePoint sites” content source and choose edit.
Remove any url’s referring to the local farm or server(s) url. Replace with http://<primaryserverurl>. Do the same for the profile content source. sps3://primaryserverurl. Create or adjust the appropriate schedules and choose OK.
Navigate to mysite settings on the SSP site. Make sure the mysite settings mirror the primary farm.
Finalize the farm by ensuring any specific customizations, modifications, and configurations are installed.

Now you should be able to browse any site and use all the features with the exception that the databases are read-only and functionality that requires alteration won't work until the databases are brought online. See this article for more information on the ramifications of read-only databases. Additionally, any new site creations created on the primary farm won't be available on the secondary farm until those databases containing the sites are reattached. See my earlier post for more on this problem. I will post a slick solution that does this for you automatically in the coming days.

SharePoint and Mirroring: Taking action on failovers

Michael Watson — Sat, 20 Oct 2007 06:26:45 GMT

You might have seen the database mirroring whitepaper for SharePoint. It mentions a great article from the SQL folks on alerting on database mirroring events. One might think that monitoring the WMI events in SQL is a good or maybe the only way to take action on failovers. However, I propose a better method: Query the [Master].sys.database_mirroring view via a task or process. Here's the query:

Select mirroring_role from sys.database_mirroring where mirroring_role is not null

If the server is principal each database will return 1. If mirror, each database will return 2. Unfortunately, this isn't the only state you have to handle. Because each SQL server can be both the mirror for one database and principal for another, you have to monitor for this situation or SharePoint will be most unhappy. The query to do this is:

Select count(distinct mirroring_role) from sys.database_mirroring where mirroring_role is not null

If this query returns more than 1, you know that you have what we call a database mirroring split situation and you need to fail over some of the databases to return to a good state. However, you can't simply failover each principal database because it's assumed that the partial failover occurred for a reason. Take for instance, the situation where one of your content databases resides on a storage array that has failed. In that situation, you would not want to fail the database back over to the original node as it would not come online until the array is repaired. To handle that situation, you want to know the original state of SQL before running your failover logic. If the role is principal, but one of the databases has failed over, you want to fail over the rest of the databases. However, if the node is the mirror, you want to ignore the situation and let the principal handle it. This means your application or process to handle failovers needs to be stateful. I've been storing this state in a user created table called roles in the MSDB database, but I will be moving this to the registry as this seems like a more robust way to handle it.

BTW.. I experimented with triggering off of the database mirroring events. I found that there were quite a few events that fired when a database fails over. It was confusing determining how to use this information. Additionally, the events didn't tell me the information I wanted to know. I ended up having to query the sys.database_mirroring table regardless. After spending some time on this, I decided it was best just to create a windows service that queries the table as described above. Additionally, I created a simple SQL agent job that tried to do the same thing as the service. Unfortunately, my SQL skills are pretty poor as I prefer C#, but it's the thought that counts right? This TSQL sample will create a SQL agent job that runs once a minute and writes the current state to a table called roles residing in MSDB. It will also attempt to handle a database mirroring split exception. This is where the code breaks down as I noticed that instead of failing over the database to the mirror, it fails it back over to the principal. Anyway's, here you go. Have fun. Oh.. and as always this code is unsupported and is provided as a sample for you to create your own failover mechanism. I left a gotcha in the code so that you'll have to clean it up before you can use it. ha HAAA! as Phil Seben says.

   1: USE [msdb]   2: GO   3: /****** Object:  Job [Mirroring_State_Monitor]    Script Date: 10/15/2007 11:46:50 ******/   4: BEGIN TRANSACTION   5: DECLARE @ReturnCode INT   6: SELECT @ReturnCode = 0   7: /****** Object:  JobCategory [[Uncategorized (Local)]]]    Script Date: 10/15/2007 11:46:51 ******/   8: IF NOT EXISTS (SELECT name FROM msdb.dbo.syscategories WHERE name=N'[Uncategorized (Local)]' AND category_class=1)   9: BEGIN  10: EXEC @ReturnCode = msdb.dbo.sp_add_category @class=N'JOB', @type=N'LOCAL', @name=N'[Uncategorized (Local)]'  11: IF (@@ERROR <> 0 OR @ReturnCode <> 0) GOTO QuitWithRollback  12:    13: END  14:    15: DECLARE @jobId BINARY(16)  16: EXEC @ReturnCode =  msdb.dbo.sp_add_job @job_name=N'Mirroring_State_Monitor',   17:         @enabled=1,   18:         @notify_level_eventlog=0,   19:         @notify_level_email=0,   20:         @notify_level_netsend=0,   21:         @notify_level_page=0,   22:         @delete_level=0,   23:         @description=N'No description available.',   24:         @category_name=N'[Uncategorized (Local)]',   25:         @owner_login_name=N'domain\username', @job_id = @jobId OUTPUT  26: IF (@@ERROR <> 0 OR @ReturnCode <> 0) GOTO QuitWithRollback  27: /****** Object:  Step [Check State]    Script Date: 10/15/2007 11:46:52 ******/  28: EXEC @ReturnCode = msdb.dbo.sp_add_jobstep @job_id=@jobId, @step_name=N'Check State',   29:         @step_id=1,   30:         @cmdexec_success_code=0,   31:         @on_success_action=1,   32:         @on_success_step_id=0,   33:         @on_fail_action=2,   34:         @on_fail_step_id=0,   35:         @retry_attempts=0,   36:         @retry_interval=0,   37:         @os_run_priority=0, @subsystem=N'TSQL',   38:         @command=N'  39: -- This script monitors the state of the mirror partners and takes action to fix any problems  40: -- This script uses a user created table [MSDB].dbo.role to store the state of the individual node  41: -- Principal = 1; Mirror = 2;  42: --  43: -- Identify if we have a split state  44: -- Get the role from the role table and make sure we don''t have more than 1 database role  45: IF (Select r.role from [MSDB].dbo.role r ) = 1 AND (Select count(distinct mirroring_role) from sys.database_mirroring where mirroring_role is not null) > 1  46:     BEGIN  47:         --Generate list of principal databases  48:         DECLARE @principalDB as table(dbname varchar(255) NOT NULL);  49:         --Insert the names of the principal databases into the table variable  50:         INSERT INTO @principalDB SELECT d.name  51:         from sys.database_mirroring s, sys.databases d  52:         WHERE s.database_id = d.database_id and s.mirroring_role = 1;  53:         --Enumerate through the principal databases and fail over  54:         WHILE (Select count(*) from @principalDB) > 0  55:             BEGIN  56:             --Get the first database name from the table  57:             DECLARE @dbName as varchar(100)  58:             SET @dbName = (SELECT Top 1 dbname from @principalDB)  59:             --Create a string to store the failover command  60:             Declare @cmdString as varchar(255)  61:             Set @cmdString = ''USE MASTER; ALTER DATABASE '' + @dbName + '' SET PARTNER FAILOVER;''  62:             --Execute the command string to failover the databases  63:             EXECUTE (@cmdString);  64:             --Delete the database from the list since it has aleady been failed over  65:             DELETE FROM @principalDB WHERE dbname = @dbName  66:             END  67:         --Finally, update the role to mirror  68:         Update [MSDB].dbo.role SET role = 2  69:     END  70: ELSE IF (Select r.role from [MSDB].dbo.role r ) = 1   71: AND (Select count(distinct mirroring_role) from sys.database_mirroring where mirroring_role is not null) = 1  72: AND (Select count(mirroring_role) from sys.database_mirroring where mirroring_role = 1) = 0  73:     -- This server is principal in name only but should be the mirror  74:     -- Update the role to mirror  75:     Update [MSDB].dbo.role SET role = 2  76: IF (Select r.role from [MSDB].dbo.role r ) = 2   77: AND (Select count(distinct mirroring_role) from sys.database_mirroring where mirroring_role is not null) = 1  78: AND (Select count(mirroring_role) from sys.database_mirroring where mirroring_role = 2) = 0  79:     -- This server is mirror in name only but should be the principal  80:     -- Update role to principal  81:     Update [MSDB].dbo.role SET role = 1  82:    83: -- We are done  84: -- Other potential states exist, but we don''t need to perform any action.  85:     Update [MSDB].dbo.role SET role = 2  86: IF (Select r.role from [MSDB].dbo.role r ) = 2   87: AND (Select count(distinct mirroring_role) from sys.database_mirroring where mirroring_role is not null) = 1  88: AND (Select count(mirroring_role) from sys.database_mirroring where mirroring_role = 2) = 0  89:     -- This server is mirror in name only but should be the principal  90:     -- Update role to principal  91:     Update [MSDB].dbo.role SET role = 1  92:    93: -- We are done  94: -- Other potential states exist, but we don''t need to perform any action.',   95:         @database_name=N'master',   96:         @flags=0  97: IF (@@ERROR <> 0 OR @ReturnCode <> 0) GOTO QuitWithRollback  98: EXEC @ReturnCode = msdb.dbo.sp_update_job @job_id = @jobId, @start_step_id = 1  99: IF (@@ERROR <> 0 OR @ReturnCode <> 0) GOTO QuitWithRollback 100: EXEC @ReturnCode = msdb.dbo.sp_add_jobschedule @job_id=@jobId, @name=N'Schedule 1',  101:         @enabled=1,  102:         @freq_type=4,  103:         @freq_interval=1,  104:         @freq_subday_type=4,  105:         @freq_subday_interval=1,  106:         @freq_relative_interval=0,  107:         @freq_recurrence_factor=0,  108:         @active_start_date=20071015,  109:         @active_end_date=99991231,  110:         @active_start_time=0,  111:         @active_end_time=235959 112: IF (@@ERROR <> 0 OR @ReturnCode <> 0) GOTO QuitWithRollback 113: EXEC @ReturnCode = msdb.dbo.sp_add_jobserver @job_id = @jobId, @server_name = N'(local)' 114: IF (@@ERROR <> 0 OR @ReturnCode <> 0) GOTO QuitWithRollback 115: COMMIT TRANSACTION 116: GOTO EndSave 117: QuitWithRollback: 118:     IF (@@TRANCOUNT > 0) ROLLBACK TRANSACTION 119: EndSave:

Quick note about index backup

Michael Watson — Sat, 20 Oct 2007 00:26:00 GMT

As you may already know, the only way to restore an index without having to recrawl is to use the out-of-the-box backup mechanism either in the UI or via STSADM catastrophic backup option. Using STSADM allows you to schedule the backup, but I was asked a question the other day that I couldn't immediately answer.

Does STSADM allow you to backup only selected items such as the index or do you have to perform a backup of everything?

The concern was that in our current design the collab sites share the same farm with the SSP and that all the collab content would have to be backed up in addition to the SSP using the out-of-the-box mechanism. Obviously, that wouldn't be good for a large farm where you depend on some sort of differencing mechanism to ensure you can backup all content within the backup window. Well I tested it and found that the SharePoint folks were on top of this (as expected, right :-) ) and created an STSADM option that allows you to chose which item you want to backup just like you do in the GUI. The syntax is:

stsadm -o backup -directory <SharePath> -backupmethod <full | diff> -item <object name>

In our case <object name> is the SSP name which allows us to backup only the content needed to restore the ssp/index. Of course, you can chose any item you can see in the UI backup option, including individual content databases.

A note about capacity and backup performance. The MSIT index contains about 25 million items and the component sizes are as follows:

Search DB: ~370GB

SSP DB: ~65GB

Index (on the file system) ~150GB

When backing up this content (selecting just the SSP in backup) I'm told I need about 700GB of storage, but you'll notice that the cumulative size of this content is about 585GB. I'm not quite sure why it needs the additional 100GB of space, but I do know that the backup only consumes 413GB after it completes. More investigation is needed to understand the differences, however, my buddy Sam Crewdson tells me that DB fragmentation contributes greatly to the overall backup size. When he defragment's the DB, the size of the backup files are reduced.

Regardless, backup performance seems to be a factor of four things. How large is the content, what are the network limitations, what are the hardware limitations, and where is the backup share. For the longest time, MSIT backed up the index to a share on the index server. The problem with this approach is that the search DB (residing in SQL) is usually the largest component and has to be streamed from the SQL server across the network to the index server. That's not cool. Even with Gig/E, it can take quite a while to transfer that much data. A much better approach is to put the share on the SQL server. Now only the index file, which is usually the smaller of the two, has to cross the network. After making the change IT saw a major decrease in backup duration. In the neighborhood of 60%.

So how long? Well in my lab with nothing else going on, I can backup the aforementioned SSP in about 4 hours. That's about 100GB/hour. That's with some pretty awesome hardware. Your mileage will vary of course.

Mike

Do Both. Now with more pics.

Michael Watson — Tue, 31 Jul 2007 08:09:17 GMT

In a previous post, I talked about how I was working on implementing mirroring inside the datacenter for high availability and log shipping between datacenters for disaster recovery. Well I wanted a reason to use my freshly installed live writer and its wicked cool capabilities so I decided to show a screenshot of that model.

I apologize about the image size. Seems my blog format doesn't offer much real estate. You'll notice that the two large circles represent two datacenters. In this model the datacenters are likely separated by hundreds or even thousands of miles. In the primary datacenter you'll find the minimal highly available topology with two web servers, an index server, and mirrored SQL servers. You'll notice that all databases are mirrored including the configuration database. Now one thing to note about the mirroring configuration is that this isn't really supported yet so if you implement this right now, you are on your on. I do hope that testing will show this configuration is valid in the not too distant future. Also noteworthy is that the index server is also running SQL server and serves as the witness role for the highly available mirroring pair.

In the disaster recovery datacenter, you'll find a single SQL server and a combined SharePoint web/app/index server. Why not another highly available farm? Money. It costs money to build and maintain servers in the remote datacenter so this configuration saves money by discarding some flexibility. Note, I said flexibility, not performance. If you have two web servers you should really be able to support their load on a single server or your farm is not truly highly available. The same goes for SQL. The exception in this model is the index server role which can be intensive, but it is also one you can control through scheduling and throttling.

From a SQL perspective, the farm has its own local versions of all the standard databases with one exception. We are log shipping the content database(s) from the primary farm to the secondary farm. With each log shipped database in standby mode we can attach them to the local web application which the local SSP is configured to search. The trick here is to make the index think it's crawling the url of the primary datacenter, not the local url. You can do this by configuring an AAM for the primary site on the DR web app and adding an entry in the hosts file pointing the url to the localhost address. Finally, remove the local url from the index scope and replace with the AAM. Now schedule the crawl and you will have near seamless failover with instant query. One caveat. When log shipping content databases, any new sites added to your databases will not automatically be available. This is because each new site must also be registered in the configuration database, but without going through the normal process of site creation the only other way to make the site available is to reattach the containing content database. That's easy enough to script, but the problem is that every time you add or reattach a content database, that database gets a new GUID assigned. That doesn't sound bad, but it has major ramifications on search. Each time your content database gets a new GUID, the index server thinks the sites and content in that database are new. I need to continue testing the overall effects of this on search, but so far the most obvious symptom is more crawling.

There are many alternatives to this approach. Check back often as I'm sure I'll be talking about these in the future.