SQL Server Storage Engine : Backup

When is too much success a bad thing?

kfarlee — Tue, 30 Oct 2007 19:18:00 GMT

I was talking to a customer the other day who had an interesting problem: Successful backups.

Specifically, their problem had to do with the success messages that SQL backup puts in the SQL errorlog and the system event log.

Seems like a nice, friendly thing to do right? Drop a note that your backup was successful, note the LSN, etc.

The problem comes when you do a LOT of backups. In this case, the customer was doing 1 T-log backup per minute on EACH of 5 production databases. That adds up to an errorlog entry every 12 seconds. As you can see, the errorlog very quickly gets bloated with success messages. This in turn causes two issues:

Managing the size of the log itself
Finding anything actionable in the flood of success messages.

In this customer's case they had an added complication: They were using a monitoring tool which scanned the SQL errorlog for certain error codes. As it so happens, every so often the LSN noted in a success message just happened to match with some error condition, and the DBA got woken up to deal with his "corrupt database".

So, I went spelunking, and found a jewel which might save some of you some problems.

There is a traceflag, 3226, which suppresses these success messages from both the SQL errorlog and the system event log. By enabling this traceflag you will no longer get the flood of success messages in your logs. Of course, if you've implemented some sort of logic which depends on the content of these messages, that would be a bad thing, but for most people who do very frequent backups, this could be an asset.

This traceflag has been in the product since SQL 2000 so any existing instances can take advantage of it immediately.

Kevin Farlee

Storage Engine PM

Fuzzy backups and RESTORE DATABASE WITH STOPAT

kfarlee — Sat, 27 Oct 2007 01:25:00 GMT

So what’s a “fuzzy backup” and how does it relate to STOPAT?

There has been some confusion about what RESTORE DATABASE WITH STOPAT does, if it works, and why it behaves as it does. This is not about RESTORE LOG WITH STOPAT, but only RESTORE DATABASE. To understand how this works and why, you need to understand the concept of fuzzy backups and how they work at restore time.

When we restore a database backup, we need to end up with all pages in the database at the same effective LSN. This is a challenge when you consider that it may take hours to copy all the pages in a very large database.

When a database backup starts, we note the LSN which is the min of:

· The last checkpoint

· Repl-start

· The oldest active transaction.

.
We then start copying pages. Some are copied at the beginning of the
backup.
Some may be copied hours later, depending on the size of the database.
In order to restore a database which is self-consistent, when we're done all
the pages need to be at the same effective LSN.
We could do that by locking the database during the backup, but that MIGHT
cause some of you operational problems :)

What we do instead is note the LSN when the last data copy operation
completes, and copy the portion of the log from the begin backup to the end
backup LSNs into the backup file.
The last stage of a database restore is to use that segment of log to run a
REDO pass to bring all pages up to the state as of the end of the backup.

Getting back to STOPAT:

A full database backup can only be restored to the time of the end of the
backup. That means that you can’t STOPAT a point midway through the backup. Consider: Some pages were copied after that STOPAT time, so we cannot make the database consistent.

So what good is STOPAT when restoring a database? Its purpose is to flag an error if the backup is too recent to be able to reach the STOPAT time. For example:

· Full backup starts at T1

· User decides to restore database to T0, using a combination of full and log backups.

· If the backup in step 1 is used, the database cannot be restored to T0, so backup throws an error.

The Ins and Outs of Offline Files

Paul Randal - MSFT — Sat, 02 Jun 2007 22:35:00 GMT

First blog post of the year from TechEd! Well, Kimberly and I arrived a day early hoping to chill out in the sun by the pool but contrary to all expectations the weather sucks today – it’s actually much better in Seattle. Instead we're hibernating in our room blogging and having a Blokus re-match – last time we were in Orlando in March she beat me soundly so I need to get my revenge :-)

Over the next few posts I want to touch on some of the issues that have been causing confusion on the MSDN Disaster Recovery forum. First up is offline files.

One of the cool things you can do in SQL Server 2005 is set a particular file to be offline and then restore just that file from backups. The rest of the database stays online as long as the file isn’t part of the primary filegroup or a transaction log file. This allows you to restore a damaged database file without having to take the whole database offline.

You have to be careful though, depending on what recovery mode the database is in:

If you’re using simple recovery mode, the file in question has to be part of a read-only filegroup and you have to have a backup of the file that was taken after the filegroup was made read-only.
If you’re using full or bulk-logged recovery mode, you have to have a backup of the file and a complete transaction log backup chain.

The main point to emphasize here is that you have to have a backup! Once a file has been set offline, the ONLY way to bring it back online is to restore it from backups. I’ve worked up a simple script that illustrates how to do this.

Firstly I create a database offlinefiletest and add a filegroup with a single file called damagedfile.

USE master;

-- create our test database

IF DATABASEPROPERTY (N'offlinefiletest', 'Version') > 0

DROP DATABASE offlinefiletest;

CREATE DATABASE offlinefiletest ON (

NAME = fgt_mdf,

FILENAME = 'c:\offlinefiletest\offlinefiletest.mdf',

SIZE = 2MB)

LOG ON (

NAME = fgt_log,

FILENAME = 'c:\offlinefiletest\offlinefiletest.ldf',

SIZE = 1MB);

-- Add a filegroup with a single file

ALTER DATABASE offlinefiletest ADD FILEGROUP offlinefg;

ALTER DATABASE offlinefiletest ADD FILE (

NAME = damagedfile,

FILENAME = 'c:\offlinefiletest\damagedfile.ndf',

SIZE = 512KB) TO FILEGROUP offlinefg;

Next I take a full database backup and a transaction log backup. Note that I’m using the CHECKSUM options – see my previous blog post here for more info on these.

BACKUP DATABASE offlinefiletest TO

DISK = 'c:\offlinefiletest\offlinefiletest.bak'

WITH INIT, CHECKSUM;

BACKUP LOG offlinefiletest TO

DISK = 'c:\offlinefiletest\offlinefiletest_log.bak'

WITH INIT, CHECKSUM;

Now I set the damagedfile offline and check the sys.database_files system catalog view to ensure the file has been taken offline.

ALTER DATABASE offlinefiletest MODIFY FILE (NAME = damagedfile, OFFLINE);

-- Check the file state in the system catalog

SELECT file_id,

CONVERT (CHAR (15), RTRIM (name)) AS name,

CONVERT (CHAR (15), RTRIM (state_desc)) AS state

FROM offlinefiletest.sys.database_files WHERE name = 'damagedfile';

And we get back:

file_id name state

----------- --------------- ---------------

3 damagedfile OFFLINE

There are a bunch of states that a file can be in – the full list is in the Books Online entry for sys.database_files. Ok – so our file is offline. Let’s try setting it online again.

ALTER DATABASE offlinefiletest MODIFY FILE (NAME = offlinefile, ONLINE);

And we get back:

Msg 155, Level 15, State 1, Line 4

'ONLINE' is not a recognized CREATE/ALTER DATABASE option.

That’s because there’s no T-SQL syntax to explicitly set a file online – you HAVE to restore it from a backup. Let’s see what our backup contains using:

RESTORE FILELISTONLY FROM DISK = 'c:\offlinefiletest\offlinefiletest.bak';

It has all three files. We’ll start the restore process by specifying which file we want to restore and also giving the NORECOVERY option – this allows us to restore subsequent log backups too.

RESTORE DATABASE offlinefiletest FILE = 'damagedfile'

FROM DISK = 'c:\offlinefiletest\offlinefiletest.bak'

WITH NORECOVERY;

Now we’ve started the restore process, let’s check the state of the file again using the same sys.database_files query as above. We get the following results:

file_id name state

----------- --------------- ---------------

3 damagedfile RESTORING

The file is now listed as RESTORING. We complete the restore process by restoring all the log backups in the backup chain (in our case only one):

RESTORE LOG offlinefiletest FROM DISK = 'c:\offlinefiletest\offlinefiletest_log.bak'

WITH RECOVERY;

And now if we check the sys.database_files system catalog view again we should see that the file is online again:

file_id name state

----------- --------------- ---------------

3 damagedfile ONLINE

It is - success!!

Bottom line here – this is a powerful feature that allows for much better availability in the event of a disaster BUT you must be setup to use it. As with all features, especially around disaster recovery, make sure you know the requirements and limitations for successful use. I've attached the complete script for you to play with.

Ok – now there’s a little bit of blue sky – chill time! Hope to see a bunch of you during the week – see here for a list of what we’re up to session-wise.

Can you mix-n-match backup devices?

Paul Randal - MSFT — Mon, 23 Apr 2007 03:15:00 GMT

I was asked this question yesterday and didn't know the answer so thought it would be good for a quick post. Can you use devices from your main and mirrored backup media sets together to perform a restore?

The code below creates a single-device backup and then examines it.

BACKUP DATABASE AdventureWorks

TO DISK = N'c:\mediaset1device1.bck'

WITH FORMAT, STATS;

RESTORE HEADERONLY FROM DISK=N'c:\mediaset1device1.bck';

The BackupSize in the output is 172,044,800 bytes and the on-disk size of the file is 164MB. Now let's use two devices and see what happens.

BACKUP DATABASE AdventureWorks

TO DISK=N'c:\mediaset1device1.bck', DISK=N'c:\mediaset1device2.bck'

MIRROR TO DISK=N'c:\mediaset2device1.bck', DISK=N'c:\mediaset2device2.bck'

WITH FORMAT, STATS;

RESTORE HEADERONLY FROM DISK=N'c:\mediaset1device1.bck';

The BackupSize is now 344,113,152 bytes (roughly double what it was previously because we've mirrored the whole backup) and the on-disk size of each file is 82MB (half of what it was for a single-device). This proves that the backup has been split over the two devices. Now let's try to mix devices from the two backup media sets and see if it's possible:

RESTORE DATABASE AdventureWorks

FROM DISK=N'c:\mediaset1device1.bck', DISK=N'c:\mediaset2device2.bck'

WITH REPLACE, STATS;

And it works fine - excellent! That's the whole point of having mirrored backups.

The second part of the question was about whether backup device types can differ between media sets in the same backup. The answer to this is no - as documented in Books Online. All the backup devices involved in a single backup, regardless of whether they're part of a mirror media set or not, must be of the same type and have similar characteristics.

Next up - can it hurt to have thousands of filegroups?

Oh no – my backup is corrupt too! Help!

Paul Randal - MSFT — Tue, 06 Mar 2007 00:59:00 GMT

Funny how a bunch of people all seem to have the same problem at the same time. Maybe it’s just that people don’t want to talk about corruption until someone else does – it’s like a dark secret that once let out of the bag – everyone talks. The common HA/DR issue over the last few days on the newsgroups and forums seems to be not just data corruption but backup corruption too.

It goes back to the statement that you don’t have a backup until you’ve done a restore. What do I mean by that? I don’t mean you need to restore it back over the original database, but unless you’ve checked the validity of the backup, how do you know it’s going to work in a disaster recovery situation?

You really want to check two things:

The integrity of the database being backed up
The integrity of the backup file itself

Check #1 is easy – run some form of DBCC consistency checks before taking a full backup – otherwise you may be backing up a corrupt database. In fact, I was just dealing with a customer who unfortunately had PFS page header corruption (which cannot be repaired through DBCC) in their database and then found out that it was also in their backup. Their choice then became restoring from a week-old backup or exporting all data into a new database – neither of which is particularly palatable…

Sometimes it may not be possible to run CHECKDB on the source database - commonly on VLDB systems or system with high, continuous workload where the extra overhead required cannot be tolerated for the length of the CHECKDB. In that case, consider running CHECKDB on a restored copy of the database. This not only proves to you that the backup is valid (because it restores correctly) but that the source database was not corrupt at the time the backup was taken. This is a common strategy on systems that where CHECKDB cannot be run. The only problem with this approach is – what does it mean if the CHECKDB comes back with errors? You can’t tell whether the restored database or the source database is corrupt so you’re forced to go back to the source database and run another CHECKDB. Either way – you’ve successfully caught some corruption before it has a chance to really bite you.

I’ve just explained a component of check #2 – validating the backup. The other thing to do before doing the backup in SQL Server 2005 is choosing to use the WITH CHECKSUM option on the backup command. This will do two things while the backup is being taken:

Evaluate and check all existing page checksums or torn-page protection on database pages as they’re read
Calculate a checksum over the whole backup stream and save it in the backup

This can help ensure that the database being backed up doesn’t have any corruption caused by the IO-subsystem.

Once the backup is taken, it’s possible to verify all of this without actually writing out anything – using RESTORE … WITH VERIFYONLY. This will do everything short of writing bytes to disk, including rechecking any existing page checksums or torn-page protection on pages in the backup and recalculating and checking the checksum on the entire backup.

So, there’s a lot you can do to cut down on the likelihood of having a corrupt backup when the time comes to use it. Now – what happens if you get into the situation from the title of this post – the database AND the backup are corrupt?

Well, many people would throw up their hands in despair, but there is one thing you can do – allow the restore to ignore errors. There’s an option on restore called CONTINUE_AFTER_ERROR which will let the restore continue even if page checksum failure or corruptions in the backup stream are found. The catch is that the database may be inconsistent afterwards so you’ll need to work out what data is corrupt and do an export/import or run repair. In the best scenario, the only corruptions needing repair will be in non-clustered indexes, in which case you can rebuild them or allow repair to fix them up. In the worst scenario, the corruptions will be in the base table data, you’ll need to use REPAIR_ALLOW_DATA_LOSS; however, this may still be better than not being able to restore the backup at all.

Bottom line – make sure you validate your backups after you take them so you don’t get into a difficult situation. Also, be sure to keep more than one business cycle’s worth of backups (not just the last week – but maybe the last month) so that if a backup becomes bad later (through tape degradation, etc), you will have other options for restore and recovery.

What Master???

kfarlee — Tue, 26 Sep 2006 20:18:00 GMT

I've come across a few interesting DR scenarios lately, (interesting in the sense that the resolution wasn't immediately obvious, and fortunately the answer didn't start out with "update your resume...")

The situation that comes to mind is a customer who was doing all the right things.

He had a disaster recovery plan. He had written scripts to implement the steps so there wouldn't be mistakes in the heat of the moment. HE WAS TESTING HIS PLAN (kudos) when he discovered the issue here:

His plan included backing up all user databases, as well as the system DBs, and then using a 3rd-party backup program to protect the system as a whole, providing one-button bare metal restore. Sounds like a great plan, right? What could go wrong?

The plan and the SQL scripts worked perfectly when he tested them out on a running system. Unfortunately, when he tested the end-to-end bare metal restore, SQL wouldn't come up. There was an error message about Master being corrupt which prevented the instance from starting.

No problem! We've got a backup of Master right here. Just restore it and we're on the road to recovery... Well, almost. The catch is that you have to have the instance running in single-user mode in order to restore master. That means it has to be running. In order for the instance to run, you need at least some viable copy of Master in place. Catch 22, right?

Fortunately, there is a solution. Setup provides a mechanism to recreate the system datbases without reinstalling all of the SQL binaries, but it's not obvious how to find it in BOL. The magic incantation turns out to be:

start /wait <CD or DVD Drive>\setup.exe /qn INSTANCENAME=<Instance> REINSTALL=SQL_Engine REBUILDDATABASE=1 SAPWD=<NewStrongPassword>

From there, you'll have a running instance with no user databases and all default settings. You can then restore master, msdb, model, and the rest of your databases and be on the road.

For our friend, there was an additional complication: He was running Express. The reason that this complicated matters is that Express is normally installed by downloading the compressed installer, which expands itself, runs the installation, and then cleans up the install files. Unfortunately, with the command above, MSI is going to look for the .MSI files in the location that SQL was installed from, which no longer exists, since it was cleaned up...

So, there are a couple of extra steps: First is to download the Express kit, and extract the contents to a local directory. This is done by saving the SQLEXPRESS.EXE file and NOT running it from the download prompt. Instead, after the download completes, manually run the program with the -x switch. i.e. D:\Expressdir\SQLEXPRESS.EXE -x

This results in the setup files being extracted, but no other steps are taken. Now we need to instruct MSI to use these files and not the remembered location when rebuilding the databases. This is done by adding the REINSTALLMODE=vomus option to the command:

start /wait <CD or DVD Drive>\setup.exe /qn INSTANCENAME=SQLExpress REINSTALL=SQL_Engine REBUILDDATABASE=1 SAPWD=<NewStrongPassword> REINSTALLMODE=vomus

And it's just that easy!

Next time I'll talk about a more disastrous situation and the desperate measures required to deal with it.

Why you should NOT optimize your backup strategy for backup speed

kfarlee — Tue, 20 Jun 2006 07:50:00 GMT

Sounds kinda funny, doesn't it? I mean, who here doesn't have someone breathing down your neck every time you go past your backup window? So why would you NOT optimize for fast backups? It just makes sense. You do backups every day, so they should be optimized.

Two words:

Disaster Recovery

If you think people get cranky when you go 20 minutes past your backup window, try taking 2 days to get your mission-critical database back on the air! That kind of attention is not pleasant.

What I'm trying to get across is that you really want to think through your DR strategy from the standpoint of a recovery plan instead of a backup plan. Plan exactly how you will recover in different scenarios. Work out the most efficient way to get back in the air from every situation you can think of. Then work back and make sure that you have a backup strategy that gives you the building blocks you'll need to accomplish those plans.

That's not to say that you can't have both; It is entirely possible to come up with a plan which minizes both backup and restore time. You just have to plan it that way. The extreme example is the person who does a full backup once a month, and LOTS of log backups in between. Log backups are quick, right? So, the backup strategy is optimal, right? But it would be a nightmare applying all of those log backups to recover from a disaster.

With SQL Server 2005 Enterprise, there are lots of features that allow higher availability and faster recovery, and they should be looked into. One Example:

Partial Database Availability This is huge! In a nutshell, this feature means that as soon as your primary filegroup is online, your database is available. Obviously, only those filegroups which are online can be accessed, but at least you can work with whatever is online. What this looks like in a DR situation is this:

Restore and roll forward your primary filegroup, and the minimal set of additional filegroups required to get your most important app running.
At this point, your most critical customers are already online.
Then, at your leisure, restore/roll forward additional filegroups containing less critical data such as historical records.

In this way you can get back on the air far quicker than what it would take to restore your entire 2TB database. Unless you plan ahead and break your database up into appropriate filegroups, this won't work for you.

The point to all this is to encourage you to plan and rehearse disaster recovery before the lights go out and everyone is in headless-chicken mode. Rational decisions are rarely made in a panic.

Kevin Farlee

Just why are you backing up your database?

kfarlee — Thu, 15 Jun 2006 17:35:00 GMT

So, I'd like to make a couple of assertions about backup and disaster recovery that will hopefully get you to thinking, and just might change the way you think about the topic.

There are two distinct motivations for doing backups, and they imply differing requirements and solutions.

OK, what am I talking about? Ask yourself the question, as obvious as it seems, "Why do you take backups every night?" I'm sure you'll come up with lots of answers, but if you think about it, they fall into two classes:

Disaster Recovery
Archiving and records retention

I will argue that these are very different requirements. For the first case, if you are doing a backup for DR purposes, you almost always want the very most recent version of the database that you can get. 1 ms before the disk crashed, or before the user hit COMMIT on that DROP TABLE where he thought he was in the test DB would be perfect.

I have rarely if ever seen a requirement to recover from a catastrophic outage from data that's over a week old. So, this implies that you may not need to keep this class of backups around for long, but they must be very quickly available.

In the second case, you do need to keep the backups around for a long time. In some cases, a VERY long time. But, for archival purposes, they really don't need to be so fine-grained. If you need the database from Q1 2005, you probably don't need to see it in 12 hour increments for every day in the quarter. Typically, your business cycles will dictate that there are distinct times from which you may need to see the data.

So, how can this be exploited?

If you separate the two backup requirements, you can come up with solutions for DR like disk snapshots or simply keeping the backups on spinning (disk) media for a week or so, where this would clearly not be economically feasible for archival purposes.

On the other hand, for archival purposes, you might be able to get away with far less frequent backups, something which would clearly not be acceptable for DR purposes.

You can see then, that you can combine these to come up with strategies which give you very fast (in some cases instantaneous) backups, very fast restores, and far less time spent in taking archival copies of your database.

Kevin Farlee

SQL Storage Engine PM