SQL Server Storage Engine : DBCC

Example corrupt database to play with and some backup/restore things to try

Paul Randal - MSFT — Tue, 17 Apr 2007 20:22:00 GMT

I've been asked several times over the last few weeks for an example corrupt database to play with, and for testing logic built around DBCC CHECKDB.

The attached WinZip file contains a backup of a simple 2005 database called 'broken' (I can do a 2000 one too if there's enough demand). It has a simple table called 'brokentable' (c1 int, c2 varchar(7000)) with one row in it. The table has a single data page with page ID (1:143) that I've corrupted so the page checksum is corrupt.

This means you can try all sorts of cool things. Below I've listed a few things you can try out to see what would happen on your database if a checksum failed.

Restore

I didn't do anything special when backing up this database so restoring works just fine, even though it contains a corrupt page.

RESTORE DATABASE broken FROM DISK='c:\broken.bck'

WITH MOVE 'broken' TO 'c:\broken.mdf',

MOVE 'broken_log' TO 'c:\broken_log.ldf';

Processed 160 pages for database 'broken', file 'broken' on file 1.

Processed 2 pages for database 'broken', file 'broken_log' on file 1.

RESTORE DATABASE successfully processed 162 pages in 0.314 seconds (4.203 MB/sec).

DBCC CHECKDB

CHECKDB throws us some nice errors:

DBCC CHECKDB ('broken') WITH NO_INFOMSGS, ALL_ERRORMSGS;

Msg 8928, Level 16, State 1, Line 1

Object ID 2073058421, index ID 0, partition ID 72057594038321152, alloc unit ID 72057594042318848 (type In-row data): Page (1:143) could not be processed. See other errors for details.

Msg 8939, Level 16, State 98, Line 1

Table error: Object ID 2073058421, index ID 0, partition ID 72057594038321152, alloc unit ID 72057594042318848 (type In-row data), page (1:143). Test (IS_OFF (BUF_IOERR, pBUF->bstat)) failed. Values are 12716041 and -4.

CHECKDB found 0 allocation errors and 2 consistency errors in table 'brokentable' (object ID 2073058421).

CHECKDB found 0 allocation errors and 2 consistency errors in database 'broken'.

repair_allow_data_loss is the minimum repair level for the errors found by DBCC CHECKDB (broken).

Query errors

Any query that touches that page is going to fail with an 824 error:

SELECT * FROM broken..brokentable;

Msg 824, Level 24, State 2, Line 1

SQL Server detected a logical consistency-based I/O error: incorrect checksum (expected: 0x7232c940; actual: 0x720e4940). It occurred during a read of page (1:143) in database ID 8 at offset 0x0000000011e000 in file 'c:\broken.mdf'. Additional messages in the SQL Server error log or system event log may provide more detail. This is a severe error condition that threatens database integrity and must be corrected immediately. Complete a full database consistency check (DBCC CHECKDB). This error can be caused by many factors; for more information, see SQL Server Books Online.

Backup with CHECKSUM

If you have page checksums turned on, you should always use the WITH CHECKSUM option on backups to validate the page checksums as they are read:

BACKUP DATABASE broken TO DISK='c:\broken2.bck'

WITH CHECKSUM;

Msg 3043, Level 16, State 1, Line 1

BACKUP 'broken' detected an error on page (1:143) in file 'c:\broken.mdf'.

Msg 3013, Level 16, State 1, Line 1

BACKUP DATABASE is terminating abnormally.

But we can force it to backup. If this is the only copy of the database we have, and we're being forced to run repair, for instance, then we want to make sure we have a backup to restore from. Even a backup that contains a corrupt database is better than no database at all. In this case, we need to use the CONTINUE_AFTER_ERROR option.

BACKUP DATABASE broken TO DISK='c:\broken2.bck'

WITH CHECKSUM, CONTINUE_AFTER_ERROR;

Processed 160 pages for database 'broken', file 'broken' on file 1.

Processed 1 pages for database 'broken', file 'broken_log' on file 1.

BACKUP WITH CONTINUE_AFTER_ERROR successfully generated a backup of the damaged database. Refer to the SQL Server error log for information about the errors that were encountered.

BACKUP DATABASE successfully processed 161 pages in 2.025 seconds (0.651 MB/sec).

RESTORE VERIFYONLY

Now that we have a backup that has checksum information in it, let's see how we can check whether the backup is valid:

RESTORE VERIFYONLY FROM DISK='c:\broken2.bck';

The backup set was written with damaged data by a BACKUP WITH CONTINUE_AFTER_ERROR.

Isn't that cool? It tells us that the backup was already corrupt when it was written. Ok - let's ask it to specifically check the checksums in the backup:

RESTORE VERIFYONLY FROM DISK='c:\broken2.bck'

WITH CHECKSUM;

The backup set was written with damaged data by a BACKUP WITH CONTINUE_AFTER_ERROR.

Same thing. What about if we try to check the checksums on the initial backup?

RESTORE VERIFYONLY FROM DISK='c:\broken.bck'

WITH CHECKSUM;

Msg 3187, Level 16, State 1, Line 1

RESTORE WITH CHECKSUM cannot be specified because the backup set does not contain checksum information.

Msg 3013, Level 16, State 1, Line 1

VERIFY DATABASE is terminating abnormally.

RESTORE

How about we try to overwrite the existing 'broken' database with the one from the second backup that has checksum information in it?

RESTORE DATABASE broken FROM DISK='c:\broken2.bck'

WITH REPLACE;

Msg 3183, Level 16, State 1, Line 1

RESTORE detected an error on page (1:143) in database "broken" as read from the backup set.

Msg 3013, Level 16, State 1, Line 1

RESTORE DATABASE is terminating abnormally.

It won't let us because the backup contains corrupt data. But we can force it if we need to with CONTINUE_AFTER_ERROR again.

RESTORE DATABASE broken FROM DISK='c:\broken2.bck'

WITH REPLACE, CONTINUE_AFTER_ERROR;

Processed 160 pages for database 'broken', file 'broken' on file 1.

Processed 1 pages for database 'broken', file 'broken_log' on file 1.

The backup set was written with damaged data by a BACKUP WITH CONTINUE_AFTER_ERROR.

RESTORE WITH CONTINUE_AFTER_ERROR was successful but some damage was encountered. Inconsistencies in the database are possible.

RESTORE DATABASE successfully processed 161 pages in 0.392 seconds (3.364 MB/sec).

Isn't that cool? It works BUT it tells us that the backup set contained corrupt data and that the database was restored but could have corrupt data in.

Summary

Have a play about with this database to familiarize yourself with the kind of responses you'll get from the various tools when a corruption exists, and how to work around it if need be. Let me know if you want to see any particular kinds of corruptions explored, or want a database with something specific corrupted in.

Ok - this turned into a longer post than I originally planned :-)

(PS - about 20 people have offered to fill in my VLDB maintenance survey so far - would be great to have some more. The DVD I'm giving as a reward has a bunch of great hands-on labs, including one that does cool backup/restore stuff to work around disasters. See the previous few posts for details. Thanks.)

Oh no – my backup is corrupt too! Help!

Paul Randal - MSFT — Tue, 06 Mar 2007 00:59:00 GMT

Funny how a bunch of people all seem to have the same problem at the same time. Maybe it’s just that people don’t want to talk about corruption until someone else does – it’s like a dark secret that once let out of the bag – everyone talks. The common HA/DR issue over the last few days on the newsgroups and forums seems to be not just data corruption but backup corruption too.

It goes back to the statement that you don’t have a backup until you’ve done a restore. What do I mean by that? I don’t mean you need to restore it back over the original database, but unless you’ve checked the validity of the backup, how do you know it’s going to work in a disaster recovery situation?

You really want to check two things:

The integrity of the database being backed up
The integrity of the backup file itself

Check #1 is easy – run some form of DBCC consistency checks before taking a full backup – otherwise you may be backing up a corrupt database. In fact, I was just dealing with a customer who unfortunately had PFS page header corruption (which cannot be repaired through DBCC) in their database and then found out that it was also in their backup. Their choice then became restoring from a week-old backup or exporting all data into a new database – neither of which is particularly palatable…

Sometimes it may not be possible to run CHECKDB on the source database - commonly on VLDB systems or system with high, continuous workload where the extra overhead required cannot be tolerated for the length of the CHECKDB. In that case, consider running CHECKDB on a restored copy of the database. This not only proves to you that the backup is valid (because it restores correctly) but that the source database was not corrupt at the time the backup was taken. This is a common strategy on systems that where CHECKDB cannot be run. The only problem with this approach is – what does it mean if the CHECKDB comes back with errors? You can’t tell whether the restored database or the source database is corrupt so you’re forced to go back to the source database and run another CHECKDB. Either way – you’ve successfully caught some corruption before it has a chance to really bite you.

I’ve just explained a component of check #2 – validating the backup. The other thing to do before doing the backup in SQL Server 2005 is choosing to use the WITH CHECKSUM option on the backup command. This will do two things while the backup is being taken:

Evaluate and check all existing page checksums or torn-page protection on database pages as they’re read
Calculate a checksum over the whole backup stream and save it in the backup

This can help ensure that the database being backed up doesn’t have any corruption caused by the IO-subsystem.

Once the backup is taken, it’s possible to verify all of this without actually writing out anything – using RESTORE … WITH VERIFYONLY. This will do everything short of writing bytes to disk, including rechecking any existing page checksums or torn-page protection on pages in the backup and recalculating and checking the checksum on the entire backup.

So, there’s a lot you can do to cut down on the likelihood of having a corrupt backup when the time comes to use it. Now – what happens if you get into the situation from the title of this post – the database AND the backup are corrupt?

Well, many people would throw up their hands in despair, but there is one thing you can do – allow the restore to ignore errors. There’s an option on restore called CONTINUE_AFTER_ERROR which will let the restore continue even if page checksum failure or corruptions in the backup stream are found. The catch is that the database may be inconsistent afterwards so you’ll need to work out what data is corrupt and do an export/import or run repair. In the best scenario, the only corruptions needing repair will be in non-clustered indexes, in which case you can rebuild them or allow repair to fix them up. In the worst scenario, the corruptions will be in the base table data, you’ll need to use REPAIR_ALLOW_DATA_LOSS; however, this may still be better than not being able to restore the backup at all.

Bottom line – make sure you validate your backups after you take them so you don’t get into a difficult situation. Also, be sure to keep more than one business cycle’s worth of backups (not just the last week – but maybe the last month) so that if a backup becomes bad later (through tape degradation, etc), you will have other options for restore and recovery.

How long does your CHECKDB take?

Paul Randal - MSFT — Tue, 13 Feb 2007 07:29:00 GMT

Following on from my post a couple of weeks ago (https://blogs.msdn.com/sqlserverstorageengine/archive/2007/01/24/how-long-will-checkdb-take-to-run.aspx), I'm very interested to know how long it takes for your CHECKDBs to run, so I can get an idea of the distribution of run-times on various kinds of hardware for various size databases.

So, if you have a couple of minutes, I'd be grateful if you could c&p the following into an email to me, prandal@microsoft.com, and answer the questions for one or more of your databases. I'd especially like to hear from those of you who work for Microsoft and look after multiple customers. If I get a worthwhile number of responses (say 100 or so) then I'll collate the responses and post a summary. I'd love to hear about databases larger than 1TB.

For the first 10 Microsoft and the first 10 non-Microsoft people to respond, I'll send you the Always-On DVD loaded with Hands-On Labs that walk you through a bunch of the various HA/DR technologies. (I'll let you know whether you win and you need to send me your address).

Many thanks!

How large is the database?
What's the size of the largest table?
Do you have any indexed views?
Which command do you run, including options?
How long does the command take to run?
How often do you run it?
Which version & edition of SQL Server are you running?
(If SQL Server 2005, did you notice an increase or decrease in runtime?)
How many CPUs/cores does the server have?
How much memory does the server have?
What platform is the server?
Describe your IO subsystem?
Any comments?

CHECKDB (Part 8): Can repair fix everything?

Paul Randal - MSFT — Sun, 04 Feb 2007 19:53:00 GMT

I was teaching at a Microsoft-internal class last week and there was a discussion on what corruptions can't be repaired using DBCC. At the same time, several threads popped up on forums and newsgroups with people hitting some of this unrepairable corruptions so I thought that would make a good topic for the next post in the CHECKDB series.

Before anyone takes this the wrong way - what do I mean by "can't be repaired"? Remember that that purpose of repair is to make the database structurally consistent, and that to do this usually means deleting the corrupt data/structure (that's why the option to do this was aptly named REPAIR_ALLOW_DATA_LOSS - see this post for more explanation on why repair can be bad). A corruption is deemed unrepairable when it doesn't make sense to repair it given the damage the repair would cause, or the corruption is so rare and so complicated to repair correctly that it's not worth the engineering effort to provide a repair. Remember also that recovery from corruptions should be based on a sound backup strategy, not on running repair, so making this trade-off in functionality makes perfect sense.

Here's a few of the more common unrepairable corruptions that people run into along with the reasons they can't be repaired by DBCC.

PFS page header corruption

An example of this is on SQL Server 2005:

Msg 8946, Level 16, State 12, Line 1
Table error: Allocation page (1:13280496) has invalid PFS_PAGE page header values.
Type is 0. Check type, alloc unit ID and page ID on the page.

CHECKDB uses the PFS pages to determine which pages are allocated - and so which pages to read to drive the various consistency checks. The only repair for a PFS page is to reconsutruct it - they can't simply be deleted as they're a fixed part of the fabric of the database. PFS pages cannot be rebuilt because there is no infallible way to determine which pages are allocated or not. There are various algorithms I've experimented with to rebuild them with optimistic or pessimistic setting of page allocation statuses and then re-run the various consisteny checks to try to sort out the incorrect choices, but they all require very long run-times. Given the frequency with which we see these corruptions, and the engineering effort required to come up with an (imperfect) solution, I made the choice to leave this as unrepairable.

Critical system table clustered-index leaf-page corruption

An example of this is on SQL Server 2000:

Server: Msg 8966, Level 16, State 1, Line 1
Could not read and latch page (1:18645) with latch type SH. sysindexes failed.

And on SQL Server 2005:

Msg 7985, Level 16, State 2, Server SUNART, Line 1
System table pre-checks: Object ID 4. Could not read and latch page (1:51) with
latch type SH. Check statement terminated due to unrepairable error.

In a previous post I described why how and why we do special checks of the clustered indexes of the critical system tables. If any of the pages at the leaf-level of these indexes are corrupt, we cannot repair them. Repairing would mean deallocating the page, wiping out the most important metadata for potentially hundreds of user tables and so effectively deleteing all of these tables. That's obviously an unpalatable repair for anyone to allow and so we don't do it.

Column value corruption

Here's an example of this on SQL Server 2005:

Msg 2570, Level 16, State 3, Line 1
Page (1:152), slot 0 in object ID 2073058421, index ID 0, partition ID 72057594038321152, alloc unit ID 72057594042318848 (type "In-row data"). Column "c1" value is out of range for data type "datetime". Update column to a legal value.

This is where a column has a stored value that is outside the valid range for the column type. There are a couple of repairs we could do for this:

delete the entire record
insert a dummy value

#1 isn't very palatable because then we've lost data and its not a structural problem in the database so doesn't have to be repaired. #2 is dangerous - what value should we choose as the dummy value? Any value we put in may adversely affect business logic, or fire a trigger, or have some unwelcome meaning in the context of the table - even a NULL. Given these problems, we chose to allow people to fix the corrupt values themselves.

Metadata corruption

Here's an example of this on SQL Server 2005:

Msg 3854, Level 16, State 1, Line 2
Attribute (referenced_major_id=2089058478) of row (class=0,object_id=2105058535,column_id=0,referenced_major_id=2089058478,referenced_minor_id=0) in sys.sql_dependencies has a matching row (object_id=2089058478) in sys.objects (type=SN) that is invalid.

This example is relatively benign. There are other examples that will cause CHECKDB to terminate - not as bad as the critical system table corruption example above, but enough that CHECKDB doesn't trust the metadata enough to use it to drive consistency checks. Repairing metadata corruption has the same problems as repairing critical system table corruption - any repair means deleting metadata about one or more tables, and hence deleting the tables themselves. It's far better to leave the corruption unrepaired so that as much data as possible can be extracted from the remaining tables.

Summary

Repair can't fix everything. You may end up having to perform manual and time-consuming data extraction from the corrupt database and losing lots of data because of, say, a critical system table corruption. Bottom line (as usual) - make sure you have valid backups so you don't get into this state!

CHECKDB (Part 7): How long will CHECKDB take to run?

Paul Randal - MSFT — Wed, 24 Jan 2007 16:12:00 GMT

This is a question I see every so often and it cropped up again this morning so I'll use it as the subject for this week's blog post.

There are several ways I could answer this:

the unhelpful answer - I've got no idea.
the almost-helpful answer - how long did it take to run last time and are the conditions exactly the same?
the answer I usually give - it depends.

Now, many people would see the third answer as being somewhat equivalent to the first answer - unhelpful. The problem is that there are many factors which influence how long CHECKDB will take to run. Let me explain the ten most important factors so you get an idea why this is actually a helpful answer. These aren't in any particular order of importance.

1) The size of the database

Pretty obvious...

2) Concurrent IO load on the server

At the simplest level, what is CHECKDB going to do? It reads every allocated page in the database. That's a lot of IO. CHECKDB takes great pains to do the most efficient IO it can and read the database pages in their physical order with plenty of readahead so that the disk heads move smoothly across the disks (rather than jumping around randomly and incurring disk head seek delays). If there's no concurrent IO load on the server, then the IOs will be as efficient as CHECKDB can make them. However, introducing any additional IO from SQL Server means that the disk heads will be jumping around - slowing down the CHECKDB IOs. If the IO subsystem is at capacity already from CHECKDB's IO demands, any additional IO is going to reduce the IO bandwidth available to CHECKDB - slowing it down.

3) Concurrent CPU activity on the server

At the next level of simplicity, CHECKDB is going to process every page it reads in some way. Depending on the various options you've specified and the database schema (details below), that's going to use a lot of CPU - it's possible that the server may be pegged at 100% CPU when CHECKDB is running. If there's any additional workload on the server, that's going to take CPU cycles away from CHECKDB and it going to slow it down.

Basically what these first two points are saying is that CHECKDB is very resource intensive! Its probably one of the most resource intensive things you can ask SQL Server to do and so it's usually a good idea to not run it during peak workload times, as you'll not only cause CHECKDB to take longer to run, you will slowdown the concurrent workload, possibly unacceptably.

4) Concurrent update activity on the database

This is relevant for both SQL 2000 and SQL 2005, but for different reasons.

In SQL 2000, CHECKDB gets its consistent view of the database from transaction log analysis of concurrent DML transactions (see CHECKDB (Part 1): How does CHECKDB get a consistent view of the database for full details). The more concurrent DML there is while CHECKDB is running, the more transaction log will be generated - and so the longer it will take for CHECKDB to analyze that transaction log. It's possible that on a large multi-CPU box with a ton of concurrent DML and CHECKDB limited to a single CPU that this phase of CHECKDB could take several times longer than the reading and processing of the database pages! (I've seen this in real-life several times.)

In SQL 2005, CHECKDB gets its consistent view of the database from a database snapshot, which is stored on the same disk volumes as the database itself. If there are a lot of changes in the database while CHECKDB is running, the changed pages are pushed to the snapshot so that it remains consistent. As the snapshot is stored on the same disks as the database, every time page is pushed to the snapshot, the disk head has to move, which interrupts the efficient IO described in #2. Also, whenever CHECKDB goes to read a page and it needs to read the page from the snapshot files instead of the database files, that's another disk head move, and another efficient IO interruption. The more concurrent changes to the database, the more interruptions to efficient IO and the slower the CHECKDB runs.

5) Throughput capabilities of the IO subsystem

This one's simple. CHECKDB is going to do a boat-load of IOs and it could even end up being IO-bound (meaning that the CPUs are idle periodically waiting for IOs to complete) depending on the options specified and the database schema. This means that the throughput of the IO subsystem is going to have a direct effect on the run-time of CHECKDB. so, if you have a 1TB database and the IO subsystem can only manage 100MB/sec, it's going to take almost 3 hours just to read the database (1TB / 100MB / 3600 secs) and there's nothing you can do to speed that up except upgrade the IO subsystem.

I've lost count of the number of times I've heard customers complain that CHECKDB (or index rebuilds or other IO-heavy operations) are running sloooowly only to find that the disk queue lengths are enormous and the IO subsystem it entirely unmatched to the server and workload.

6) The number of CPUs on the box

This also really encompasses the Edition of SQL Server that's being run. In Enterprise Edition, CHECKDB can run in parallel across all the CPUs in the box (or as many as the query processor decides to parallelize over when the CHECKDB internal queries are compiled). Running in parallel can give a significant performance boost to CHECKDB and lower run times, as long as the database is also spread over multiple files too (so the IOs can be parallelized). There's a nifty algorithm we use that allows us to run in parallel whcih I'll explain in detail in a future post.

On the other hand, however, the fact that CHECKDB can run in parallel in Enterprise Edition can be bad for some scenarios, and so some DBAs chose to force CHECKDB to be single-threaded. The way to do this is to turn on the documented trace flag 2528.

7) The speed of the disks where tempdb is placed

Running CHECKDB against a VLDB uses lots of memory for internal state and for VLDBs the memory requirement usually exceeds the amount of memory available to SQL Server. In this case, the state is spooled out to tempdb and so the performance of tempdb can be a critical factor in CHECKDB performance.

8) The complexity of the database schema

This can have a really high impact on the run-time of CHECKDB because it impacts the amount of CPU that CHECKDB requires. For example, the most expensive checks that CHECKDB does are for non-clustered indexes. We need to check that each row in a non-clustered index maps to exactly one row in the heap or clustered index for the table, and that every heap/clustered index row has exactly one matching row in each non-clustered index. Although there's a highly efficient algorithm for doing this, it still takes around 30% of the total CPU that CHECKDB uses!

There are a bunch of other checks that are only done if the features have been used in the database - e.g. computed column evaluation, links between off-row LOB values, Service Broker, XML indexes, indexed views - so you can see that empirical factors along aren't enough to determine the run-time.

9) Which options you've specified

This is almost the same as #7 in that by specifying various options you're limiting what checks CHECKDB actually performs. For instance, using the WITH NOINDEX option will turn off the non-clustered index checks that I described in #7 and using the WITH PHYSICAL_ONLY option will turn off all logical checks, vastly decreasing the run-time of CHECKDB and making it nearly always IO-bound rather than CPU-bound (in fact this is the most common option that DBAs of VLDBs use to make the run-time of CHECKDB manageable).

One thing to be aware of - if you specify any repair options, CHECKDB always runs single-threaded, even on a multi-proc box.

10) The number and type of corruptions that exist in the database

Again, this is similar to #7 and #8. If there are any corruptions present, there may be extra checks triggered to try to figure out more details of the corruptions. For instance, for the non-clustered index checks, the algorithm is tuned very heavily for the case when there are no corruptions present (the overwhelming majority of cases considering the millions of times CHECKDB is run every day around the world). When a non-clustered index corruption is detected, a more in-depth algorithm has to be used to figure out exactly where the corruption is, which involves re-scanning a bunch of data and so taking a bunch more time. There are a few other algorithms like this too.

Summary

So you can see that there's no simple answer.

Now, there's only one time when you should be trying to work out how long a CHECKDB is going to take - when you're planning your regular database maintenance (if you find that it's going to take too long, checkout some of the options for splitting up the checks in my previous post on CHECKDB (Part 6): Consistency checking options for a VLDB). If you're faced with a corrupt (or suspected corrupt) database and you're only just starting to think about how long a CHECKDB is going to take - you've screwed up. You should at least know how long a CHECKDB usually takes.

Of course, there's one more really unhelpful answer I could have given - how long is a piece of string?!? :-)

Fixing damaged pages using page restore or manual inserts

Paul Randal - MSFT — Thu, 18 Jan 2007 03:02:00 GMT

Here's an interesting scenario that cropped up today. You have a database on a RAID array that failed and has zero'd out a page. How can you get the data back?

There are two ways to do it, depending on the database recovery model and version of SQL Server - single-page restore or manual insert/select - both of which rely on you having a backup of the database. You can use single-page restore if you're on SQL Server 2005 and the database is in Full or Bulk-Logged recovery mode, otherwise you need to use the manual method, and that will only work as long as you know the data being salvaged hasn't changed since the last backup.

Let's try them both. Here's a script to create a test database and make a backup of it:

-- Create the database.

USE master;

GO

CREATE DATABASE dbccpagetest;

GO

ALTER DATABASE dbccpagetest SET RECOVERY FULL;

GO

-- Create a table to play with.

USE dbccpagetest;

GO

CREATE TABLE sales (

salesID INT IDENTITY,

customerID INT DEFAULT CONVERT (INT, 100000 * RAND ()),

salesDate DATETIME DEFAULT GETDATE (),

salesAmount MONEY);

CREATE CLUSTERED INDEX salesCI ON sales (salesID);

GO

-- Populate the table

SET NOCOUNT ON;

GO

DECLARE @count INT

SELECT @count = 0

WHILE (@count < 5000)

BEGIN

INSERT INTO sales (salesAmount) VALUES (100 * RAND ());

SELECT @count = @count + 1

END;

GO

-- Take a full backup.

BACKUP DATABASE dbccpagetest TO DISK = 'C:\dbccpagetest.bak' WITH INIT;

GO

I'm going to simulate our scenario by shutting down the database and using a hex editor to zero out page 158 of the database. (This translates to byte offset 1294336 of the file being zero'd for 8192 bytes).

Now if I run checkdb, I get the following:

Msg 8909, Level 16, State 1, Line 1

Table error: Object ID 0, index ID -1, partition ID 0, alloc unit ID 0 (type Unknown), page ID (1:158) contains an incorrect page ID in its page header. The PageId in the page header = (0:0).

CHECKDB found 0 allocation errors and 1 consistency errors not associated with any single object.

Msg 8928, Level 16, State 1, Line 1

Object ID 2073058421, index ID 1, partition ID 72057594038386688, alloc unit ID 72057594042384384 (type In-row data): Page (1:158) could not be processed. See other errors for details.

CHECKDB found 0 allocation errors and 1 consistency errors in table 'sales' (object ID 2073058421).

CHECKDB found 0 allocation errors and 2 consistency errors in database 'dbccpagetest'.

repair_allow_data_loss is the minimum repair level for the errors found by DBCC CHECKDB (dbccpagetest).

What does the page look like?

DBCC TRACEON (3604);

GO

DBCC PAGE (dbccpagetest, 1, 158, 3);

GO

DBCC execution completed. If DBCC printed error messages, contact your system administrator.

PAGE: (0:0)

BUFFER:

BUF @0x02C0632C

bpage = 0x04C12000 bhash = 0x00000000 bpageno = (1:158)

bdbid = 9 breferences = 0 bUse1 = 37241

bstat = 0xc00009 blog = 0x89898989 bnext = 0x00000000

PAGE HEADER:

Page @0x04C12000

m_pageId = (0:0) m_headerVersion = 0 m_type = 0

m_typeFlagBits = 0x0 m_level = 0 m_flagBits = 0x200

m_objId (AllocUnitId.idObj) = 0 m_indexId (AllocUnitId.idInd) = 0 Metadata: AllocUnitId = 0

Metadata: PartitionId = 0 Metadata: IndexId = -1 Metadata: ObjectId = 0

m_prevPage = (0:0) m_nextPage = (0:0) pminlen = 0

m_slotCnt = 0 m_freeCnt = 0 m_freeData = 0

m_reservedCnt = 0 m_lsn = (0:0:0) m_xactReserved = 0

m_xdesId = (0:0) m_ghostRecCnt = 0 m_tornBits = 16777216

Allocation Status

GAM (1:2) = ALLOCATED SGAM (1:3) = NOT ALLOCATED

PFS (1:1) = 0x60 MIXED_EXT ALLOCATED 0_PCT_FULL DIFF (1:6) = NOT CHANGED

ML (1:7) = NOT MIN_LOGGED

Msg 2514, Level 16, State 5, Line 2

DBCC PAGE error: Invalid page type - dump style 3 not possible.

Note the error at the end of the output - DBCC PAGE can't do an in-depth dump because it doesn't know what page type the page is. Let's try a full page hex dump using dump style 2 instead:

DBCC PAGE (dbccpagetest, 1, 158, 2);

GO

PAGE: (0:0)

<page header etc skipped for brevity>

DATA:

Memory Dump @0x44F3C000

44F3C000: 00000000 00020000 00000000 00000000 †................

44F3C010: 00000000 00000000 00000000 00000000 †................

<deleted for brevity>

44F3DFE0: 00000000 00000000 00000000 00000000 †................

44F3DFF0: 00000000 00000000 00000000 00000000 †................

DBCC execution completed. If DBCC printed error messages, contact your system administrator.

It really is all zero. First we'll fix it using page restore.

USE master;

GO

RESTORE DATABASE dbccpagetest PAGE = '1:158' FROM DISK = 'C:\dbccpagetest.bak';

GO

Processed 1 pages for database 'dbccpagetest', file 'dbccpagetest' on file 1.

The roll forward start point is now at log sequence number (LSN) 32000000047000001. Additional roll forward past LSN 33000000001700001 is required to complete the restore sequence.

RESTORE DATABASE ... FILE=<name> successfully processed 1 pages in 0.176 seconds (0.046 MB/sec).

Isn't that cool? You can restore up to 1000 single pages from a backup at a time. For VLDBs, this cuts the recovery time WAY down. Now we need to roll forward the log. We don't have any more log backups so we can finish the roll forward by backing up and restoring the tail of the log.

-- Need to complete roll forward. Backup the log tail...

BACKUP LOG dbccpagetest TO DISK = 'C:\dbccpagetest_log.bak' WITH INIT;

GO

-- ... and restore it again.

RESTORE LOG dbccpagetest FROM DISK = 'C:\dbccpagetest_log.bak';

GO

Processed 5 pages for database 'dbccpagetest', file 'dbccpagetest_log' on file 1.

BACKUP LOG successfully processed 5 pages in 0.146 seconds (0.248 MB/sec).

Processed 0 pages for database 'dbccpagetest', file 'dbccpagetest' on file 1.

RESTORE LOG successfully processed 0 pages in 0.004 seconds (0.000 MB/sec).

And now we should have a clean database:

DBCC CHECKDB (dbccpagetest) WITH NO_INFOMSGS;

GO

Command(s) completed successfully.

Easy. But what if we can't do a page restore? Assuming I've corrupted the database in exactly the same way again, the first thing is to do is make sure we can restore the backup and then see what data range is on that page:

RESTORE DATABASE dbccpagetest_copy FROM DISK = 'C:\dbccpagetest.bak' WITH

MOVE N'dbccpagetest' TO N'C:\dbccpagetest_copy.mdf',

MOVE N'dbccpagetest_log' TO N'C:\dbccpagetest_log.ldf',

REPLACE;

GO

DBCC PAGE (dbccpagetest_copy, 1, 158, 3);

GO

Processed 184 pages for database 'dbccpagetest_copy', file 'dbccpagetest' on file 1.

Processed 2 pages for database 'dbccpagetest_copy', file 'dbccpagetest_log' on file 1.

RESTORE DATABASE successfully processed 186 pages in 0.361 seconds (4.205 MB/sec).

PAGE: (1:158)

BUFFER:

BUF @0x02BE8D38

bpage = 0x03FB4000 bhash = 0x00000000 bpageno = (1:158)

bdbid = 10 breferences = 1 bUse1 = 38283

bstat = 0xc00009 blog = 0x159a2159 bnext = 0x00000000

PAGE HEADER:

Page @0x03FB4000

m_pageId = (1:158) m_headerVersion = 1 m_type = 1

m_typeFlagBits = 0x4 m_level = 0 m_flagBits = 0x8200

m_objId (AllocUnitId.idObj) = 68 m_indexId (AllocUnitId.idInd) = 256

Metadata: AllocUnitId = 72057594042384384

Metadata: PartitionId = 72057594038386688 Metadata: IndexId = 1

Metadata: ObjectId = 2073058421 m_prevPage = (1:157) m_nextPage = (1:159)

pminlen = 28 m_slotCnt = 245 m_freeCnt = 11

m_freeData = 7691 m_reservedCnt = 0 m_lsn = (24:453:8)

m_xactReserved = 0 m_xdesId = (0:0) m_ghostRecCnt = 0

m_tornBits = -1020457745

Allocation Status

GAM (1:2) = ALLOCATED SGAM (1:3) = NOT ALLOCATED

PFS (1:1) = 0x60 MIXED_EXT ALLOCATED 0_PCT_FULL DIFF (1:6) = NOT CHANGED

ML (1:7) = NOT MIN_LOGGED

Slot 0 Offset 0x60 Length 31

Record Type = PRIMARY_RECORD Record Attributes = NULL_BITMAP

Memory Dump @0x4542C060

00000000: 10001c00 d5030000 5bd30000 3f771101 †........[...?w..

00000010: b9980000 baa10a00 00000000 0500e0††††...............

UNIQUIFIER = [NULL]

Slot 0 Column 1 Offset 0x4 Length 4

salesID = 981

Slot 0 Column 2 Offset 0x8 Length 4

customerID = 54107

Slot 0 Column 3 Offset 0xc Length 8

salesDate = Jan 17 2007 4:35PM

Slot 0 Column 4 Offset 0x14 Length 8

salesAmount = 69.68

<deleted for brevity>

Slot 244 Offset 0x1dec Length 31

Record Type = PRIMARY_RECORD Record Attributes = NULL_BITMAP

Memory Dump @0x4542DDEC

00000000: 10001c00 c9040000 bfa10000 57771101 †............Ww..

00000010: b9980000 c6b80500 00000000 0500e0††††...............

UNIQUIFIER = [NULL]

Slot 244 Column 1 Offset 0x4 Length 4

salesID = 1225

Slot 244 Column 2 Offset 0x8 Length 4

customerID = 41407

Slot 244 Column 3 Offset 0xc Length 8

salesDate = Jan 17 2007 4:35PM

Slot 244 Column 4 Offset 0x14 Length 8

salesAmount = 37.50

DBCC execution completed. If DBCC printed error messages, contact your system administrator.

So we're looking at salesID range 981 to 1225 inclusive. Before we can copy the rows back to the damaged database, we need to get rid of the corrupt page. Repair should delete the page for us. First I'm going to take another backup though - just in case something goes wrong!

BACKUP DATABASE dbccpagetest TO DISK = 'C:\dbccpagetest_corrupt.bak' WITH INIT;

GO

ALTER DATABASE dbccpagetest SET SINGLE_USER;

GO

DBCC CHECKDB (dbccpagetest, REPAIR_ALLOW_DATA_LOSS) WITH NO_INFOMSGS;

GO

ALTER DATABASE dbccpagetest SET MULTI_USER;

GO

Processed 184 pages for database 'dbccpagetest', file 'dbccpagetest' on file 1.

Processed 4 pages for database 'dbccpagetest', file 'dbccpagetest_log' on file 1.

BACKUP DATABASE successfully processed 188 pages in 0.380 seconds (4.052 MB/sec).

Msg 8909, Level 16, State 1, Line 1

Table error: Object ID 0, index ID -1, partition ID 0, alloc unit ID 0 (type Unknown), page ID (1:158) contains an incorrect page ID in its page header. The PageId in the page header = (0:0).

The error has been repaired.

CHECKDB found 0 allocation errors and 1 consistency errors not associated with any single object.

CHECKDB fixed 0 allocation errors and 1 consistency errors not associated with any single object.

Repair: The Clustered index successfully rebuilt for the object "dbo.sales" in database "dbccpagetest".

Repair: The page (1:158) has been deallocated from object ID 2073058421, index ID 1, partition ID 72057594038386688, alloc unit ID 72057594042384384 (type In-row data).

Msg 8945, Level 16, State 1, Line 1

Table error: Object ID 2073058421, index ID 1 will be rebuilt.

The error has been repaired.

Msg 8928, Level 16, State 1, Line 1

Object ID 2073058421, index ID 1, partition ID 72057594038386688, alloc unit ID 72057594042384384 (type In-row data): Page (1:158) could not be processed. See other errors for details.

The error has been repaired.

Msg 8976, Level 16, State 1, Line 1

Table error: Object ID 2073058421, index ID 1, partition ID 72057594038386688, alloc unit ID 72057594042384384 (type In-row data). Page (1:158) was not seen in the scan although its parent (1:154) and previous (1:157) refer to it. Check any previous errors.

The error has been repaired.

Msg 8978, Level 16, State 1, Line 1

Table error: Object ID 2073058421, index ID 1, partition ID 72057594038386688, alloc unit ID 72057594042384384 (type In-row data). Page (1:159) is missing a reference from previous page (1:158). Possible chain linkage problem.

The error has been repaired.

CHECKDB found 0 allocation errors and 3 consistency errors in table 'sales' (object ID 2073058421).

CHECKDB fixed 0 allocation errors and 3 consistency errors in table 'sales' (object ID 2073058421).

CHECKDB found 0 allocation errors and 4 consistency errors in database 'dbccpagetest'.

CHECKDB fixed 0 allocation errors and 4 consistency errors in database 'dbccpagetest'.

We should check the row count to see that the count has dropped from the initial 5000 rows we inserted:

USE dbccpagetest;

GO

SELECT COUNT (*) FROM SALES;

GO

SELECT COUNT (*) FROM sales WHERE salesID > 980 AND salesID < 1226;

GO

And we're down to 4755 rows, as expected with zero rows in that range. All we need to do now is to copy the missing rows over from the restored copy. Remember, this will only work if you know that the data being salvaged hasn't changed since the backup was taken - otherwise you'll have old and new data mixed in the table which will play havoc with your business. Before we copy the rows, we know we're got an identity column we'd like to preserve so we set IDENTITY_INSERT on which tells the server not to generate new identity values for the inserted rows.

-- Make sure identity values survive.

SET IDENTITY_INSERT sales ON;

GO

-- Insert the missing rows.

SET NOCOUNT OFF;

GO

INSERT INTO sales (salesID, customerID, salesDate, salesAmount)

SELECT * FROM dbccpagetest_copy.dbo.sales AS R

WHERE R.salesID > 980 AND R.salesID < 1226;

GO

-- Restore identity behavior.

SET IDENTITY_INSERT sales OFF;

GO

(245 row(s) affected)

We copy over 245 rows and checking the row count again says we're back to 5000 rows.

I want to post a lot more scenarios like this from now on - this may mean the posts are a little less meaty than last year but should happen a lot more often. If there's any particular scenario you'd like to see covered, add a comment and let me know.

(First post of the year! The diving vacation was excellent - 100 foot visibility, 85F water, and amazing creatures. I managed to get in 28 dives in the 8 diving days I had. Here's a picture of me about 40ft underwater in Indonesia to close.)

More undocumented fun: DBCC IND, DBCC PAGE, and off-row columns

Paul Randal - MSFT — Thu, 14 Dec 2006 01:46:00 GMT

(Final blog post of the year for me. Its been a bit of a wild ride the last 6 months - 7 TechEds on 3 continents, 46 blog posts and some major life changes - but now things have calmed down and I should be back to more regular posting in 2007. Tomorrow I fly out to Wakatobi in Indonesia to go diving with my partner until January, which is some long overdue R&R. I'll post a link to some photos in January - she's already been out there for two weeks on a live-aboard and has over 3000 photos... In the meantime, I have all my loose-ends tied up at work so time to squeeze in a post. Thanks for all your comments and emails this year - I hope you all have a great holiday and may your pagers be silent throughout!)

At TechEd in November, in one of the sessions I did with Kimberly we demo'd both DBCC IND and DBCC PAGE in the context of following links to columns that have been pushed off-row when the row size exceeds 8060 bytes. The point of the demo is to show that even if an index is perfectly defragmented, the performance of a query that does a range scan may suffer if off-row columns are part of the result set because getting to those off-row columns involves random IOs.

What I'd like to do in this post is run through the demo we did, and at the same time introduce you to the undocumented DBCC IND command. This is like DBCC PAGE - it's used extensively internally but isn't documented or supported - use at your own risk. You already know about using DBCC PAGE to investigate page contents from my previous posts (here and here) so I'm not going to go into details.

To start off, we create a table with a schema that can cause rows to be greater than 8060 bytes. In previous versions, creating a schema like this has always been possible, but actually getting rows larger then 8060 was not. Checkout my previous post on IAM chains and allocation units in SQL Server 2005 for more details of large rows (and a few other cool new features in 2005). I'll assume that you've got a database called dbccpagetest that you're using.

CREATE TABLE rowoverflowtest (c1 INT, c2 VARCHAR (8000), c3 VARCHAR (8000));

CREATE CLUSTERED INDEX row_cl ON rowoverflowtest (c1);

Now we need to populate the table so we have something to look at.

INSERT INTO rowoverflowtest VALUES (1, REPLICATE ('a', 100), REPLICATE ('b', 100));

INSERT INTO rowoverflowtest VALUES (2, REPLICATE ('a', 100), REPLICATE ('b', 100));

INSERT INTO rowoverflowtest VALUES (3, REPLICATE ('a', 100), REPLICATE ('b', 100));

INSERT INTO rowoverflowtest VALUES (4, REPLICATE ('a', 100), REPLICATE ('b', 100));

INSERT INTO rowoverflowtest VALUES (5, REPLICATE ('a', 100), REPLICATE ('b', 100));

And now the new command - use DBCC IND to find out which page IDs to look at with DBCC PAGE.

DBCC IND ('dbccpagetest', 'rowoverflowtest', 1);

The output is (prettified in Excel):

PageFID	PagePID	IAMFID	IAMPID	ObjectID	IndexID	PartitionNumber	PartitionID	iam_chain_type	PageType	IndexLevel
1	156	NULL	NULL	133575514	1	1	72057594039959552	In-row data	10	NULL
1	155	1	156	133575514	1	1	72057594039959552	In-row data	1	0

(I've stripped off the 4 trailing columns, NextPageFID, NextPagePID, PrevPageFID, PrevPagePID so it all fits in the window. They're all zero.)

The columns mean:

PageFID - the file ID of the page
PagePID - the page number in the file
IAMFID - the file ID of the IAM page that maps this page (this will be NULL for IAM pages themselves as they're not self-referential)
IAMPID - the page number in the file of the IAM page that maps this page
ObjectID - the ID of the object this page is part of
IndexID - the ID of the index this page is part of
PartitionNumber - the partition number (as defined by the partitioning scheme for the index) of the partition this page is part of
PartitionID - the internal ID of the partition this page is part of
iam_chain_type - see IAM chains and allocation units in SQL Server 2005
PageType - the page type. Some common ones are:

1 - data page
2 - index page
3 and 4 - text pages
8 - GAM page
9 - SGAM page
10 - IAM page
11 - PFS page

IndexLevel - what level the page is at in the index (if at all). Remember that index levels go from 0 at the leaf to N at the root page (except in clustered indexes in SQL Server 2000 and 7.0 - where there's a 0 at the leaf level (data pages) and a 0 at the next level up (first level of index pages))
NextPageFID and NextPagePID - the page ID of the next page in the doubly-linked list of pages at this level of the index
PrevPageFID and PrevPagePID - the page ID of the previous page in the doubly-linked list of pages at this level of the index

So you can see we've got a single page clustered index with an IAM page. Note that the page IDs returned may differ on your server. Let's look at the data page, focusing on the record for c1 = 3.

DBCC TRACEON (3604);

DBCC PAGE (dbccpagetest, 1, 155, 3);

Don't forget that we need to turn on T3604 first to get the output back to the console. The dump for the 3rd row is (remembering that slots are number from zero, so slot 2 is row 3):

Slot 2 Offset 0x216 Length 219

Record Type = PRIMARY_RECORD Record Attributes = NULL_BITMAP VARIABLE_COLUMNS

Memory Dump @0x44DCC216

00000000: 30000800 03000000 04000003 00130077 †0..............w

00000010: 00db0061 61616161 61616161 61616161 †...aaaaaaaaaaaaa

00000020: 61616161 61616161 61616161 61616161 †aaaaaaaaaaaaaaaa

00000030: 61616161 61616161 61616161 61616161 †aaaaaaaaaaaaaaaa

00000040: 61616161 61616161 61616161 61616161 †aaaaaaaaaaaaaaaa

00000050: 61616161 61616161 61616161 61616161 †aaaaaaaaaaaaaaaa

00000060: 61616161 61616161 61616161 61616161 †aaaaaaaaaaaaaaaa

00000070: 61616161 61616162 62626262 62626262 †aaaaaaabbbbbbbbb

00000080: 62626262 62626262 62626262 62626262 †bbbbbbbbbbbbbbbb

00000090: 62626262 62626262 62626262 62626262 †bbbbbbbbbbbbbbbb

000000A0: 62626262 62626262 62626262 62626262 †bbbbbbbbbbbbbbbb

000000B0: 62626262 62626262 62626262 62626262 †bbbbbbbbbbbbbbbb

000000C0: 62626262 62626262 62626262 62626262 †bbbbbbbbbbbbbbbb

000000D0: 62626262 62626262 626262†††††††††††††bbbbbbbbbbb

UNIQUIFIER = [NULL]

Slot 2 Column 1 Offset 0x4 Length 4

c1 = 3

Slot 2 Column 2 Offset 0x13 Length 100

c2 = aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa

Slot 2 Column 3 Offset 0x77 Length 100

c3 = bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb

You can see that all the data is stored in-row. Now we update one of the columns so that the row size exceeds 8060 bytes.

UPDATE rowoverflowtest SET c3 = REPLICATE ('c', 8000) WHERE c1 = 3;

And run DBCC IND again to see if anything changed. The output is:

PageFID	PagePID	IAMFID	IAMPID	ObjectID	IndexID	PartitionNumber	PartitionID	iam_chain_type	PageType	IndexLevel
1	156	NULL	NULL	133575514	1	1	72057594039959552	In-row data	10	NULL
1	155	1	156	133575514	1	1	72057594039959552	In-row data	1	0
1	158	NULL	NULL	133575514	1	1	72057594039959552	Row-overflow data	10	NULL
1	157	1	158	133575514	1	1	72057594039959552	Row-overflow data	3	0

(Again, I've stripped off the trailing 4 columns - they're all zero)

Cool - we have another IAM page and a text page, in a row-overflow IAM chain. So something must have been pushed off-row into page (1:157). Let's have another look at the clustered-index data page, focusing on the 3rd row again:

Slot 2 Offset 0x4a7 Length 143

Record Type = PRIMARY_RECORD Record Attributes = NULL_BITMAP VARIABLE_COLUMNS

Memory Dump @0x44DCC4A7

00000000: 30000800 03000000 04000003 00130077 †0..............w

00000010: 008f8061 61616161 61616161 61616161 †...aaaaaaaaaaaaa

00000020: 61616161 61616161 61616161 61616161 †aaaaaaaaaaaaaaaa

00000030: 61616161 61616161 61616161 61616161 †aaaaaaaaaaaaaaaa

00000040: 61616161 61616161 61616161 61616161 †aaaaaaaaaaaaaaaa

00000050: 61616161 61616161 61616161 61616161 †aaaaaaaaaaaaaaaa

00000060: 61616161 61616161 61616161 61616161 †aaaaaaaaaaaaaaaa

00000070: 61616161 61616102 00004401 000000a6 †aaaaaaa...D.....

00000080: 2e000040 1f00009d 00000001 000000††††...@...........

UNIQUIFIER = [NULL]

Slot 2 Column 1 Offset 0x4 Length 4

c1 = 3

Slot 2 Column 2 Offset 0x13 Length 100

c2 = aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa

c3 = [BLOB Inline Root] Slot 2 Column 3 Offset 0x77 Length 24

Level = 0 Unused = 68 UpdateSeq = 1

TimeStamp = 782630912

Link 0

Size = 8000 RowId = (1:157:0)

The 2nd varchar column is no longer stored on the page - instead the value have been replaced with an in-row blob root, pointing to an 8000 byte long blob fragment stored in slot 0 of page (1:157). If we dump it with DBCC PAGE we should see its got our value in:

DBCC PAGE (dbccpagetest, 1, 157, 3);

And the output is (slightly curtailed):

PAGE: (1:157)

BUFFER:

BUF @0x02BE8D84

bpage = 0x03FB6000 bhash = 0x00000000 bpageno = (1:157)

bdbid = 9 breferences = 0 bUse1 = 8105

bstat = 0xc0010b blog = 0x1432159b bnext = 0x00000000

PAGE HEADER:

Page @0x03FB6000

m_pageId = (1:157) m_headerVersion = 1 m_type = 3

m_typeFlagBits = 0x0 m_level = 0 m_flagBits = 0x8000

m_objId (AllocUnitId.idObj) = 117 m_indexId (AllocUnitId.idInd) = 256

Metadata: AllocUnitId = 72057594045595648

Metadata: PartitionId = 72057594039959552 Metadata: IndexId = 1

Metadata: ObjectId = 133575514 m_prevPage = (0:0) m_nextPage = (0:0)

pminlen = 0 m_slotCnt = 1 m_freeCnt = 80

m_freeData = 8110 m_reservedCnt = 0 m_lsn = (21:107:19)

m_xactReserved = 0 m_xdesId = (0:0) m_ghostRecCnt = 0

m_tornBits = 0

Allocation Status

GAM (1:2) = ALLOCATED SGAM (1:3) = ALLOCATED

PFS (1:1) = 0x64 MIXED_EXT ALLOCATED 100_PCT_FULL DIFF (1:6) = CHANGED

ML (1:7) = NOT MIN_LOGGED

Blob row at: Page (1:157) Slot 0 Length: 8014 Type: 3 (DATA)

Blob Id:782630912

44DCC06E: 63636363 63636363 63636363 63636363 cccccccccccccccc

44DCC07E: 63636363 63636363 63636363 63636363 cccccccccccccccc

44DCC08E: 63636363 63636363 63636363 63636363 cccccccccccccccc

44DCDF7E: 63636363 63636363 63636363 63636363 cccccccccccccccc

44DCDF8E: 63636363 63636363 63636363 63636363 cccccccccccccccc

44DCDF9E: 63636363 63636363 63636363 63636363 cccccccccccccccc

DBCC execution completed. If DBCC printed error messages, contact your system administrator.

Yep - that's our updated column.

Now, this is a contrived example to force columns off-row, but you can imagine a case where a column is unexpectedly pushed off-row in some rows of a table and suddenly a query slows down. You look at the fragmentation and the query plan and everything looks good but don't realize that you're taking random IOs to return the columns you've asked for.

That's all - now it's time for me to go and pack my diving gear and head off towards the sun (20 1/2 hours of flights over 31 1/2 hours to get to Bali tomorrow - ugh!). Have fun playing with DBCC over the holidays and I'll be back in January.

CHECKDB (Part 6): Consistency checking options for a VLDB

Paul Randal - MSFT — Fri, 20 Oct 2006 05:57:00 GMT

(Yippee - just finished my certification dives and got my PADI Open Water certification - just in time for our dive trip to Indonesia in December :-)

This is a question that comes up a lot - in fact 3 times this week already - most recently during a guest lecture I did on DBCC for one of Kimberly's popular "Immersion Events". The question is: how can I run consistency checks on a VLDB?

We're talking hundreds of GBs or 1 TB or more. These databases are now common on SQL Server 2000 and 2005, with more migrations happening all the time. Any sensible DBA knows the value of running consistency checks, even when the system is behaving perfectly and the hardware is rock-solid. The two problems that people have with running a full DBCC CHECKDB on their VLDB are:

It takes a long time to run (proportional to the database size and schema complexity).
It uses lots of resources.

So it uses lots of resources for a long time. Even with a decent sized maintenance window, the CHECKDB may well run over into normal operations. There's also the case of a system that's already pegged in more or more resource dimensions (memory, CPU, IO bandwidth). Whatever the case, there are a number of options:

Don't run consistency checks
Run a DBCC CHECKDB using the WITH PHYSICAL_ONLY option
Use SQL Server 2005's partitioning feature and devise a consistency checking plan around that
Figure out your own scheme to divide up the consistency checking work over several days
Use a separate system

Let's look at each in turn.

Don't run consistency checks

Don't be daft. Don't even think about using this option. If you absolutely cannot figure out a way to get consistency checks on your system, send me email and I'll help you. Now let's move on to serious options...

Use WITH PHYSICAL_ONLY

A full CHECKDB does a lot of stuff - see my CHECKDB internals series for more details. You can vastly reduce the run-time and resource-usage of CHECKDB by using the WITH PHYSICAL_ONLY option. With this option, CHECKDB will

Run the equivalent of DBCC CHECKALLOC (i.e. check all the allocation structures)
Read and audit every allocated page in the database

So it skips all the logical checks, inter-page checks, and things like DBCC CHECKCATALOG. The fact that all allocated pages are read means that:

Any pages that cannot be read at all (i.e. 823 errors) will be discovered
If page checksums are enabled in SQL Server 2005, any corruptions caused by storage hardware will be discovered (as the page checksum will have changed).

So there's a trade-off of consistency checking depth against runtime and resource usage.

Use the partitioning feature to your advantage

One of the obvious ways to reduce the time/resources issue is to partition the load. If you're using the partitioning feature in SQL Server 2005 then you're already setup for this. Given that you've hopefully got your partitions stored on seperate filegroups, you can use the DBCC CHECKFILEGROUP command.

Consider this example - you have the database partitioned by date such that the current month is on a read-write filegroup and the past 11 months are on read-only filegroups (data from more than a year ago is on some offline storage medium). The prior months also have multiple backups on various media so are considered much 'safer' than the current month. It makes sense then that you don't need to check these filegroups as often as the current month's filegroup so an example consistency checking scheme would be:

Run a DBCC CHECKFILEGROUP on each read-only filegroup every week or two
Run a DBCC CHECKFILEGROUP on the read-write filegroup every day or two (depending on the stability of the hardware, the criticality of the data, and the frequency and comprehensiveness of your backup strategy).

I know of several companies who've made the decision to move to SQL Server 2005 in part because of this capability to easily divide up the consistency checking.

Figure out your own way to partition the checks

If you're on SQL Server 2000, or you just haven't partitioned your database, then there are ways you can split up the consistency checking workload so that it fits within a maintenance window. Here's one scheme that I've recommended to several customers:

Run a bi-weekly DBCC CHECKALLOC
Figure out your largest tables (by number of pages) and split the total number into 7 buckets, such that there are a roughly equal number of database pages in each bucket.
Take all the remaining tables in the database and divide them equally between the 7 buckets (using number of pages again)
On Sunday:

Run a DBCC CHECKALLOC
Run a DBCC CHECKCATALOG
Run a DBCC CHECKTABLE on each table in the first bucket

On Monday, Tuesday, Wednesday:

Run a DBCC CHECKTABLE on each table in the 2nd, 3rd, 4th buckets, respectively

On Thursday:

Run a DBCC CHECKALLOC
Run a DBCC CHECKTABLE on each table in the 5th bucket

On Friday and Saturday:

Run a DBCC CHECKTABLE on each table in the 6th and 7th buckets, respectively

In pre-RTM builds of SQL Server 2005, DBCC CHECKTABLE could not bind to the critical system tables, just like with T-SQL - but that's fixed so you can cover all system tables in SQL Server 2000 and 2005 using the method above. Here's what I mean:

C:\Documents and Settings\prandal>osql /E
1> select * from sys.sysallocunits
2> go
Msg 208, Level 16, State 1, Server SUNART, Line 1
Invalid object name 'sys.sysallocunits'.
1> dbcc checktable ('sys.sysallocunits')
2> go
DBCC results for 'sysallocunits'.
There are 112 rows in 2 pages for object "sysallocunits".
DBCC execution completed. If DBCC printed error messages, contact your system
administrator.
1>

There's one drawback to this method - a new internal database snapshot is created each time you start a new DBCC command, even for a DBCC CHECKTABLE. If the update workload on the database is significant, then there could be a lot of transaction log to recover each time the database snapshot is created - leading to a long total run-time. In this case, you may need to alter the number of buckets you use to make the total operation fit within your available window.

Use a separate system

This alternative is relatively simple - restore your backup (you are taking regular backups, right?) on another system and run a full CHECKDB on the restored database. This offloads the consistency checking burden from the production system and also allows you to check that your backups are valid (which you're already checking though, right?). There are some drawbacks to this however:

You need to have sufficient disk space on the spare system to be able to restore the backup onto. If the production database is several TB, you need the same several TB on the spare box. This equates to a non-trivial amount of money - initial capital investment plus ongoing storage mgmt costs. (I'm working on this though - I have a patent on consistency checking a database in a backup without restoring it - unclear at this time whether it will make it into Katmai.)
If the consistency checks find an error, you don't know for sure that the database is corrupt on the production system. It could be a problem with the spare box that's caused the corruption. The only way to know for sure is to run a consistency check on the production system. This is a small price to pay though, because most of the time the consistency checks on the spare system will be ok, so you know the production database was clean at the time the backup was taken.

Summary

You've got a bunch of choices to allow you to run consistency checks, so there's really no excuse for not knowing (within a reasonable timeframe) that something's gone wrong with your database. If you need further help working out what to do, or just want a critical eye cast over the plan you've come up with, send me an email at prandal@microsoft.com

(One of the other questions that also keeps coming up is: when are you going to write the whitepaper you promised? Q1CY07 - honestly!)

CHECKDB (Part 5): What does CHECKDB really do? (part 4 of 4)

Paul Randal - MSFT — Wed, 20 Sep 2006 12:03:00 GMT

(Another airport, another blog post - I really must make an effort to come up with more original banter - I'm sure I've used that line before. TechEd Guangzhou has finished and I'm on the way to Beijing for TechEd #3. Checking in and getting through security were challenging to say the least this morning - there seemed to be problems at every turn, exacerbated by my China Southern airlines chaperone who spoke limited English (or what I should really say is that the problem was I don't speak Mandarin). Finally at the security checkpoint they asked for my ticket as well as my boarding pass - try explaining "e-ticket" to someone who doesn't speak the same language...)

So part 4 of the high-level explanation of what DBCC CHECKDB does will focus on the three completely new sets of checks that are in CHECKDB in SQL Server 2005. Well, the metadata checks (i.e. DBCC CHECKCATALOG) used to exist but so many people used to run both CHECKDB and DBCC CHECKCATALOG that I thought I'd make it easier in SQL Server 2005 by running DBCC CHECKCATALOG as part of CHECKDB. Maybe in Katmai we'll add in DBCC CHECKCONSTRAINTS - haven't decided yet.

Primitive checks of critical system tables
(Part 1...)

Allocation checks
(Part2...)

Logical checks of critical system tables
Logical checks of all tables
(Part 3...)

Service Broker checks
Metadata cross-checks
Indexed view and XML index checks
Its important that these checks are done after the per-table logical checks. This is because any repairs done as part of the per-table checks may affect the outcome of these checks quite substantially.

Let's look at an example of this. Imagine the case where an indexed view is based on a join between two tables, foo and bar. Table foo has a damaged page and is damaged in a way that reading the page as part of a query would not recognize - because full page audits are not done as part of regular page reads. The damage is such that the page has to be deleted as part of repair - thereby changing the results of the view. If the indexed view was checked before tables foo and bar, then the repair that was done would not get reflected in the indexed view and so the indexed view would essentially be corrupt. The same logic holds for checking XML indexes and Service Broker tables.

(Ok - in the air now with some serious turbulence - luckily I popped a Dramamine before we took off. And, more jellyfish in the in-flight meal - very cool :-) The 1st class cabin has what looks like a fully stocked bar - a little too early at 10am to be starting, although it could make for some interesting blog posts...)

Service Broker checks
The Service Broker dev team wrote a comprehensive set of checks of the data stored in their internal on-disk structures. The checks validate the relationships between conversations, endpoints, messages and queues. For example:

a conversation must have two endpoints
a service must be related to a valid contract
a service must be related to a valid queue
a message must have a valid message type

What's even cooler is that they implemented a set of logical repairs, so if one of their internal tables was damaged, and repaired by the earlier logical checks, then the Service Broker repair code can clean up any Service Broker entities and entity relationships that were damaged too.

Metadata cross-checks
The Metadata team also wrote a great set of checks for the relational metadata stored in the system tables. These checks that are run here are the same code that's run for DBCC CHECKCATALOG. The actual checks themselves in SQL Server 2005 are far more involved than in SQL Server 2000, and they're done way more efficiently too. However, they're not comprehensive by any means - we'll add some more checks during Katmai.

The checks only cover the relational metadata - i.e. the relationships between system table storing relational metadata. There are no such checks for the tables storing the storage engine metadata (also known as the critical system tables that were described in Part 1 of this 4 part sub-series. At present, these relationships between these tables are checked implicitly by the metadata checks I described in the last post. The checks are a series of validations between the various system tables, such as checking each rowset referenced in the sysrowsetcolumns system table exists in the sysrowsets system table.

There are no repairs for metadata corruptions. Metadata corruptions are extremely difficult to deal with because changing/deleting metadata has an effect on the entire table the corrupt metadata describes and could potentially be as bad as deleting the table - a table without metadata is just a collection of pages with indecipherable rows (well, not quite true - its possible to decipher any record without metadata but it requires human intervention and is incredibly hard).

Its possible we may put in some limited metadata repairs in a future release, but the frequency of their occurence in the field is so low that I decided that the engineering investment was not justified for SQL Server 2005 and so didn't push it. So - if you get any metadata corruption, you need to restore from your backups which, of course, after reading through my blog, you've been scared into making sure you have, right?...

Indexed view and XML index checks
These are very cool and many thanks to Conor Cunningham (dev lead of one of the two Query Optimizer teams) for helping work out how to do this.The indexed view contains the persisted results of a query, and we have the actual query it persists stored in metadata - so the easiest way to check whether the indexed view is accurate is to recalculate the view results into a temp table in TEMPDB and then compare the calculated values with the values persisted in the indexed view. The view is regenerated into a temporary table and then two concurrent left-anti-semi-joins are run, that basically return all the rows in the indexed view that are not in the recalculated view results, and vice-versa. This gives us all the rows that are extraneous in the indexed view and all the rows that are missing from it.

Indexed-view problems can also be repaired. The repair for extra rows is to delete them one by one (using internal query syntax that only works from DBCC), and the repair for missing rows is to rebuild the indexed view. This is done by simply disabling the indexed view and then bringing it back online (which rebuilds it).

There are two drawbacks to the indexed view checks if the views are large:

it can take up a lot of space in tempdb
it can take a lot of time to run the regeneration of the indexed views and to run the left-anti-semi-joins

So if you upgraded your database from SQL Server 2000 and you regularly run full CHECKDBs (i.e. without using WITH PHYSICAL_ONLY), then you may see a run-time increase for CHECKDB on SQL Server 2005 - this is documented in BOL and the README.

XML index checks work in a similar way. The XML blobs in the table are re-shredded and checked against the shredding that's persisted in the primary XML index. If anything is wrong, the primary XML index is recreated.

And that's it. Now you have a complete picture of what's going on with CHECKDB when it runs. In the next few posts I want to give some insight into how CHECKDB manages to do all these checks while only making a single pass through the database and shed some light on how repair works. Until then - time to chill out before doing more sessions in Beijing tomorrow... qing lai yiping pijiu!!!

CHECKDB (Part 4): What does CHECKDB really do? (part 3 of 4)

Paul Randal - MSFT — Tue, 19 Sep 2006 00:43:00 GMT

(OK - I was wrong about the posting frequency - events overtook me and I got bogged down preparing for these TechEds. China is an amazing place and I wish I had more time to soak up some of its culture while I'm here but with the TechEds being so close together there's little time for anything except sessions, email, taxis, flying, sleeping, jellyfish... I did manage to go to the 400 year old Yuyuan Gardens in Shanghai, which are well worth a visit. Now I'm sitting here next to the Pearl River in Guangzhou waiting to do DAT305: Choosing a High Availability Solution - so time to squeeze in another post.)

In the previous posts of this series, I covered the system table checks that have to be done before anything else can be checked by CHECKDB, and the allocation checks that have to be done before any of the logical checks. Now I want to cover the meat of the functionality in CHECKDB - the logical checks. Note - this is a description of what happens for SQL Server 2005 - its pretty similar in SQL Server 2000.

Primitive checks of critical system tables
(Part 1...)

Allocation checks

(Part 2...)

Logical checks of critical system tables
Logical checks of all tables
Although this section has two titles, the actual logical checks performed in each stage are the same - i.e. everything I'm going to describe below. If any errors are found in the critical system tables (remember these from Part 1?) and:

repair is not specified; or,
repair is specified, but not all the errors can be repaired

then the CHECKDB finishes. An example of an unrepairable system table error is something that would require deleting data from one of the system tables - e.g. a corrupt key value in a clustered index data page of sysallocunits (remember that this is the actual table I'm talking about, not the sys.allocation_units catalog view you may have seen or used).

So if the system tables are clean, all the tables in the database are checked. This includes indexed views and primary XML indexes (which are both stored as clustered indexes - and as far as the storage engine is concerned are objects in their own right - its the relational layer that knows that they're not really separate objects).

The following checks are performed:

Validate each table's storage engine metadata
Read and check all data, index and text pages, depending on the page type
Check all inter-page relationships
Check the page header counts
Perform any necessary repairs

Let's look at each of these stages in more detail.

Validate each table's storage engine metadata

Why do we need to do this? Well, the storage engine metadata is what tells CHECKDB how to crack open records on pages in particular rowsets - so if there's something wrong with it then CHECKDB will generate a bunch of misleading errors or, worse yet, miss some errors. Here's roughly what happens while the metadata is parsed:

Loop through all the indexes, and then all the rowsets of each index, building a list of known allocation units for the index (including all the DATA, LOB, and SLOB allocation units). This allows reverse lookups - when we read a page, because the information stamped on the page is the allocation unit, we need a very fast way to convert between an allocation unit and an index/object ID.
Make sure we skip any indexes that are in the middle of being built/rebuilt online, because they will not have a complete set of data.
Build a list of computed columns and generate the necessary code to enable the column values to be recomputed.
Build a list of columns that are used in non-clustered indexes
Make sure the various USED, DATA, RSVD counts are not negative (see the BOL for DBCC CHECKDB for an explanation of how this can happen - yes, I'm deliberately not explaining here so that you read through the new BOL entry :-)
Figure out what each kind of column is (e.g. a regular column, a generated uniquifier, a dropped column)
Build a series of mappings to allow conversion between column IDs at different levels of storage abstraction
Check that the relational and storage-engine nullability flags for a column agree
Make sure the columns counters in metadata match what we've just seen

Read and check all data, index and text pages

No matter what type a page is, the page is audited and then all the records on it are audited.

Page audit checks first of all for IO errors when the page is read (e.g. page checksums or torn-page detection). If there are some, then the page is not processed any further. This is what can lead to errors like 8976 being reported (e.g. Table error: Object ID 132765433, index ID 1, partition ID 72057594038321152, alloc unit ID 72057594042318848 (type DATA). Page (1:3874) was not seen in the scan although its parent (1:3999) and previous (1:3873) refer to it. Check any previous errors.). Then it checks for page header correctness and that the page has an appropriate type for the allocation unit its in (e.g. a DATA page should not be found in an allocation unit for a non-clustered index)

Record audits include checking the various fields in the record header and that the various offsets in the record make sense (e.g. the offset to the variable length columns section of the record should not point off the end of the record. See the post on cracking records using DBCC PAGE for more info on the record format.

The more complex checks that are done per-page depend on what type the page is. As an example, here's what is done for a DATA page in the leaf level of a clustered index (excluding the inter-page relationships - I'll list those in the next section):

Records in a page must be strictly ordered by the defined keys of the index (although the records themselves aren't necessarily stored in sorted order in the data portion of the page, accessing the records through the slot array must yield them in the correct order)
No two records can have duplicate key values (remember that non-unique indexes have a hidden, automatically-generated uniquifier column added to the key - to make or extend the composite key - so that record uniqueness is guaranteed)
If the index is partitioned, each record is run through the partitioning function to ensure its stored in the correct partition
All the complex columns in each record are checked:
- Complex columns are those storing legacy text or LOB values (text, ntext, image, XML, nvarchar(max), varchar(max), varbinary(max)) or in-row pointers to variable length columns that have been pushed off-row in rows that are longer than 8060 bytes
- The column is checked to make sure its storing the right kind of data - either the value itself or a text pointer or some kind of in-row root containing pointers to portions of an off-row value.
- The linkages between what is stored in-row and the off-row values stored in other pages are eventually checked too
Check computed columns
- If the column is persisted (either because its defined as a persisted computed column or because its used as a non-clustered index key), its value is recomputed and checked against the persisted value
- This is also important when we come to do the non-clustered index cross-checks - as any discrepancy in the stored column values will cause mismatches.
Data purity checks
- The column value is checked to ensure its within the bounds for its data-type (e.g. the minutes-past-midnight portion of the internal representation of a datetime value cannot be greater than 1440 - 24 hours x 60 minutes)
Non-clustered index cross-checks

This is probably my favorite part of CHECKDB and is one of the most complicated bits of code.
What CHECKDB is trying to do is make sure that each record in a heap or clustered index has exactly one matching record in each non-clustered index, and vice-versa. The brute-force (n-squared complexity) way to do this is to do a physical lookup of all the matching rows - but that's incredibly time consuming so instead we have a fast algorithm to detect problems.
Imagine a table defined by the following DDL:

      CREATE TABLE t (c1 INT, c2 char(10), c3 varchar(max))

      CREATE INDEX index1 ON t (c1)

      CREATE INDEX index2 ON t (c2) INCLUDE (c3)

Each row in the heap has to have two matching non-clustered index rows, on in each of index1 and index2. But how can we tell without doing the direct lookup or having to store tremendous amounts of state? We use a hash-bitmap algorithm.
Imagine a large bitmap - say half a million bits. Initially all the bits are zero.
For each record in the heap that we come across, we generate the matching non-clustered index records and hash them to a value. This is very quick because we have all the column values we need. We then take the hash value, map it directly to a bit in the bitmap and flip the bit. So in our example above, each record in the heap will produce two hash values (one for the record in each non-clustered index) and so will cause two bits to be flipped.
For each record in each non-clustered index, just hash the record and flip the corresponding bit.
The idea is that the bit-flips should cancel each other out and the bitmap should be left with all zeroes at the end of the checks.
Taking the view that corruptions are magnitudes rarer than clean databases, we went a step further and allowed the checks for multiple tables to use the same bitmap. If you think about it, this won't cause any problems, even if two sets of records map to the same bit in the bitmap - as long as the number of bit-flips is a power of 2 (i.e. each record really does have its correct matching record) then there should be no problem.
Here's the catch - what happens when there's a bit left on in the bitmap? Well, this is where the trade-off comes into play. If there's a bit left on, we can't tell which records in which table or index mapped to it so, we have to re-scan the tables and indexs that used the bitmap to see which records map to the bit. For every one we find, we actually do the physical lookup of the matching record and then do a comparison of all the columns, including any LOB columns used as INCLUDE'd columns in non-clustered indexes. This process is called the deep-dive and can add a significant amount to the run-time of CHECKDB if it occurs.

Check all inter-page relationships

Inter-page relationships are relevant for:

pages in heaps that have forwarding or forwarded records
pages in indexes
pages that have records with LOB columns that have their data stored off-row (these can be heap pages, clustered index data pages or non-clustered index leaf pages in SQL Server 2005)
text pages that have records with child nodes on other pages

Continuing the example above, here are the checks that are done for index pages:

All pages in an index level should be pointed to by a page in the next level higher in the b-tree, and also by the left- and right-hand neighboring pages in the same level of the b-tree. Exceptions are made for the left-most and right-most pages in a level (where the m_prevPage and m_nextPage fields are (0:0)) and for the root page of the b-tree, where the parent page link comes from the storage-engine metadata. The 8976 error message that I referenced above is generated from this set of checks.
Key ranges in neighboring pages in a level should not overlap
Key ranges of pages should be correctly defined by the parent pages in the next level up in the b-tree (the parent pages contain the minimum key value that can exist on a child page)

I'll go into details of how the inter-page checks are done in a future post. For now, its enough to say that its not an n-squared complexity algorithm.

Check the page header counts

The page header contains a bunch of counters - the ones we need to check are:

slot count
ghost record count
free space count

The first two are obvious - count the rows as they're processed and make sure page header counts are valid. The free space count is only checked for text pages and data pages in a heap (the only pages for which free space is tracked in a PFS page).

Perform any necessary repairs

I want to leave discussing repairs for another post as there's a ton of info there.

So, that's the majority of the work in running a DBCC CHECKDB and now its time for my session on choosing an HA solution in SQL Server 2005. Next time I'll post the next part of the Fragmentation series.

(So I mentioned jellyfish above... I was having a meal last night with Scott Schnoll from the Exchange team and we had what we thought were noodles on the side of one dish. Scott asked what kind of noodles they were and then we found out we'd been eating cold, sliced jellyfish - it was really very tasty!)

Service Broker checks
Metadata cross-checks
Indexed view and XML index checks
(Part 4...)

Don't try this at home kids... (data recovery using DBCC PAGE)

Paul Randal - MSFT — Wed, 09 Aug 2006 22:41:00 GMT

Time to cough up a real-life data recovery scenario to temporarily stop those people who badger me relentlessly for scenarios...

We have a situation currently with a customer who managed to delete all their data during a SAN reconfig and then found out that their tape backups hadn't been working properly (the same sad story I've heard a thousand times). They did have a primary filegroup backup plus a backup of a secondary filegroup from a different point. This is on SQL Server 2005.

The backups are restored, but the storage engine metadata for the table with the critical data in is messed up - sys.allocunits somehow has the wrong allocation unit IDs, first page, and root pages. So, what the hell can you do?

Easy (and fun if you're twisted like me).

We need to find the following info for the data allocation unit:

the page ID of the head of the IAM chain for the allocation unit
the root page of the clustered index
the first page of the clustered index

And there's a twist, the table has a column of type image so we also need to find the allocation unit ID and page ID of the head of the IAM chain for the LOB allocation unit.

The easiest way to do this is to scan through the data files, doing DBCC PAGEs of every page and storing the results in a table for later analysis. How? Ah, there's something I haven't told you about DBCC PAGE - it has a WITH TABLERESULTS option.

So, setup a simple script to INSERT/EXEC a DBCC PAGE of every page with option 0 (for speed, otherwise it has to take the time to go through and crack and format every record on every page) and save the results to a table.

To find the root page of the clustered index, query the table for the page with m_type = 2 (an index page) and the highest m_level. The level number increases from the leaf of the b-tree to the root page, so the page with the highest level is the root.

To find the first page, crack open the first record on the root page, and extract the page ID of the child page that the index record points to. This is the left-hand edge of the next level lower in the b-tree. Continue doing this until you find the page with m_level = 0. (Note that on SQL Server 2000, you'll need to look for m_level = 0 AND m_type = 1 - this is because pages at the leaf level and the level above in a SQL Server 2000 clustered index all have m_level = 0).

To find the head of the IAM chain is a little trickier. You need to find any page with m_type = 10 and following the m_prevPage links until its NULL. Then do an option 3 dump of the page and ensure that the sequenceNumber in the IAM page header is 0 (remember that an IAM chain is ordered by the sequence numbers in the IAM pages).

Cool - we've done the clustered index.

Now for the LOB allocation unit. Crack any row at the leaf level of the clustered index. You'll see that one of the columns is a text pointer, containing a page ID and slot number of the text fragment in the LOB allocation unit. Do a DBCC PAGE of that page and now you know the allocation unit ID of the LOB allocation unit and can use the methodology above to find the head of its IAM chain.

Now all you have to do is alter the hidden and unbindable sys.allocunits table to contain the correct data... Who's buying the beer? :-)

Poking about with DBCC PAGE (Part 1 of ?)

Paul Randal - MSFT — Wed, 09 Aug 2006 04:52:00 GMT

(It's been a while since I last posted - summer fun's been in the way obviously and I've been busy picking up some new and exciting (for me) SQL skills, partially to help with four upcoming TechEds I'll be doing in China and Hong Kong. The posting frequency should increase towards the end of the week but in the meantime, this one's a special request.)

I've been getting lots of requests for more info about how to use DBCC PAGE since my Paul-tells-all post a while back and I had been planning to wait until I'd posted more about the internals of DBCC CHECKDB and moved on to various examples of corruptions before starting to use DBCC PAGE again in the blog. However, a friend made me an offer I couldn't refuse and so I've caved in and I'm starting the series now :-)

I'm going to start with cracking a record from scratch. Yes, I know that option 3 of DBCC PAGE will do it for you - but how does it do it? Sometimes you may not have table metadata and so DBCC PAGE won't be able to use option 3 - in that case you'll need to crack it yourself. Also, there are plenty of curious people who like to be immersed in the bits and bytes and besides, cracking records manually is fun (if you're twisted as well as curious like I am :-)

I've created an example database, dbccpagetest, on SQL Server 2005 which I've zipped up and attached to this post. This way you can all use the same data and pages as me without having to go through the tedium of working out where pages are using the PFS pages (as I explained in the initial DBCC PAGE post).

You won't be able to attach it to SQL Server 2000 but you can always download SQL Server 2005 Express Edition and play with it on that. Unzip the files and attached them using the CREATE DATABASE ... FOR ATTACH syntax:

CREATE DATABASE dbccpagetest

ON (FILENAME='C:\Program Files\Microsoft SQL Server\MSSQL.1\MSSQL\Data\dbccpagetest.mdf')

FOR ATTACH;

Note that if the filenames haven't changed since they were created, you only need to specify the primary file in the CREATE DATABASE statement. Very cool.

The database has a single table with a couple of rows, created using the following T-SQL:

CREATE TABLE example (

destination VARCHAR(100),

activity VARCHAR(100),

duration INT);

INSERT INTO example VALUES ('Banff', 'sightseeing', 5);

INSERT INTO example VALUES ('Chicago', 'sailing', 4);

The data page is page (1:152). Dumping that page using option 3 will give output including:

Slot 0 Offset 0x60 Length 33

Record Type = PRIMARY_RECORD Record Attributes = NULL_BITMAP VARIABLE_COLUMNS

Memory Dump @0x44DFC060

00000000: 30000800 05000000 0300f802 00160021 †0..............!

00000010: 0042616e 66667369 67687473 6565696e †.Banffsightseein

00000020: 67†††††††††††††††††††††††††††††††††††g

Slot 0 Column 0 Offset 0x11 Length 5

destination = Banff

Slot 0 Column 1 Offset 0x16 Length 11

activity = sightseeing

Slot 0 Column 2 Offset 0x4 Length 4

duration = 5

Previously (that word always makes me think of Twin Peaks - 'Previously in Twin Peaks...', but I digress...) I'd posted about the structure of a record - here it is again:

record header
- 4 bytes long
- two bytes of record metadata (record type)
- two bytes pointing forward in the record to the NULL bitmap
fixed length portion of the record, containing the columns storing data types that have fixed lengths (e.g. bigint, char(10), datetime)
NULL bitmap
- two bytes for count of columns in the record
- variable number of bytes to store one bit per column in the record, regardless of whether the column is nullable or not
- this allows an optimization when reading columns that are NULL (see here for more info)
variable-length column offset array
- two bytes for the count of variable-length columns
- two bytes per variable length column, giving the offset to the start of the column valu
versioning tag
- this is in SQL Server 2005 only
- this is a 14-byte structure that contains a timestamp plus a pointer into the version store in tempdb

So, let's go through the record from the dump fragment above and figure out what everything means. I'm not going to give all the excruciating detail on every possible value of every byte, but will give you enough to be able to crack records yourself (if you're twisted like me...)

Byte 0 is the TagA byte of the record metadata. Its 0x30, which corresponds to 0x10 (bit 4) and 0x20 (bit 5). Bit 4 means the record has a NULL bitmap and bit 5 means the record has variable length columns. If 0x40 (bit 6) was also set, that would indicate that the record has a versioning tag. If 0x80 (bit 7) was also set, that would indicate that byte 1 has a value in it.

Bits 1-3 of byte 0 give the record type. The possible values are:

0 = primary record
- a data record in a heap that hasn't been forwarded or a data record at the leaf level of a clustered index.
1 = forwarded record
- a data record in a heap that's been updated and was too large to fit in-place on its original page and so has been moved to another page. A forwarding record is left in its place and points to the new location of the record. This is done to avoid having to update any non-clustered index records that point back directly to the original physical location of the record.
2 = forwarding record
- sometimes also called a forwarding stub.
3 = index record
- an index record in the tree of a clustered index, or in any level of a non-clustered index.
4 = blob fragment
5 = ghost index record
6 = ghost data record
- these two are self-explanatory
7 = ghost version record
- a special 15-byte record containing a single byte record header plus a 14-byte versioning tag that is used in some circumstances (like ghosting a versioned blob record)

In our example, none of these bits are set which means the record is a primary record. If the record was an index record, byte 0 would have the value 0x36. Remember that the record type starts on bit 1, not bit 0, and so the record type value from the enumeration above needs to be shifted left a bit (multiplied by two) to get its value in the byte.

Byte 1 is the TagB byte of the record metadata. It can either be 0x00 or 0x01. If it is 0x01, that means the record type is ghost forwarded record. In this case its 0x00, which is what we expect given the TagA byte value.

Bytes 2 and 3 are the offset of the NULL bitmap in the record. This is 0x0008 (DBCC PAGE presents multi-byte values in hex dumps as least-significant byte first). This means that there's a 4-byte fixed length portion of the record starting at byte 4. We expect this because we know the table schema.

Bytes 4 to 7 are the fixed length portion. Again, because we know the table schema, we know to interpret these bytes as a 4-byte integer. Without that knowledge, you'd have to guess. The value therefore is 0x00000005, which is what we'd expect to see as the value of the duration column.

Bytes 8 and 9 are the count of columns in the record. This is 0x0003 which is correct. Given that there are only 3 columns, the NULL bitmap of one bit per column will fit in a single byte.

Byte 10 is the NULL bitmap. Hey - it's value is 0xF8 - what's up with that? Convert it to binary and what do you get? 11111000 This makes perfect sense - bits 0-2 represent columns 1-3 and they're all 0, meaning the columns aren't NULL. Bits 3-7 represent non-existent columns and they're set to 1 for clarity. So the 0xF8 value makes sense - phew.

Bytes 11 and 12 are the count of variable length columns in the record. That value is 0x0002, which we again know to be correct. This means there will be two two-byte entries in the variable length column offset array. These will be bytes 13-14 and 15-16, having values of 0x0016 and 0x0021 respectively.

Now, the explanation that the variable length column offset array stores the offsets to the start of the column value is over-simplified. The entries actually point to the start of the following column value - this is done so that we know how long each column is (they are variable length after all). Notice that the offset of the first variable length column value isn't stored - it doesn't need to be because by definition it must begin right after the last offset in the variable length column offset array.

So, the final offset is bytes 15 and 16, which means the offset of the start of the first variable length column must be byte 17 (or 0x11 in hex), which agrees with the DBCC PAGE dump. The offset of the second variable length column is 0x0016, so the first value is from byte 17 to byte 21 inclusive. This value is 0x42616E6666. We know from the table metadata that this is the first varchar column, destination. Checking our handy ASCII conversion table we find that this translates to 'Banff'.

Using similar logic, the second value is from byte 22 to byte 32 inclusive and has the value 'sightseeing'. Both of these match the data we're expecting.

And that's it. We've just cracked a record from scratch. Now you try it with the second row in the table. In future posts I'll start making things more complicated (maybe trace into the version store or add in some off-row LOB values).

Hopefully this will satisfy all of those who've been waiting for more info on using DBCC PAGE and now I'm looking forward to getting my side of the bargain from my friend...

Can't I ever get a guarantee?

Paul Randal - MSFT — Sun, 23 Jul 2006 21:19:00 GMT

I've been sitting and thinking a lot about guarantees this morning while drinking coffee in the debilitating heat here (85 degrees at 11am) and things get more complicated the more I think about it, so time to stop and post some thoughts.

Over the last few days, the question of guarantees has come up several times in various forums, essentially - what guarantee do I get from a clean CHECKDB run?

The simple answer is 'not much', but let me go into more detail.

Here's an analogy to help illustrate my point. Consider Paul, who works for the Seattle Police Department in traffic control. Paul's in a control room somewhere in the city with a large bank of monitors connected to various highway cameras. Every 1/2 hour Paul cycles through the various cameras feeds looking for problems. At the end of the 1/2 hour cycle, Paul knows that there are no accidents in the city.

Ah - but hold on. Does Paul really know that? No. All Paul knows is that at the point he looked at a particular camera, there was not an accident at that spot in the highway network. The very instant he switches to another camera feed, an accident could happen at spot covered by the previous camera.

Hopefully you're starting to see where I'm going with this analogy. A database page is known good at the point that its read from disk. After that point, anything could happen to it, the very next millisecond after the page read has completed.

So, when a CHECKDB completes cleanly, you know that the set of database pages that were read were clean at the point they were read. You can't even say that the entire database is clean at any instant of time, because its not possible to read an entire database in an instant of time.

Now, this may come as a shock to some people reading this but really it shouldn't - its common sense. Basically, there's no guarantee of any future lack of corruptions. You can say that with good, well-behaving and up-to-date hardware the likelihood of future corruptions is small, but you can't guarantee it.

An extrapolation of this point applies to database backups. A common sequence of events is to run a CHECKDB, take a backup, restore the backup on another system and then run a CHECKDB on the resulting database. Yes, this gives you a good sense that the database was clean before it was backed up, the backup worked, and what's contained in the backup is a clean database - but that's where it ends. At the instant that the final CHECKDB completes, you still don't have any guarantees. Something could happen to the backup file - you could have a double failure of the original database and the backup file. Unlikely, sure, but perfectly possible and I've seen it happen many times.

So now that I've scared the hell out of some of you reading this, I'd better touch on the questions that you're no-doubt asking yourselves:

What's the point of running CHECKDB then?
What can I do to get closer to a guarantee?

#1 is the easier to answer. Although you don't get any future guarantees from a clean CHECKDB run, you do at least know that everything's good at present and that all database pages could be cleanly read from disk. That's a pretty strong reassurance that your hardware is holding up.

How often to run a CHECKDB (or combination of other check commands) is way more complicated, and is a subject for a seperate blog post (yes, all those people who keep asking me to post about this - I'll get to it eventually :-)

#2 has no easy answer and I could easily fill a book writing about high availability and disaster recovery. However, I'm not going to do that yet so here are some simple things off the top of my head for you to think about and investigate and I'll follow up with a bunch of blog postings over the next few months:

This is the most important one - if you really have mission-critical data, employ DBAs who have a clue what they're doing - I've seen some scarily naive people running things for large companies
Convince those that control your budget of the consequences of not spending a little money to protect the data
Ensure that all your drivers and firmware are up-to-date
Run stress tests on your hardware to ensure that it can cope with the load that you may drive SQL Server to place on it
Turn on page checksums in SQL Server 2005
Run regular DBCCs to forcibly verify the integrity of all the database pages
Use mirrored backups
Use decent hardware to store mission critical data
Investigate the various Always-On technologies in SQL Server
Practice your disaster recovery strategy

Yes, there are plenty more things I could add to the list but I didn't mean it to be exhaustive.

So, even though its not possible to get a guarantee, I should be able to get some degree of assurance that things are ok and will continue to be that way in the face of any eventuality - the trick is to think things through carefully and make appropriate choices.

More on this early next week hopefully...

CHECKDB (Part 3): What does CHECKDB really do? (2 of 4)

Paul Randal - MSFT — Wed, 19 Jul 2006 01:16:00 GMT

In the previous post of this series, I covered the system table checks that have to be done before anything else can be checked by CHECKDB. Now that I've described pages, extents, IAM chains/allocation units and the major allocation bitmaps, in this post I'll cover the allocation checks.

1. Primitive checks of critical system tables
(Part 1...)

2. Allocation checks

This stage is in SQL Server 2000 and 2005. Allocation checks are checks of the various structures (IAM pages, IAM chains, allocation units, GAM/SGAM pages, PFS pages) that describe the allocations (pages, extents) that have been done within a database.

I'll describe how we go about collecting information from the various pages and then describe what some of the actual checks are.

The allocation checks are very fast (orders of magnitude faster than the logical checks, so fast in fact that they happen in the blink of an eye! well, perhaps I'm getting carried away a little as usual but you get the idea) because the number of database pages that have to be read is very small (so small in fact that... ok, I'll shut-up). The algorithm for gathering allocation data is as follows:

For each file in each online filegroup in the database (except transaction log files):

read all PFS pages

This gives us a bitmap showing all IAM pages, plus another one showing all mixed pages

read the GAM pages

This gives us bitmaps of all allocated extents

read the SGAM pages

This gives us bitmaps of all mixed extents with at least one unallocated page

read the DIFF_MAP pages (a 'differential bitmap' page shows which extents in the GAM interval have been modified since the last full or differential backup - a differential backup only needs to backup those extents marked modified in the various DIFF_MAP pages)

This is just to make sure the pages can be read

read the ML_MAP pages (a 'minimally-logged bitmap' page shows which extents in the GAM interval have been modified in bulk-logged recovery mode since the last log backup - a log backup must also backup all such extents to ensure that all changes to the database have been backed up. This can make the log backup quite large (although the log itself stays much smaller) - but that's a topic for another blog post.

This is just to make sure the pages can be read

read all IAM pages

This gives us:

a list of all the mixed pages in the file, and by derivation, a list of all mixed extents in the file (remember that the first IAM page in an IAM chain/allocation unit contains an array to hold up to 8 mixed pages for the object/index/partition it represents
a list of all the valid IAM pages in the file
a list of all the allocated dedicated extents in the file
linkage information for IAM chains

After all the per-file stuff, read the Storage Engine metadata

This gives us:

information about the root of each IAM chain. Each row in the sys.allocation_units hidden metadata table contains the page Id of the first IAM page in the IAM chain for that allocation unit.
information about IAM chains currently waiting to be deferred-dropped. (This is the process by which an IAM chain with > 128 extents that is dropped - by dropping/rebuilding an index or dropping/truncating a table - does not have its actual pages and extents deallocated until after the transaction has committed. The IAM chain is unhooked from sys.allocation_units though and hooked into an internal queue - if we didn't scan that queue too as part of the allocation checks, we'd see all kinds of inconsistencies with the various allocation bitmaps)

So, now we've got a whole bunch of allocation data that we're juggling and we need to make sense of it all to see if all the allocation structures are correct. Here's a non-exhaustive list of checks that we do with all this data:

Check that each extent is either allocated to:

the GAM page for the GAM interval
or, the SGAM page for the GAM interval, as a non-full mixed extent
or, exactly one IAM page that covers the GAM interval
or, to no bitmap page, but all pages in the extent must be allocated to IAM pages as mixed pages
This could result in an 8903 (GAM and SGAM), 8904 (multiple IAMs), or 8905 (no page) errors depending on the combination of bitmaps that have the extent allocated

Check that all pages marked as being IAM pages in PFS pages really are IAM pages when they're read
Check that all pages marked as being mixed pages in PFS pages appear somewhere in a mixed page array on an IAM page
Check that each mixed page is only allocated in a single IAM page
Check that the IAM pages in an IAM chain have monatonically increasing sequence numbers
Check that the first IAM page in an IAM chain has a reference from a row in sys.allocation_units
Check that no two IAM pages within the same IAM chain map the same GAM interval
Check that all IAM pages within an IAM chain belong to the same object/index/partition
Check that the linkages within an IAM chain are correct (no missing pages for instance)
Check that all IAM/GAM/SGAM pages that map the final GAM interval in a file do not have extents marked allocated that are beyond the physical end of the file

Any errors found here will require REPAIR_ALLOW_DATA_LOSS to repair and some of the repairs are very complicated (e.g. multiply-allocated extents) - topic for a future blog post.

So, the allocation checks lay the next foundation level over the system table primitive checks and we're ready to move on to the logical checks.

3. Logical checks of critical system tables
4. Logical checks of all tables
(Part 3...)

5. Service Broker checks
6. Metadata cross-checks
7. Indexed view and XML index checks
(Part 4...)

CHECKDB (Part 2): What does CHECKDB really do? (1 of 4)

Paul Randal - MSFT — Mon, 19 Jun 2006 04:09:00 GMT

Hmmm - I sat for 5 minutes thinking of something amusing to say to start this one off and nothing came to mind, so I'm afraid this will be a humor-free post. Maybe I'm jet-lagged from being on the East coast all last week.

As with all things related to DBCC, this topic has its share of misinformation. In this post I'll set the record straight by running through all the stages of CHECKDB in SQL Server 2000 and 2005. I'll need to split this up into seperate posts otherwise I'll be writing a book. I also introduce a whole raft of new terms which will also be subjects for future posts (my list is already getting pretty long!)

So the very first thing it does is work out how to get the transactionally consistent view it requires (see CHECKDB Part 1) and then, if needed, either record the relevant LSN and switch to full-logging (for SQL Server 2000) or create a database snapshot (for SQL Server 2005).

Then it runs through the checks in the order shown below:

1. Primitive checks of critical system tables

This stage is in SQL Server 2000 and 2005. First of all, what are critical system tables? These are the system tables that hold Storage Engine metadata. Without these we'd have no idea where any data was stored in the database files or how to interpret records.

In SQL Server 2000, the critical system tables are:

sysindexes
sysobjects
syscolumns

These tables have to be checked first because we use the metadata they contain to access all the other tables and indexes in the database. These tables are freely queryable so poke about and see what's stored in there.

In SQL Server 2005, the metadata layer has been rewritten and the critical system tables are:

sysallocunits
syshobts
syshobtcolumns
sysrowsets
sysrowsetcolumns

More on allocation units, hobts, and rowsets later in the week - for now you can assume they serve the same function as the three critical system tables in SQL Server 2000. You can't see these system tables because they're 'hidden' - the parser won't allow them to be bound to in a query. Try running 'select * from sysallocunits' to see what I mean.

The primitive checks are designed to check that internal queries on the metadata tables won't throw errors. Each of the critical system tables has a clustered index. The primitive checks just check the leaf-level data pages of the clustered indexes. For every one of these pages, the following is done:

Read and latch the page (a latch is a lightweight internal version of a lock). This makes sure that there aren't any IO problems with the page such as a torn-page or bad page checksum and ensures that we can put the page in the buffer pool correctly. This is the most common cause of failure of the primitive system table checks and results in error 8966, which in SQL Server 2000 could look something like:

Server: Msg 8966, Level 16, State 1, Line 1
Could not read and latch page (1:33245) with latch type SH. sysobjects failed.

Audit the page. This is a series of checks of the page structures which I'll cover in a seperate post. If these pass, the page looks like a SQL Server page of type its supposed to be.
Check the basic page linkage. Pages in each level of a clustered index are linked together in a doubly-linked list to allow range scans to work. At this stage we only check the left-to-right linkage to ensure the linked-to page actually exists.
Check the page linkage for loops. This is simple to do - have two pointers into the page linked-list with one advancing at every step and one advancing at every second step. If they ever point to the same thing before the faster-advancing pointer reaches the right-hand side of the leaf level then there's a loop. Its important that there are no linkage loops otherwise a range scan may turn into an infinite loop. I've never seen this occur in the field.

Any error found at this stage cannot be repaired so you must restore from a backup. This is because the repair would have to deallocate the page, effectively deleting metadata for a whole lot of tables and indexes. As people's databases get larger and more complex (thousands of tables and indexes), the percentage of pages that comprise these critical system tables rises and so the chance of a hardware problem corrupting one of these pages also rises - I see several of these a month on the forums. Without a backup, the only alternative is to try to export as much data as you can - not good.

If all the pages are ok then we know we've got solid enough metadata on which to base the next set of checks.

2. Allocation checks
(Part 2...)

3. Logical checks of critical system tables
4. Logical checks of all tables
(Part 3...)

5. Service Broker checks
6. Metadata cross-checks
7. Indexed view and XML index checks
(Part 4...)

SQL Server Storage Engine : DBCC

Example corrupt database to play with and some backup/restore things to try

Oh no – my backup is corrupt too! Help!

How long does *your* CHECKDB take?

CHECKDB (Part 8): Can repair fix everything?

CHECKDB (Part 7): How long will CHECKDB take to run?

Fixing damaged pages using page restore or manual inserts

More undocumented fun: DBCC IND, DBCC PAGE, and off-row columns

CHECKDB (Part 6): Consistency checking options for a VLDB

CHECKDB (Part 5): What does CHECKDB really do? (part 4 of 4)

CHECKDB (Part 4): What does CHECKDB really do? (part 3 of 4)

Don't try this at home kids... (data recovery using DBCC PAGE)

Poking about with DBCC PAGE (Part 1 of ?)

Can't I ever get a guarantee?

CHECKDB (Part 3): What does CHECKDB really do? (2 of 4)

CHECKDB (Part 2): What does CHECKDB really do? (1 of 4)

How long does your CHECKDB take?