SQL Server Storage Engine : CHECKDB Series

How long does your CHECKDB take?

Paul Randal - MSFT — Tue, 13 Feb 2007 04:29:00 GMT

Following on from my post a couple of weeks ago (https://blogs.msdn.com/sqlserverstorageengine/archive/2007/01/24/how-long-will-checkdb-take-to-run.aspx), I'm very interested to know how long it takes for your CHECKDBs to run, so I can get an idea of the distribution of run-times on various kinds of hardware for various size databases.

So, if you have a couple of minutes, I'd be grateful if you could c&p the following into an email to me, prandal@microsoft.com, and answer the questions for one or more of your databases. I'd especially like to hear from those of you who work for Microsoft and look after multiple customers. If I get a worthwhile number of responses (say 100 or so) then I'll collate the responses and post a summary. I'd love to hear about databases larger than 1TB.

For the first 10 Microsoft and the first 10 non-Microsoft people to respond, I'll send you the Always-On DVD loaded with Hands-On Labs that walk you through a bunch of the various HA/DR technologies. (I'll let you know whether you win and you need to send me your address).

Many thanks!

How large is the database?
What's the size of the largest table?
Do you have any indexed views?
Which command do you run, including options?
How long does the command take to run?
How often do you run it?
Which version & edition of SQL Server are you running?
(If SQL Server 2005, did you notice an increase or decrease in runtime?)
How many CPUs/cores does the server have?
How much memory does the server have?
What platform is the server?
Describe your IO subsystem?
Any comments?

CHECKDB (Part 8): Can repair fix everything?

Paul Randal - MSFT — Sun, 04 Feb 2007 16:53:00 GMT

I was teaching at a Microsoft-internal class last week and there was a discussion on what corruptions can't be repaired using DBCC. At the same time, several threads popped up on forums and newsgroups with people hitting some of this unrepairable corruptions so I thought that would make a good topic for the next post in the CHECKDB series.

Before anyone takes this the wrong way - what do I mean by "can't be repaired"? Remember that that purpose of repair is to make the database structurally consistent, and that to do this usually means deleting the corrupt data/structure (that's why the option to do this was aptly named REPAIR_ALLOW_DATA_LOSS - see this post for more explanation on why repair can be bad). A corruption is deemed unrepairable when it doesn't make sense to repair it given the damage the repair would cause, or the corruption is so rare and so complicated to repair correctly that it's not worth the engineering effort to provide a repair. Remember also that recovery from corruptions should be based on a sound backup strategy, not on running repair, so making this trade-off in functionality makes perfect sense.

Here's a few of the more common unrepairable corruptions that people run into along with the reasons they can't be repaired by DBCC.

PFS page header corruption

An example of this is on SQL Server 2005:

Msg 8946, Level 16, State 12, Line 1
Table error: Allocation page (1:13280496) has invalid PFS_PAGE page header values.
Type is 0. Check type, alloc unit ID and page ID on the page.

CHECKDB uses the PFS pages to determine which pages are allocated - and so which pages to read to drive the various consistency checks. The only repair for a PFS page is to reconsutruct it - they can't simply be deleted as they're a fixed part of the fabric of the database. PFS pages cannot be rebuilt because there is no infallible way to determine which pages are allocated or not. There are various algorithms I've experimented with to rebuild them with optimistic or pessimistic setting of page allocation statuses and then re-run the various consisteny checks to try to sort out the incorrect choices, but they all require very long run-times. Given the frequency with which we see these corruptions, and the engineering effort required to come up with an (imperfect) solution, I made the choice to leave this as unrepairable.

Critical system table clustered-index leaf-page corruption

An example of this is on SQL Server 2000:

Server: Msg 8966, Level 16, State 1, Line 1
Could not read and latch page (1:18645) with latch type SH. sysindexes failed.

And on SQL Server 2005:

Msg 7985, Level 16, State 2, Server SUNART, Line 1
System table pre-checks: Object ID 4. Could not read and latch page (1:51) with
latch type SH. Check statement terminated due to unrepairable error.

In a previous post I described why how and why we do special checks of the clustered indexes of the critical system tables. If any of the pages at the leaf-level of these indexes are corrupt, we cannot repair them. Repairing would mean deallocating the page, wiping out the most important metadata for potentially hundreds of user tables and so effectively deleteing all of these tables. That's obviously an unpalatable repair for anyone to allow and so we don't do it.

Column value corruption

Here's an example of this on SQL Server 2005:

Msg 2570, Level 16, State 3, Line 1
Page (1:152), slot 0 in object ID 2073058421, index ID 0, partition ID 72057594038321152, alloc unit ID 72057594042318848 (type "In-row data"). Column "c1" value is out of range for data type "datetime". Update column to a legal value.

This is where a column has a stored value that is outside the valid range for the column type. There are a couple of repairs we could do for this:

delete the entire record
insert a dummy value

#1 isn't very palatable because then we've lost data and its not a structural problem in the database so doesn't have to be repaired. #2 is dangerous - what value should we choose as the dummy value? Any value we put in may adversely affect business logic, or fire a trigger, or have some unwelcome meaning in the context of the table - even a NULL. Given these problems, we chose to allow people to fix the corrupt values themselves.

Metadata corruption

Here's an example of this on SQL Server 2005:

Msg 3854, Level 16, State 1, Line 2
Attribute (referenced_major_id=2089058478) of row (class=0,object_id=2105058535,column_id=0,referenced_major_id=2089058478,referenced_minor_id=0) in sys.sql_dependencies has a matching row (object_id=2089058478) in sys.objects (type=SN) that is invalid.

This example is relatively benign. There are other examples that will cause CHECKDB to terminate - not as bad as the critical system table corruption example above, but enough that CHECKDB doesn't trust the metadata enough to use it to drive consistency checks. Repairing metadata corruption has the same problems as repairing critical system table corruption - any repair means deleting metadata about one or more tables, and hence deleting the tables themselves. It's far better to leave the corruption unrepaired so that as much data as possible can be extracted from the remaining tables.

Summary

Repair can't fix everything. You may end up having to perform manual and time-consuming data extraction from the corrupt database and losing lots of data because of, say, a critical system table corruption. Bottom line (as usual) - make sure you have valid backups so you don't get into this state!

CHECKDB (Part 7): How long will CHECKDB take to run?

Paul Randal - MSFT — Wed, 24 Jan 2007 13:12:00 GMT

This is a question I see every so often and it cropped up again this morning so I'll use it as the subject for this week's blog post.

There are several ways I could answer this:

the unhelpful answer - I've got no idea.
the almost-helpful answer - how long did it take to run last time and are the conditions exactly the same?
the answer I usually give - it depends.

Now, many people would see the third answer as being somewhat equivalent to the first answer - unhelpful. The problem is that there are many factors which influence how long CHECKDB will take to run. Let me explain the ten most important factors so you get an idea why this is actually a helpful answer. These aren't in any particular order of importance.

1) The size of the database

Pretty obvious...

2) Concurrent IO load on the server

At the simplest level, what is CHECKDB going to do? It reads every allocated page in the database. That's a lot of IO. CHECKDB takes great pains to do the most efficient IO it can and read the database pages in their physical order with plenty of readahead so that the disk heads move smoothly across the disks (rather than jumping around randomly and incurring disk head seek delays). If there's no concurrent IO load on the server, then the IOs will be as efficient as CHECKDB can make them. However, introducing any additional IO from SQL Server means that the disk heads will be jumping around - slowing down the CHECKDB IOs. If the IO subsystem is at capacity already from CHECKDB's IO demands, any additional IO is going to reduce the IO bandwidth available to CHECKDB - slowing it down.

3) Concurrent CPU activity on the server

At the next level of simplicity, CHECKDB is going to process every page it reads in some way. Depending on the various options you've specified and the database schema (details below), that's going to use a lot of CPU - it's possible that the server may be pegged at 100% CPU when CHECKDB is running. If there's any additional workload on the server, that's going to take CPU cycles away from CHECKDB and it going to slow it down.

Basically what these first two points are saying is that CHECKDB is very resource intensive! Its probably one of the most resource intensive things you can ask SQL Server to do and so it's usually a good idea to not run it during peak workload times, as you'll not only cause CHECKDB to take longer to run, you will slowdown the concurrent workload, possibly unacceptably.

4) Concurrent update activity on the database

This is relevant for both SQL 2000 and SQL 2005, but for different reasons.

In SQL 2000, CHECKDB gets its consistent view of the database from transaction log analysis of concurrent DML transactions (see CHECKDB (Part 1): How does CHECKDB get a consistent view of the database for full details). The more concurrent DML there is while CHECKDB is running, the more transaction log will be generated - and so the longer it will take for CHECKDB to analyze that transaction log. It's possible that on a large multi-CPU box with a ton of concurrent DML and CHECKDB limited to a single CPU that this phase of CHECKDB could take several times longer than the reading and processing of the database pages! (I've seen this in real-life several times.)

In SQL 2005, CHECKDB gets its consistent view of the database from a database snapshot, which is stored on the same disk volumes as the database itself. If there are a lot of changes in the database while CHECKDB is running, the changed pages are pushed to the snapshot so that it remains consistent. As the snapshot is stored on the same disks as the database, every time page is pushed to the snapshot, the disk head has to move, which interrupts the efficient IO described in #2. Also, whenever CHECKDB goes to read a page and it needs to read the page from the snapshot files instead of the database files, that's another disk head move, and another efficient IO interruption. The more concurrent changes to the database, the more interruptions to efficient IO and the slower the CHECKDB runs.

5) Throughput capabilities of the IO subsystem

This one's simple. CHECKDB is going to do a boat-load of IOs and it could even end up being IO-bound (meaning that the CPUs are idle periodically waiting for IOs to complete) depending on the options specified and the database schema. This means that the throughput of the IO subsystem is going to have a direct effect on the run-time of CHECKDB. so, if you have a 1TB database and the IO subsystem can only manage 100MB/sec, it's going to take almost 3 hours just to read the database (1TB / 100MB / 3600 secs) and there's nothing you can do to speed that up except upgrade the IO subsystem.

I've lost count of the number of times I've heard customers complain that CHECKDB (or index rebuilds or other IO-heavy operations) are running sloooowly only to find that the disk queue lengths are enormous and the IO subsystem it entirely unmatched to the server and workload.

6) The number of CPUs on the box

This also really encompasses the Edition of SQL Server that's being run. In Enterprise Edition, CHECKDB can run in parallel across all the CPUs in the box (or as many as the query processor decides to parallelize over when the CHECKDB internal queries are compiled). Running in parallel can give a significant performance boost to CHECKDB and lower run times, as long as the database is also spread over multiple files too (so the IOs can be parallelized). There's a nifty algorithm we use that allows us to run in parallel whcih I'll explain in detail in a future post.

On the other hand, however, the fact that CHECKDB can run in parallel in Enterprise Edition can be bad for some scenarios, and so some DBAs chose to force CHECKDB to be single-threaded. The way to do this is to turn on the documented trace flag 2528.

7) The speed of the disks where tempdb is placed

Running CHECKDB against a VLDB uses lots of memory for internal state and for VLDBs the memory requirement usually exceeds the amount of memory available to SQL Server. In this case, the state is spooled out to tempdb and so the performance of tempdb can be a critical factor in CHECKDB performance.

8) The complexity of the database schema

This can have a really high impact on the run-time of CHECKDB because it impacts the amount of CPU that CHECKDB requires. For example, the most expensive checks that CHECKDB does are for non-clustered indexes. We need to check that each row in a non-clustered index maps to exactly one row in the heap or clustered index for the table, and that every heap/clustered index row has exactly one matching row in each non-clustered index. Although there's a highly efficient algorithm for doing this, it still takes around 30% of the total CPU that CHECKDB uses!

There are a bunch of other checks that are only done if the features have been used in the database - e.g. computed column evaluation, links between off-row LOB values, Service Broker, XML indexes, indexed views - so you can see that empirical factors along aren't enough to determine the run-time.

9) Which options you've specified

This is almost the same as #7 in that by specifying various options you're limiting what checks CHECKDB actually performs. For instance, using the WITH NOINDEX option will turn off the non-clustered index checks that I described in #7 and using the WITH PHYSICAL_ONLY option will turn off all logical checks, vastly decreasing the run-time of CHECKDB and making it nearly always IO-bound rather than CPU-bound (in fact this is the most common option that DBAs of VLDBs use to make the run-time of CHECKDB manageable).

One thing to be aware of - if you specify any repair options, CHECKDB always runs single-threaded, even on a multi-proc box.

10) The number and type of corruptions that exist in the database

Again, this is similar to #7 and #8. If there are any corruptions present, there may be extra checks triggered to try to figure out more details of the corruptions. For instance, for the non-clustered index checks, the algorithm is tuned very heavily for the case when there are no corruptions present (the overwhelming majority of cases considering the millions of times CHECKDB is run every day around the world). When a non-clustered index corruption is detected, a more in-depth algorithm has to be used to figure out exactly where the corruption is, which involves re-scanning a bunch of data and so taking a bunch more time. There are a few other algorithms like this too.

Summary

So you can see that there's no simple answer.

Now, there's only one time when you should be trying to work out how long a CHECKDB is going to take - when you're planning your regular database maintenance (if you find that it's going to take too long, checkout some of the options for splitting up the checks in my previous post on CHECKDB (Part 6): Consistency checking options for a VLDB). If you're faced with a corrupt (or suspected corrupt) database and you're only just starting to think about how long a CHECKDB is going to take - you've screwed up. You should at least know how long a CHECKDB usually takes.

Of course, there's one more really unhelpful answer I could have given - how long is a piece of string?!? :-)

CHECKDB (Part 6): Consistency checking options for a VLDB

Paul Randal - MSFT — Fri, 20 Oct 2006 02:57:00 GMT

(Yippee - just finished my certification dives and got my PADI Open Water certification - just in time for our dive trip to Indonesia in December :-)

This is a question that comes up a lot - in fact 3 times this week already - most recently during a guest lecture I did on DBCC for one of Kimberly's popular "Immersion Events". The question is: how can I run consistency checks on a VLDB?

We're talking hundreds of GBs or 1 TB or more. These databases are now common on SQL Server 2000 and 2005, with more migrations happening all the time. Any sensible DBA knows the value of running consistency checks, even when the system is behaving perfectly and the hardware is rock-solid. The two problems that people have with running a full DBCC CHECKDB on their VLDB are:

It takes a long time to run (proportional to the database size and schema complexity).
It uses lots of resources.

So it uses lots of resources for a long time. Even with a decent sized maintenance window, the CHECKDB may well run over into normal operations. There's also the case of a system that's already pegged in more or more resource dimensions (memory, CPU, IO bandwidth). Whatever the case, there are a number of options:

Don't run consistency checks
Run a DBCC CHECKDB using the WITH PHYSICAL_ONLY option
Use SQL Server 2005's partitioning feature and devise a consistency checking plan around that
Figure out your own scheme to divide up the consistency checking work over several days
Use a separate system

Let's look at each in turn.

Don't run consistency checks

Don't be daft. Don't even think about using this option. If you absolutely cannot figure out a way to get consistency checks on your system, send me email and I'll help you. Now let's move on to serious options...

Use WITH PHYSICAL_ONLY

A full CHECKDB does a lot of stuff - see my CHECKDB internals series for more details. You can vastly reduce the run-time and resource-usage of CHECKDB by using the WITH PHYSICAL_ONLY option. With this option, CHECKDB will

Run the equivalent of DBCC CHECKALLOC (i.e. check all the allocation structures)
Read and audit every allocated page in the database

So it skips all the logical checks, inter-page checks, and things like DBCC CHECKCATALOG. The fact that all allocated pages are read means that:

Any pages that cannot be read at all (i.e. 823 errors) will be discovered
If page checksums are enabled in SQL Server 2005, any corruptions caused by storage hardware will be discovered (as the page checksum will have changed).

So there's a trade-off of consistency checking depth against runtime and resource usage.

Use the partitioning feature to your advantage

One of the obvious ways to reduce the time/resources issue is to partition the load. If you're using the partitioning feature in SQL Server 2005 then you're already setup for this. Given that you've hopefully got your partitions stored on seperate filegroups, you can use the DBCC CHECKFILEGROUP command.

Consider this example - you have the database partitioned by date such that the current month is on a read-write filegroup and the past 11 months are on read-only filegroups (data from more than a year ago is on some offline storage medium). The prior months also have multiple backups on various media so are considered much 'safer' than the current month. It makes sense then that you don't need to check these filegroups as often as the current month's filegroup so an example consistency checking scheme would be:

Run a DBCC CHECKFILEGROUP on each read-only filegroup every week or two
Run a DBCC CHECKFILEGROUP on the read-write filegroup every day or two (depending on the stability of the hardware, the criticality of the data, and the frequency and comprehensiveness of your backup strategy).

I know of several companies who've made the decision to move to SQL Server 2005 in part because of this capability to easily divide up the consistency checking.

Figure out your own way to partition the checks

If you're on SQL Server 2000, or you just haven't partitioned your database, then there are ways you can split up the consistency checking workload so that it fits within a maintenance window. Here's one scheme that I've recommended to several customers:

Run a bi-weekly DBCC CHECKALLOC
Figure out your largest tables (by number of pages) and split the total number into 7 buckets, such that there are a roughly equal number of database pages in each bucket.
Take all the remaining tables in the database and divide them equally between the 7 buckets (using number of pages again)
On Sunday:

Run a DBCC CHECKALLOC
Run a DBCC CHECKCATALOG
Run a DBCC CHECKTABLE on each table in the first bucket

On Monday, Tuesday, Wednesday:

Run a DBCC CHECKTABLE on each table in the 2nd, 3rd, 4th buckets, respectively

On Thursday:

Run a DBCC CHECKALLOC
Run a DBCC CHECKTABLE on each table in the 5th bucket

On Friday and Saturday:

Run a DBCC CHECKTABLE on each table in the 6th and 7th buckets, respectively

In pre-RTM builds of SQL Server 2005, DBCC CHECKTABLE could not bind to the critical system tables, just like with T-SQL - but that's fixed so you can cover all system tables in SQL Server 2000 and 2005 using the method above. Here's what I mean:

C:\Documents and Settings\prandal>osql /E
1> select * from sys.sysallocunits
2> go
Msg 208, Level 16, State 1, Server SUNART, Line 1
Invalid object name 'sys.sysallocunits'.
1> dbcc checktable ('sys.sysallocunits')
2> go
DBCC results for 'sysallocunits'.
There are 112 rows in 2 pages for object "sysallocunits".
DBCC execution completed. If DBCC printed error messages, contact your system
administrator.
1>

There's one drawback to this method - a new internal database snapshot is created each time you start a new DBCC command, even for a DBCC CHECKTABLE. If the update workload on the database is significant, then there could be a lot of transaction log to recover each time the database snapshot is created - leading to a long total run-time. In this case, you may need to alter the number of buckets you use to make the total operation fit within your available window.

Use a separate system

This alternative is relatively simple - restore your backup (you are taking regular backups, right?) on another system and run a full CHECKDB on the restored database. This offloads the consistency checking burden from the production system and also allows you to check that your backups are valid (which you're already checking though, right?). There are some drawbacks to this however:

You need to have sufficient disk space on the spare system to be able to restore the backup onto. If the production database is several TB, you need the same several TB on the spare box. This equates to a non-trivial amount of money - initial capital investment plus ongoing storage mgmt costs. (I'm working on this though - I have a patent on consistency checking a database in a backup without restoring it - unclear at this time whether it will make it into Katmai.)
If the consistency checks find an error, you don't know for sure that the database is corrupt on the production system. It could be a problem with the spare box that's caused the corruption. The only way to know for sure is to run a consistency check on the production system. This is a small price to pay though, because most of the time the consistency checks on the spare system will be ok, so you know the production database was clean at the time the backup was taken.

Summary

You've got a bunch of choices to allow you to run consistency checks, so there's really no excuse for not knowing (within a reasonable timeframe) that something's gone wrong with your database. If you need further help working out what to do, or just want a critical eye cast over the plan you've come up with, send me an email at prandal@microsoft.com

(One of the other questions that also keeps coming up is: when are you going to write the whitepaper you promised? Q1CY07 - honestly!)

CHECKDB (Part 5): What does CHECKDB really do? (part 4 of 4)

Paul Randal - MSFT — Wed, 20 Sep 2006 09:03:00 GMT

(Another airport, another blog post - I really must make an effort to come up with more original banter - I'm sure I've used that line before. TechEd Guangzhou has finished and I'm on the way to Beijing for TechEd #3. Checking in and getting through security were challenging to say the least this morning - there seemed to be problems at every turn, exacerbated by my China Southern airlines chaperone who spoke limited English (or what I should really say is that the problem was I don't speak Mandarin). Finally at the security checkpoint they asked for my ticket as well as my boarding pass - try explaining "e-ticket" to someone who doesn't speak the same language...)

So part 4 of the high-level explanation of what DBCC CHECKDB does will focus on the three completely new sets of checks that are in CHECKDB in SQL Server 2005. Well, the metadata checks (i.e. DBCC CHECKCATALOG) used to exist but so many people used to run both CHECKDB and DBCC CHECKCATALOG that I thought I'd make it easier in SQL Server 2005 by running DBCC CHECKCATALOG as part of CHECKDB. Maybe in Katmai we'll add in DBCC CHECKCONSTRAINTS - haven't decided yet.

Primitive checks of critical system tables
(Part 1...)

Allocation checks
(Part2...)

Logical checks of critical system tables
Logical checks of all tables
(Part 3...)

Service Broker checks
Metadata cross-checks
Indexed view and XML index checks
Its important that these checks are done after the per-table logical checks. This is because any repairs done as part of the per-table checks may affect the outcome of these checks quite substantially.

Let's look at an example of this. Imagine the case where an indexed view is based on a join between two tables, foo and bar. Table foo has a damaged page and is damaged in a way that reading the page as part of a query would not recognize - because full page audits are not done as part of regular page reads. The damage is such that the page has to be deleted as part of repair - thereby changing the results of the view. If the indexed view was checked before tables foo and bar, then the repair that was done would not get reflected in the indexed view and so the indexed view would essentially be corrupt. The same logic holds for checking XML indexes and Service Broker tables.

(Ok - in the air now with some serious turbulence - luckily I popped a Dramamine before we took off. And, more jellyfish in the in-flight meal - very cool :-) The 1st class cabin has what looks like a fully stocked bar - a little too early at 10am to be starting, although it could make for some interesting blog posts...)

Service Broker checks
The Service Broker dev team wrote a comprehensive set of checks of the data stored in their internal on-disk structures. The checks validate the relationships between conversations, endpoints, messages and queues. For example:

a conversation must have two endpoints
a service must be related to a valid contract
a service must be related to a valid queue
a message must have a valid message type

What's even cooler is that they implemented a set of logical repairs, so if one of their internal tables was damaged, and repaired by the earlier logical checks, then the Service Broker repair code can clean up any Service Broker entities and entity relationships that were damaged too.

Metadata cross-checks
The Metadata team also wrote a great set of checks for the relational metadata stored in the system tables. These checks that are run here are the same code that's run for DBCC CHECKCATALOG. The actual checks themselves in SQL Server 2005 are far more involved than in SQL Server 2000, and they're done way more efficiently too. However, they're not comprehensive by any means - we'll add some more checks during Katmai.

The checks only cover the relational metadata - i.e. the relationships between system table storing relational metadata. There are no such checks for the tables storing the storage engine metadata (also known as the critical system tables that were described in Part 1 of this 4 part sub-series. At present, these relationships between these tables are checked implicitly by the metadata checks I described in the last post. The checks are a series of validations between the various system tables, such as checking each rowset referenced in the sysrowsetcolumns system table exists in the sysrowsets system table.

There are no repairs for metadata corruptions. Metadata corruptions are extremely difficult to deal with because changing/deleting metadata has an effect on the entire table the corrupt metadata describes and could potentially be as bad as deleting the table - a table without metadata is just a collection of pages with indecipherable rows (well, not quite true - its possible to decipher any record without metadata but it requires human intervention and is incredibly hard).

Its possible we may put in some limited metadata repairs in a future release, but the frequency of their occurence in the field is so low that I decided that the engineering investment was not justified for SQL Server 2005 and so didn't push it. So - if you get any metadata corruption, you need to restore from your backups which, of course, after reading through my blog, you've been scared into making sure you have, right?...

Indexed view and XML index checks
These are very cool and many thanks to Conor Cunningham (dev lead of one of the two Query Optimizer teams) for helping work out how to do this.The indexed view contains the persisted results of a query, and we have the actual query it persists stored in metadata - so the easiest way to check whether the indexed view is accurate is to recalculate the view results into a temp table in TEMPDB and then compare the calculated values with the values persisted in the indexed view. The view is regenerated into a temporary table and then two concurrent left-anti-semi-joins are run, that basically return all the rows in the indexed view that are not in the recalculated view results, and vice-versa. This gives us all the rows that are extraneous in the indexed view and all the rows that are missing from it.

Indexed-view problems can also be repaired. The repair for extra rows is to delete them one by one (using internal query syntax that only works from DBCC), and the repair for missing rows is to rebuild the indexed view. This is done by simply disabling the indexed view and then bringing it back online (which rebuilds it).

There are two drawbacks to the indexed view checks if the views are large:

it can take up a lot of space in tempdb
it can take a lot of time to run the regeneration of the indexed views and to run the left-anti-semi-joins

So if you upgraded your database from SQL Server 2000 and you regularly run full CHECKDBs (i.e. without using WITH PHYSICAL_ONLY), then you may see a run-time increase for CHECKDB on SQL Server 2005 - this is documented in BOL and the README.

XML index checks work in a similar way. The XML blobs in the table are re-shredded and checked against the shredding that's persisted in the primary XML index. If anything is wrong, the primary XML index is recreated.

And that's it. Now you have a complete picture of what's going on with CHECKDB when it runs. In the next few posts I want to give some insight into how CHECKDB manages to do all these checks while only making a single pass through the database and shed some light on how repair works. Until then - time to chill out before doing more sessions in Beijing tomorrow... qing lai yiping pijiu!!!

CHECKDB (Part 4): What does CHECKDB really do? (part 3 of 4)

Paul Randal - MSFT — Mon, 18 Sep 2006 21:43:00 GMT

(OK - I was wrong about the posting frequency - events overtook me and I got bogged down preparing for these TechEds. China is an amazing place and I wish I had more time to soak up some of its culture while I'm here but with the TechEds being so close together there's little time for anything except sessions, email, taxis, flying, sleeping, jellyfish... I did manage to go to the 400 year old Yuyuan Gardens in Shanghai, which are well worth a visit. Now I'm sitting here next to the Pearl River in Guangzhou waiting to do DAT305: Choosing a High Availability Solution - so time to squeeze in another post.)

In the previous posts of this series, I covered the system table checks that have to be done before anything else can be checked by CHECKDB, and the allocation checks that have to be done before any of the logical checks. Now I want to cover the meat of the functionality in CHECKDB - the logical checks. Note - this is a description of what happens for SQL Server 2005 - its pretty similar in SQL Server 2000.

Primitive checks of critical system tables
(Part 1...)

Allocation checks

(Part 2...)

Logical checks of critical system tables
Logical checks of all tables
Although this section has two titles, the actual logical checks performed in each stage are the same - i.e. everything I'm going to describe below. If any errors are found in the critical system tables (remember these from Part 1?) and:

repair is not specified; or,
repair is specified, but not all the errors can be repaired

then the CHECKDB finishes. An example of an unrepairable system table error is something that would require deleting data from one of the system tables - e.g. a corrupt key value in a clustered index data page of sysallocunits (remember that this is the actual table I'm talking about, not the sys.allocation_units catalog view you may have seen or used).

So if the system tables are clean, all the tables in the database are checked. This includes indexed views and primary XML indexes (which are both stored as clustered indexes - and as far as the storage engine is concerned are objects in their own right - its the relational layer that knows that they're not really separate objects).

The following checks are performed:

Validate each table's storage engine metadata
Read and check all data, index and text pages, depending on the page type
Check all inter-page relationships
Check the page header counts
Perform any necessary repairs

Let's look at each of these stages in more detail.

Validate each table's storage engine metadata

Why do we need to do this? Well, the storage engine metadata is what tells CHECKDB how to crack open records on pages in particular rowsets - so if there's something wrong with it then CHECKDB will generate a bunch of misleading errors or, worse yet, miss some errors. Here's roughly what happens while the metadata is parsed:

Loop through all the indexes, and then all the rowsets of each index, building a list of known allocation units for the index (including all the DATA, LOB, and SLOB allocation units). This allows reverse lookups - when we read a page, because the information stamped on the page is the allocation unit, we need a very fast way to convert between an allocation unit and an index/object ID.
Make sure we skip any indexes that are in the middle of being built/rebuilt online, because they will not have a complete set of data.
Build a list of computed columns and generate the necessary code to enable the column values to be recomputed.
Build a list of columns that are used in non-clustered indexes
Make sure the various USED, DATA, RSVD counts are not negative (see the BOL for DBCC CHECKDB for an explanation of how this can happen - yes, I'm deliberately not explaining here so that you read through the new BOL entry :-)
Figure out what each kind of column is (e.g. a regular column, a generated uniquifier, a dropped column)
Build a series of mappings to allow conversion between column IDs at different levels of storage abstraction
Check that the relational and storage-engine nullability flags for a column agree
Make sure the columns counters in metadata match what we've just seen

Read and check all data, index and text pages

No matter what type a page is, the page is audited and then all the records on it are audited.

Page audit checks first of all for IO errors when the page is read (e.g. page checksums or torn-page detection). If there are some, then the page is not processed any further. This is what can lead to errors like 8976 being reported (e.g. Table error: Object ID 132765433, index ID 1, partition ID 72057594038321152, alloc unit ID 72057594042318848 (type DATA). Page (1:3874) was not seen in the scan although its parent (1:3999) and previous (1:3873) refer to it. Check any previous errors.). Then it checks for page header correctness and that the page has an appropriate type for the allocation unit its in (e.g. a DATA page should not be found in an allocation unit for a non-clustered index)

Record audits include checking the various fields in the record header and that the various offsets in the record make sense (e.g. the offset to the variable length columns section of the record should not point off the end of the record. See the post on cracking records using DBCC PAGE for more info on the record format.

The more complex checks that are done per-page depend on what type the page is. As an example, here's what is done for a DATA page in the leaf level of a clustered index (excluding the inter-page relationships - I'll list those in the next section):

Records in a page must be strictly ordered by the defined keys of the index (although the records themselves aren't necessarily stored in sorted order in the data portion of the page, accessing the records through the slot array must yield them in the correct order)
No two records can have duplicate key values (remember that non-unique indexes have a hidden, automatically-generated uniquifier column added to the key - to make or extend the composite key - so that record uniqueness is guaranteed)
If the index is partitioned, each record is run through the partitioning function to ensure its stored in the correct partition
All the complex columns in each record are checked:
- Complex columns are those storing legacy text or LOB values (text, ntext, image, XML, nvarchar(max), varchar(max), varbinary(max)) or in-row pointers to variable length columns that have been pushed off-row in rows that are longer than 8060 bytes
- The column is checked to make sure its storing the right kind of data - either the value itself or a text pointer or some kind of in-row root containing pointers to portions of an off-row value.
- The linkages between what is stored in-row and the off-row values stored in other pages are eventually checked too
Check computed columns
- If the column is persisted (either because its defined as a persisted computed column or because its used as a non-clustered index key), its value is recomputed and checked against the persisted value
- This is also important when we come to do the non-clustered index cross-checks - as any discrepancy in the stored column values will cause mismatches.
Data purity checks
- The column value is checked to ensure its within the bounds for its data-type (e.g. the minutes-past-midnight portion of the internal representation of a datetime value cannot be greater than 1440 - 24 hours x 60 minutes)
Non-clustered index cross-checks

This is probably my favorite part of CHECKDB and is one of the most complicated bits of code.
What CHECKDB is trying to do is make sure that each record in a heap or clustered index has exactly one matching record in each non-clustered index, and vice-versa. The brute-force (n-squared complexity) way to do this is to do a physical lookup of all the matching rows - but that's incredibly time consuming so instead we have a fast algorithm to detect problems.
Imagine a table defined by the following DDL:

      CREATE TABLE t (c1 INT, c2 char(10), c3 varchar(max))

      CREATE INDEX index1 ON t (c1)

      CREATE INDEX index2 ON t (c2) INCLUDE (c3)

Each row in the heap has to have two matching non-clustered index rows, on in each of index1 and index2. But how can we tell without doing the direct lookup or having to store tremendous amounts of state? We use a hash-bitmap algorithm.
Imagine a large bitmap - say half a million bits. Initially all the bits are zero.
For each record in the heap that we come across, we generate the matching non-clustered index records and hash them to a value. This is very quick because we have all the column values we need. We then take the hash value, map it directly to a bit in the bitmap and flip the bit. So in our example above, each record in the heap will produce two hash values (one for the record in each non-clustered index) and so will cause two bits to be flipped.
For each record in each non-clustered index, just hash the record and flip the corresponding bit.
The idea is that the bit-flips should cancel each other out and the bitmap should be left with all zeroes at the end of the checks.
Taking the view that corruptions are magnitudes rarer than clean databases, we went a step further and allowed the checks for multiple tables to use the same bitmap. If you think about it, this won't cause any problems, even if two sets of records map to the same bit in the bitmap - as long as the number of bit-flips is a power of 2 (i.e. each record really does have its correct matching record) then there should be no problem.
Here's the catch - what happens when there's a bit left on in the bitmap? Well, this is where the trade-off comes into play. If there's a bit left on, we can't tell which records in which table or index mapped to it so, we have to re-scan the tables and indexs that used the bitmap to see which records map to the bit. For every one we find, we actually do the physical lookup of the matching record and then do a comparison of all the columns, including any LOB columns used as INCLUDE'd columns in non-clustered indexes. This process is called the deep-dive and can add a significant amount to the run-time of CHECKDB if it occurs.

Check all inter-page relationships

Inter-page relationships are relevant for:

pages in heaps that have forwarding or forwarded records
pages in indexes
pages that have records with LOB columns that have their data stored off-row (these can be heap pages, clustered index data pages or non-clustered index leaf pages in SQL Server 2005)
text pages that have records with child nodes on other pages

Continuing the example above, here are the checks that are done for index pages:

All pages in an index level should be pointed to by a page in the next level higher in the b-tree, and also by the left- and right-hand neighboring pages in the same level of the b-tree. Exceptions are made for the left-most and right-most pages in a level (where the m_prevPage and m_nextPage fields are (0:0)) and for the root page of the b-tree, where the parent page link comes from the storage-engine metadata. The 8976 error message that I referenced above is generated from this set of checks.
Key ranges in neighboring pages in a level should not overlap
Key ranges of pages should be correctly defined by the parent pages in the next level up in the b-tree (the parent pages contain the minimum key value that can exist on a child page)

I'll go into details of how the inter-page checks are done in a future post. For now, its enough to say that its not an n-squared complexity algorithm.

Check the page header counts

The page header contains a bunch of counters - the ones we need to check are:

slot count
ghost record count
free space count

The first two are obvious - count the rows as they're processed and make sure page header counts are valid. The free space count is only checked for text pages and data pages in a heap (the only pages for which free space is tracked in a PFS page).

Perform any necessary repairs

I want to leave discussing repairs for another post as there's a ton of info there.

So, that's the majority of the work in running a DBCC CHECKDB and now its time for my session on choosing an HA solution in SQL Server 2005. Next time I'll post the next part of the Fragmentation series.

(So I mentioned jellyfish above... I was having a meal last night with Scott Schnoll from the Exchange team and we had what we thought were noodles on the side of one dish. Scott asked what kind of noodles they were and then we found out we'd been eating cold, sliced jellyfish - it was really very tasty!)

Service Broker checks
Metadata cross-checks
Indexed view and XML index checks
(Part 4...)

CHECKDB (Part 3): What does CHECKDB really do? (2 of 4)

Paul Randal - MSFT — Tue, 18 Jul 2006 22:16:00 GMT

In the previous post of this series, I covered the system table checks that have to be done before anything else can be checked by CHECKDB. Now that I've described pages, extents, IAM chains/allocation units and the major allocation bitmaps, in this post I'll cover the allocation checks.

1. Primitive checks of critical system tables
(Part 1...)

2. Allocation checks

This stage is in SQL Server 2000 and 2005. Allocation checks are checks of the various structures (IAM pages, IAM chains, allocation units, GAM/SGAM pages, PFS pages) that describe the allocations (pages, extents) that have been done within a database.

I'll describe how we go about collecting information from the various pages and then describe what some of the actual checks are.

The allocation checks are very fast (orders of magnitude faster than the logical checks, so fast in fact that they happen in the blink of an eye! well, perhaps I'm getting carried away a little as usual but you get the idea) because the number of database pages that have to be read is very small (so small in fact that... ok, I'll shut-up). The algorithm for gathering allocation data is as follows:

For each file in each online filegroup in the database (except transaction log files):

read all PFS pages

This gives us a bitmap showing all IAM pages, plus another one showing all mixed pages

read the GAM pages

This gives us bitmaps of all allocated extents

read the SGAM pages

This gives us bitmaps of all mixed extents with at least one unallocated page

read the DIFF_MAP pages (a 'differential bitmap' page shows which extents in the GAM interval have been modified since the last full or differential backup - a differential backup only needs to backup those extents marked modified in the various DIFF_MAP pages)

This is just to make sure the pages can be read

read the ML_MAP pages (a 'minimally-logged bitmap' page shows which extents in the GAM interval have been modified in bulk-logged recovery mode since the last log backup - a log backup must also backup all such extents to ensure that all changes to the database have been backed up. This can make the log backup quite large (although the log itself stays much smaller) - but that's a topic for another blog post.

This is just to make sure the pages can be read

read all IAM pages

This gives us:

a list of all the mixed pages in the file, and by derivation, a list of all mixed extents in the file (remember that the first IAM page in an IAM chain/allocation unit contains an array to hold up to 8 mixed pages for the object/index/partition it represents
a list of all the valid IAM pages in the file
a list of all the allocated dedicated extents in the file
linkage information for IAM chains

After all the per-file stuff, read the Storage Engine metadata

This gives us:

information about the root of each IAM chain. Each row in the sys.allocation_units hidden metadata table contains the page Id of the first IAM page in the IAM chain for that allocation unit.
information about IAM chains currently waiting to be deferred-dropped. (This is the process by which an IAM chain with > 128 extents that is dropped - by dropping/rebuilding an index or dropping/truncating a table - does not have its actual pages and extents deallocated until after the transaction has committed. The IAM chain is unhooked from sys.allocation_units though and hooked into an internal queue - if we didn't scan that queue too as part of the allocation checks, we'd see all kinds of inconsistencies with the various allocation bitmaps)

So, now we've got a whole bunch of allocation data that we're juggling and we need to make sense of it all to see if all the allocation structures are correct. Here's a non-exhaustive list of checks that we do with all this data:

Check that each extent is either allocated to:

the GAM page for the GAM interval
or, the SGAM page for the GAM interval, as a non-full mixed extent
or, exactly one IAM page that covers the GAM interval
or, to no bitmap page, but all pages in the extent must be allocated to IAM pages as mixed pages
This could result in an 8903 (GAM and SGAM), 8904 (multiple IAMs), or 8905 (no page) errors depending on the combination of bitmaps that have the extent allocated

Check that all pages marked as being IAM pages in PFS pages really are IAM pages when they're read
Check that all pages marked as being mixed pages in PFS pages appear somewhere in a mixed page array on an IAM page
Check that each mixed page is only allocated in a single IAM page
Check that the IAM pages in an IAM chain have monatonically increasing sequence numbers
Check that the first IAM page in an IAM chain has a reference from a row in sys.allocation_units
Check that no two IAM pages within the same IAM chain map the same GAM interval
Check that all IAM pages within an IAM chain belong to the same object/index/partition
Check that the linkages within an IAM chain are correct (no missing pages for instance)
Check that all IAM/GAM/SGAM pages that map the final GAM interval in a file do not have extents marked allocated that are beyond the physical end of the file

Any errors found here will require REPAIR_ALLOW_DATA_LOSS to repair and some of the repairs are very complicated (e.g. multiply-allocated extents) - topic for a future blog post.

So, the allocation checks lay the next foundation level over the system table primitive checks and we're ready to move on to the logical checks.

3. Logical checks of critical system tables
4. Logical checks of all tables
(Part 3...)

5. Service Broker checks
6. Metadata cross-checks
7. Indexed view and XML index checks
(Part 4...)

CHECKDB (Part 2): What does CHECKDB really do? (1 of 4)

Paul Randal - MSFT — Mon, 19 Jun 2006 01:09:00 GMT

Hmmm - I sat for 5 minutes thinking of something amusing to say to start this one off and nothing came to mind, so I'm afraid this will be a humor-free post. Maybe I'm jet-lagged from being on the East coast all last week.

As with all things related to DBCC, this topic has its share of misinformation. In this post I'll set the record straight by running through all the stages of CHECKDB in SQL Server 2000 and 2005. I'll need to split this up into seperate posts otherwise I'll be writing a book. I also introduce a whole raft of new terms which will also be subjects for future posts (my list is already getting pretty long!)

So the very first thing it does is work out how to get the transactionally consistent view it requires (see CHECKDB Part 1) and then, if needed, either record the relevant LSN and switch to full-logging (for SQL Server 2000) or create a database snapshot (for SQL Server 2005).

Then it runs through the checks in the order shown below:

1. Primitive checks of critical system tables

This stage is in SQL Server 2000 and 2005. First of all, what are critical system tables? These are the system tables that hold Storage Engine metadata. Without these we'd have no idea where any data was stored in the database files or how to interpret records.

In SQL Server 2000, the critical system tables are:

sysindexes
sysobjects
syscolumns

These tables have to be checked first because we use the metadata they contain to access all the other tables and indexes in the database. These tables are freely queryable so poke about and see what's stored in there.

In SQL Server 2005, the metadata layer has been rewritten and the critical system tables are:

sysallocunits
syshobts
syshobtcolumns
sysrowsets
sysrowsetcolumns

More on allocation units, hobts, and rowsets later in the week - for now you can assume they serve the same function as the three critical system tables in SQL Server 2000. You can't see these system tables because they're 'hidden' - the parser won't allow them to be bound to in a query. Try running 'select * from sysallocunits' to see what I mean.

The primitive checks are designed to check that internal queries on the metadata tables won't throw errors. Each of the critical system tables has a clustered index. The primitive checks just check the leaf-level data pages of the clustered indexes. For every one of these pages, the following is done:

Read and latch the page (a latch is a lightweight internal version of a lock). This makes sure that there aren't any IO problems with the page such as a torn-page or bad page checksum and ensures that we can put the page in the buffer pool correctly. This is the most common cause of failure of the primitive system table checks and results in error 8966, which in SQL Server 2000 could look something like:

Server: Msg 8966, Level 16, State 1, Line 1
Could not read and latch page (1:33245) with latch type SH. sysobjects failed.

Audit the page. This is a series of checks of the page structures which I'll cover in a seperate post. If these pass, the page looks like a SQL Server page of type its supposed to be.
Check the basic page linkage. Pages in each level of a clustered index are linked together in a doubly-linked list to allow range scans to work. At this stage we only check the left-to-right linkage to ensure the linked-to page actually exists.
Check the page linkage for loops. This is simple to do - have two pointers into the page linked-list with one advancing at every step and one advancing at every second step. If they ever point to the same thing before the faster-advancing pointer reaches the right-hand side of the leaf level then there's a loop. Its important that there are no linkage loops otherwise a range scan may turn into an infinite loop. I've never seen this occur in the field.

Any error found at this stage cannot be repaired so you must restore from a backup. This is because the repair would have to deallocate the page, effectively deleting metadata for a whole lot of tables and indexes. As people's databases get larger and more complex (thousands of tables and indexes), the percentage of pages that comprise these critical system tables rises and so the chance of a hardware problem corrupting one of these pages also rises - I see several of these a month on the forums. Without a backup, the only alternative is to try to export as much data as you can - not good.

If all the pages are ok then we know we've got solid enough metadata on which to base the next set of checks.

2. Allocation checks
(Part 2...)

3. Logical checks of critical system tables
4. Logical checks of all tables
(Part 3...)

5. Service Broker checks
6. Metadata cross-checks
7. Indexed view and XML index checks
(Part 4...)

CHECKDB (Part 1): How does CHECKDB get a consistent view of the database?

Paul Randal - MSFT — Fri, 09 Jun 2006 01:44:00 GMT

As you can guess from the title, I'm planning a long series over the summer to go into the guts of how CHECKDB works (both the consistency checks part and the repair part). And as you can guess from 'CHECKDB', I'm already bored with putting DBCC in front of it all the time and changing the font to Courier New to make it stand out. I don't do that in real life so why on the blog?

I have no idea how long it'll be - easily more than 20, probably less than 50, but there's a ton of info in my head that's desperate to get out (ever seen the scene in the movie Scanners where that guy's head explodes? Well, its not quite that bad but remembering that scene was fun). As part of this I'll need to go into some of the low-level structural details of the database, which will hopefully be interesting too. (How do I know all this stuff? Read my bio). If there's something I need to explain, post a comment with a question in and I'll do a post to answer it. I may even do a post on how to use DBCC PAGE...

So here's the problem statement: CHECKDB needs a consistent view of the database.

Why? Well, usually its running on a live database with all kinds of stuff going on. It needs to read and analyze the whole database but it can't do it instantaneously (this isn't Star Trek) so it has to take steps to ensure that what it reads is transactionally consistent.

Here's an example. Consider a transaction to insert a record into a table that is a heap and has a non-clustered index, with a concurrent CHECKDB that doesn't enforce a consistent view.

The table record is inserted first, and then the non-clustered index record is inserted (that's just the way the operations are split up in the database engine)
Because this hypothetical CHECKDB doesn't have a consistent view, it could read see the record in the table but not that in the index, conclude that the non-clustered index is out of sync with the table and flag an 8951 error (missing index row).
How could this happen? Depending on the order in which the pages are read, the page on which the new non-clustered index record should go could be read before the page on which the new heap record should go. (I use record and row somewhat interchangeably to mean the physically-stored contents of a table or index row). If the index page read happens just before the record is inserted into the table page, and then the table page is read, then we see the inconsistent state.

The easy way to get the consistent state is through locking, which is what SQL Server 7.0 did. You can still do that in SQL Server 2000 and 2005 using the TABLOCK option. Another way to do it is to put the database into single-user or read-only mode.

However, excessive locking is a drag and taking the database essentially offline tends to irritate users so with SQL Server 2000 we came up with a neat way to get the consistent view and be able to run CHECKDB online - log analysis. In a nutshell, after we've read through all the database, we read the transaction log to make sure we didn't miss anything. Sounds simple, right? Dream on. Here's how it works:

The log is read from the LSN of the 'begin tran' log record of the oldest transaction that is active at the time the database scan started, to the LSN at the time the database scan stops.
Log records from transactions that commit during that time are used to generate REDO facts. (We have a chicken-and-egg situation here - it's difficult to explain how CHECKDB works without referencing some mechanisms that haven't been explained yet - I'll get to what facts are in part <single-digits-I-promise>.) The REDO facts either reinforce something we've already seen (in which case we ignore them) or provide information on something we haven't seen. For example:

a page allocation log record would produce a REDO fact of 'page X is allocated to IAM chain Y' (yes, I'm throwing around unexplained terms again - unavoidable I'm afraid and I'll explain them later - or read Kalen's books)
a row insertion record (such as from the index example above) would produce a REDO fact of 'a row with these index keys was inserted into page A of table B, index C at slot position S'

Log records from transactions that rollback or don't commit during that time are used to generate UNDO facts. The UNDO facts either cancel something that we've already seen (e.g. the first half of the index example above, if it didn't commit while CHECKDB was doing the database scan) or reference something we haven't seen (in which case we ignore them). For example:

page allocation log record would produce an UNDO fact of 'page X was deallocated from IAM chain Y'
a row insert record would produce an UNDO fact of 'a row with these index keys was removed from page A of table B, index C at slot position S'

As you may have realized, what we're essentially doing is our own log recovery, inside CHECKDB, but without actually affecting the database.
This can get excruciatingly complicated (e.g. having to generate UNDO facts from the compensation log records that wrap sections of a cancelled index rebuild transaction...) I spent too many days of 2000 working out what was going on in the log and making tweaks to this code. However, it worked really well and we had online CHECKDB finally. The kudos for writing most of this stuff goes to Steve Lindell - while he was busy writing the online code I was up to my eyes writing DBCC INDEXDEFRAG (another post).

Back in late 2000, it became apparent that with all the new features we were planning for 'Yukon' (we had no idea it would be called SQL Server 2005 back then), including some changes to the transaction log to allow for fast recovery and deferred transactions and stuff like versioning and online index build, the transaction log analysis was a non-starter. While it had given us the holy-grail of online consistency checks, with all the added complications of Yukon it would become impossible to maintain and get right.

But what to use instead? Who would come to my rescue? Turns out that database snapshots would be my savior. (That is their eventual name. I preferred their first name COW databases - Copy-On-Write databases - and my TechEd slide deck has a nice animated cow in homage). The in-depth details of database snapshots are beyond the scope of this post. To put it simply:

they use NTFS sparse-file technology
database recovery is run when the snapshot is created, but the recovered database is stored in the snapshot, not the source database
they only hold pages from the source database that have been changed since the database snapshot was created (either by the recovery process, or as part of normal operations on the source database)
Books Online has a bunch more info about their use by DBCC - look in the 'DBCC Statements' section.

By moving to database snapshots we changed to using mainline server code to get our transactionally consistent view. This vastly reduced the complexity of the code and meant that someone else was responsible for finding and fixing its bugs during development :-)

So when CHECKDB starts, the first thing we do is work out whether we'd like to run online - if so we create a hidden database snapshot of the source database (i.e. CHECKDB's target database). That could cause you a problem - depending on your transaction load concurrent with the CHECKDB, the database snapshot can grow in size. As the we create a hidden one, you have no control over where we places the files - we just place them as alternate streams of the files comprising the source database. If you don't have room for this, just create your own database snapshot and check that.

Once the database snapshot is created, we're guaranteed a transactionally consistent view of the database and can merrily run our check algorithms against the database snapshot. Ah, you might say, but that means CHECKDB is checking the database as it was at some point in the past! Yes, I'd say, but that point is the start time of the CHECKDB, just as it was (essentially) with the log analysis mechanism in SQL Server 2000.

There are a few slight gotchas (all documented) with this approach:

sparse files are only available with NTFS so online checks can't be run on databases stored on FAT or FAT32 volumes
recovery cannot be run on TEMPDB, so online checks can't be run on TEMPDB (CHECKDB automatically switches to locking in that case)

And that's that. Now its time for breakfast - no oatmeal hopefully...

SQL Server Storage Engine : CHECKDB Series

How long does *your* CHECKDB take?

CHECKDB (Part 8): Can repair fix everything?

CHECKDB (Part 7): How long will CHECKDB take to run?

CHECKDB (Part 6): Consistency checking options for a VLDB

CHECKDB (Part 5): What does CHECKDB really do? (part 4 of 4)

CHECKDB (Part 4): What does CHECKDB really do? (part 3 of 4)

CHECKDB (Part 3): What does CHECKDB really do? (2 of 4)

CHECKDB (Part 2): What does CHECKDB really do? (1 of 4)

CHECKDB (Part 1): How does CHECKDB get a consistent view of the database?

How long does your CHECKDB take?