SQL Server Storage Engine : On-Disk Structures

CHECKDB (Part 8): Can repair fix everything?

Paul Randal - MSFT — Sun, 04 Feb 2007 19:53:00 GMT

I was teaching at a Microsoft-internal class last week and there was a discussion on what corruptions can't be repaired using DBCC. At the same time, several threads popped up on forums and newsgroups with people hitting some of this unrepairable corruptions so I thought that would make a good topic for the next post in the CHECKDB series.

Before anyone takes this the wrong way - what do I mean by "can't be repaired"? Remember that that purpose of repair is to make the database structurally consistent, and that to do this usually means deleting the corrupt data/structure (that's why the option to do this was aptly named REPAIR_ALLOW_DATA_LOSS - see this post for more explanation on why repair can be bad). A corruption is deemed unrepairable when it doesn't make sense to repair it given the damage the repair would cause, or the corruption is so rare and so complicated to repair correctly that it's not worth the engineering effort to provide a repair. Remember also that recovery from corruptions should be based on a sound backup strategy, not on running repair, so making this trade-off in functionality makes perfect sense.

Here's a few of the more common unrepairable corruptions that people run into along with the reasons they can't be repaired by DBCC.

PFS page header corruption

An example of this is on SQL Server 2005:

Msg 8946, Level 16, State 12, Line 1
Table error: Allocation page (1:13280496) has invalid PFS_PAGE page header values.
Type is 0. Check type, alloc unit ID and page ID on the page.

CHECKDB uses the PFS pages to determine which pages are allocated - and so which pages to read to drive the various consistency checks. The only repair for a PFS page is to reconsutruct it - they can't simply be deleted as they're a fixed part of the fabric of the database. PFS pages cannot be rebuilt because there is no infallible way to determine which pages are allocated or not. There are various algorithms I've experimented with to rebuild them with optimistic or pessimistic setting of page allocation statuses and then re-run the various consisteny checks to try to sort out the incorrect choices, but they all require very long run-times. Given the frequency with which we see these corruptions, and the engineering effort required to come up with an (imperfect) solution, I made the choice to leave this as unrepairable.

Critical system table clustered-index leaf-page corruption

An example of this is on SQL Server 2000:

Server: Msg 8966, Level 16, State 1, Line 1
Could not read and latch page (1:18645) with latch type SH. sysindexes failed.

And on SQL Server 2005:

Msg 7985, Level 16, State 2, Server SUNART, Line 1
System table pre-checks: Object ID 4. Could not read and latch page (1:51) with
latch type SH. Check statement terminated due to unrepairable error.

In a previous post I described why how and why we do special checks of the clustered indexes of the critical system tables. If any of the pages at the leaf-level of these indexes are corrupt, we cannot repair them. Repairing would mean deallocating the page, wiping out the most important metadata for potentially hundreds of user tables and so effectively deleteing all of these tables. That's obviously an unpalatable repair for anyone to allow and so we don't do it.

Column value corruption

Here's an example of this on SQL Server 2005:

Msg 2570, Level 16, State 3, Line 1
Page (1:152), slot 0 in object ID 2073058421, index ID 0, partition ID 72057594038321152, alloc unit ID 72057594042318848 (type "In-row data"). Column "c1" value is out of range for data type "datetime". Update column to a legal value.

This is where a column has a stored value that is outside the valid range for the column type. There are a couple of repairs we could do for this:

delete the entire record
insert a dummy value

#1 isn't very palatable because then we've lost data and its not a structural problem in the database so doesn't have to be repaired. #2 is dangerous - what value should we choose as the dummy value? Any value we put in may adversely affect business logic, or fire a trigger, or have some unwelcome meaning in the context of the table - even a NULL. Given these problems, we chose to allow people to fix the corrupt values themselves.

Metadata corruption

Here's an example of this on SQL Server 2005:

Msg 3854, Level 16, State 1, Line 2
Attribute (referenced_major_id=2089058478) of row (class=0,object_id=2105058535,column_id=0,referenced_major_id=2089058478,referenced_minor_id=0) in sys.sql_dependencies has a matching row (object_id=2089058478) in sys.objects (type=SN) that is invalid.

This example is relatively benign. There are other examples that will cause CHECKDB to terminate - not as bad as the critical system table corruption example above, but enough that CHECKDB doesn't trust the metadata enough to use it to drive consistency checks. Repairing metadata corruption has the same problems as repairing critical system table corruption - any repair means deleting metadata about one or more tables, and hence deleting the tables themselves. It's far better to leave the corruption unrepaired so that as much data as possible can be extracted from the remaining tables.

Summary

Repair can't fix everything. You may end up having to perform manual and time-consuming data extraction from the corrupt database and losing lots of data because of, say, a critical system table corruption. Bottom line (as usual) - make sure you have valid backups so you don't get into this state!

More undocumented fun: DBCC IND, DBCC PAGE, and off-row columns

Paul Randal - MSFT — Thu, 14 Dec 2006 01:46:00 GMT

(Final blog post of the year for me. Its been a bit of a wild ride the last 6 months - 7 TechEds on 3 continents, 46 blog posts and some major life changes - but now things have calmed down and I should be back to more regular posting in 2007. Tomorrow I fly out to Wakatobi in Indonesia to go diving with my partner until January, which is some long overdue R&R. I'll post a link to some photos in January - she's already been out there for two weeks on a live-aboard and has over 3000 photos... In the meantime, I have all my loose-ends tied up at work so time to squeeze in a post. Thanks for all your comments and emails this year - I hope you all have a great holiday and may your pagers be silent throughout!)

At TechEd in November, in one of the sessions I did with Kimberly we demo'd both DBCC IND and DBCC PAGE in the context of following links to columns that have been pushed off-row when the row size exceeds 8060 bytes. The point of the demo is to show that even if an index is perfectly defragmented, the performance of a query that does a range scan may suffer if off-row columns are part of the result set because getting to those off-row columns involves random IOs.

What I'd like to do in this post is run through the demo we did, and at the same time introduce you to the undocumented DBCC IND command. This is like DBCC PAGE - it's used extensively internally but isn't documented or supported - use at your own risk. You already know about using DBCC PAGE to investigate page contents from my previous posts (here and here) so I'm not going to go into details.

To start off, we create a table with a schema that can cause rows to be greater than 8060 bytes. In previous versions, creating a schema like this has always been possible, but actually getting rows larger then 8060 was not. Checkout my previous post on IAM chains and allocation units in SQL Server 2005 for more details of large rows (and a few other cool new features in 2005). I'll assume that you've got a database called dbccpagetest that you're using.

CREATE TABLE rowoverflowtest (c1 INT, c2 VARCHAR (8000), c3 VARCHAR (8000));

CREATE CLUSTERED INDEX row_cl ON rowoverflowtest (c1);

Now we need to populate the table so we have something to look at.

INSERT INTO rowoverflowtest VALUES (1, REPLICATE ('a', 100), REPLICATE ('b', 100));

INSERT INTO rowoverflowtest VALUES (2, REPLICATE ('a', 100), REPLICATE ('b', 100));

INSERT INTO rowoverflowtest VALUES (3, REPLICATE ('a', 100), REPLICATE ('b', 100));

INSERT INTO rowoverflowtest VALUES (4, REPLICATE ('a', 100), REPLICATE ('b', 100));

INSERT INTO rowoverflowtest VALUES (5, REPLICATE ('a', 100), REPLICATE ('b', 100));

And now the new command - use DBCC IND to find out which page IDs to look at with DBCC PAGE.

DBCC IND ('dbccpagetest', 'rowoverflowtest', 1);

The output is (prettified in Excel):

PageFID	PagePID	IAMFID	IAMPID	ObjectID	IndexID	PartitionNumber	PartitionID	iam_chain_type	PageType	IndexLevel
1	156	NULL	NULL	133575514	1	1	72057594039959552	In-row data	10	NULL
1	155	1	156	133575514	1	1	72057594039959552	In-row data	1	0

(I've stripped off the 4 trailing columns, NextPageFID, NextPagePID, PrevPageFID, PrevPagePID so it all fits in the window. They're all zero.)

The columns mean:

PageFID - the file ID of the page
PagePID - the page number in the file
IAMFID - the file ID of the IAM page that maps this page (this will be NULL for IAM pages themselves as they're not self-referential)
IAMPID - the page number in the file of the IAM page that maps this page
ObjectID - the ID of the object this page is part of
IndexID - the ID of the index this page is part of
PartitionNumber - the partition number (as defined by the partitioning scheme for the index) of the partition this page is part of
PartitionID - the internal ID of the partition this page is part of
iam_chain_type - see IAM chains and allocation units in SQL Server 2005
PageType - the page type. Some common ones are:

1 - data page
2 - index page
3 and 4 - text pages
8 - GAM page
9 - SGAM page
10 - IAM page
11 - PFS page

IndexLevel - what level the page is at in the index (if at all). Remember that index levels go from 0 at the leaf to N at the root page (except in clustered indexes in SQL Server 2000 and 7.0 - where there's a 0 at the leaf level (data pages) and a 0 at the next level up (first level of index pages))
NextPageFID and NextPagePID - the page ID of the next page in the doubly-linked list of pages at this level of the index
PrevPageFID and PrevPagePID - the page ID of the previous page in the doubly-linked list of pages at this level of the index

So you can see we've got a single page clustered index with an IAM page. Note that the page IDs returned may differ on your server. Let's look at the data page, focusing on the record for c1 = 3.

DBCC TRACEON (3604);

DBCC PAGE (dbccpagetest, 1, 155, 3);

Don't forget that we need to turn on T3604 first to get the output back to the console. The dump for the 3rd row is (remembering that slots are number from zero, so slot 2 is row 3):

Slot 2 Offset 0x216 Length 219

Record Type = PRIMARY_RECORD Record Attributes = NULL_BITMAP VARIABLE_COLUMNS

Memory Dump @0x44DCC216

00000000: 30000800 03000000 04000003 00130077 †0..............w

00000010: 00db0061 61616161 61616161 61616161 †...aaaaaaaaaaaaa

00000020: 61616161 61616161 61616161 61616161 †aaaaaaaaaaaaaaaa

00000030: 61616161 61616161 61616161 61616161 †aaaaaaaaaaaaaaaa

00000040: 61616161 61616161 61616161 61616161 †aaaaaaaaaaaaaaaa

00000050: 61616161 61616161 61616161 61616161 †aaaaaaaaaaaaaaaa

00000060: 61616161 61616161 61616161 61616161 †aaaaaaaaaaaaaaaa

00000070: 61616161 61616162 62626262 62626262 †aaaaaaabbbbbbbbb

00000080: 62626262 62626262 62626262 62626262 †bbbbbbbbbbbbbbbb

00000090: 62626262 62626262 62626262 62626262 †bbbbbbbbbbbbbbbb

000000A0: 62626262 62626262 62626262 62626262 †bbbbbbbbbbbbbbbb

000000B0: 62626262 62626262 62626262 62626262 †bbbbbbbbbbbbbbbb

000000C0: 62626262 62626262 62626262 62626262 †bbbbbbbbbbbbbbbb

000000D0: 62626262 62626262 626262†††††††††††††bbbbbbbbbbb

UNIQUIFIER = [NULL]

Slot 2 Column 1 Offset 0x4 Length 4

c1 = 3

Slot 2 Column 2 Offset 0x13 Length 100

c2 = aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa

Slot 2 Column 3 Offset 0x77 Length 100

c3 = bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb

You can see that all the data is stored in-row. Now we update one of the columns so that the row size exceeds 8060 bytes.

UPDATE rowoverflowtest SET c3 = REPLICATE ('c', 8000) WHERE c1 = 3;

And run DBCC IND again to see if anything changed. The output is:

PageFID	PagePID	IAMFID	IAMPID	ObjectID	IndexID	PartitionNumber	PartitionID	iam_chain_type	PageType	IndexLevel
1	156	NULL	NULL	133575514	1	1	72057594039959552	In-row data	10	NULL
1	155	1	156	133575514	1	1	72057594039959552	In-row data	1	0
1	158	NULL	NULL	133575514	1	1	72057594039959552	Row-overflow data	10	NULL
1	157	1	158	133575514	1	1	72057594039959552	Row-overflow data	3	0

(Again, I've stripped off the trailing 4 columns - they're all zero)

Cool - we have another IAM page and a text page, in a row-overflow IAM chain. So something must have been pushed off-row into page (1:157). Let's have another look at the clustered-index data page, focusing on the 3rd row again:

Slot 2 Offset 0x4a7 Length 143

Record Type = PRIMARY_RECORD Record Attributes = NULL_BITMAP VARIABLE_COLUMNS

Memory Dump @0x44DCC4A7

00000000: 30000800 03000000 04000003 00130077 †0..............w

00000010: 008f8061 61616161 61616161 61616161 †...aaaaaaaaaaaaa

00000020: 61616161 61616161 61616161 61616161 †aaaaaaaaaaaaaaaa

00000030: 61616161 61616161 61616161 61616161 †aaaaaaaaaaaaaaaa

00000040: 61616161 61616161 61616161 61616161 †aaaaaaaaaaaaaaaa

00000050: 61616161 61616161 61616161 61616161 †aaaaaaaaaaaaaaaa

00000060: 61616161 61616161 61616161 61616161 †aaaaaaaaaaaaaaaa

00000070: 61616161 61616102 00004401 000000a6 †aaaaaaa...D.....

00000080: 2e000040 1f00009d 00000001 000000††††...@...........

UNIQUIFIER = [NULL]

Slot 2 Column 1 Offset 0x4 Length 4

c1 = 3

Slot 2 Column 2 Offset 0x13 Length 100

c2 = aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa

c3 = [BLOB Inline Root] Slot 2 Column 3 Offset 0x77 Length 24

Level = 0 Unused = 68 UpdateSeq = 1

TimeStamp = 782630912

Link 0

Size = 8000 RowId = (1:157:0)

The 2nd varchar column is no longer stored on the page - instead the value have been replaced with an in-row blob root, pointing to an 8000 byte long blob fragment stored in slot 0 of page (1:157). If we dump it with DBCC PAGE we should see its got our value in:

DBCC PAGE (dbccpagetest, 1, 157, 3);

And the output is (slightly curtailed):

PAGE: (1:157)

BUFFER:

BUF @0x02BE8D84

bpage = 0x03FB6000 bhash = 0x00000000 bpageno = (1:157)

bdbid = 9 breferences = 0 bUse1 = 8105

bstat = 0xc0010b blog = 0x1432159b bnext = 0x00000000

PAGE HEADER:

Page @0x03FB6000

m_pageId = (1:157) m_headerVersion = 1 m_type = 3

m_typeFlagBits = 0x0 m_level = 0 m_flagBits = 0x8000

m_objId (AllocUnitId.idObj) = 117 m_indexId (AllocUnitId.idInd) = 256

Metadata: AllocUnitId = 72057594045595648

Metadata: PartitionId = 72057594039959552 Metadata: IndexId = 1

Metadata: ObjectId = 133575514 m_prevPage = (0:0) m_nextPage = (0:0)

pminlen = 0 m_slotCnt = 1 m_freeCnt = 80

m_freeData = 8110 m_reservedCnt = 0 m_lsn = (21:107:19)

m_xactReserved = 0 m_xdesId = (0:0) m_ghostRecCnt = 0

m_tornBits = 0

Allocation Status

GAM (1:2) = ALLOCATED SGAM (1:3) = ALLOCATED

PFS (1:1) = 0x64 MIXED_EXT ALLOCATED 100_PCT_FULL DIFF (1:6) = CHANGED

ML (1:7) = NOT MIN_LOGGED

Blob row at: Page (1:157) Slot 0 Length: 8014 Type: 3 (DATA)

Blob Id:782630912

44DCC06E: 63636363 63636363 63636363 63636363 cccccccccccccccc

44DCC07E: 63636363 63636363 63636363 63636363 cccccccccccccccc

44DCC08E: 63636363 63636363 63636363 63636363 cccccccccccccccc

44DCDF7E: 63636363 63636363 63636363 63636363 cccccccccccccccc

44DCDF8E: 63636363 63636363 63636363 63636363 cccccccccccccccc

44DCDF9E: 63636363 63636363 63636363 63636363 cccccccccccccccc

DBCC execution completed. If DBCC printed error messages, contact your system administrator.

Yep - that's our updated column.

Now, this is a contrived example to force columns off-row, but you can imagine a case where a column is unexpectedly pushed off-row in some rows of a table and suddenly a query slows down. You look at the fragmentation and the query plan and everything looks good but don't realize that you're taking random IOs to return the columns you've asked for.

That's all - now it's time for me to go and pack my diving gear and head off towards the sun (20 1/2 hours of flights over 31 1/2 hours to get to Bali tomorrow - ugh!). Have fun playing with DBCC over the holidays and I'll be back in January.

When can allocation order scans be used?

Paul Randal - MSFT — Thu, 09 Nov 2006 05:31:00 GMT

I know this is jumping the gun a little as I haven't made it this far in my series on fragmentation, but this came up in a chalk-talk I did yesterday at TechEd Developers in Barcelona and is worth blogging about.

You'd expect a select * from mytable query on a table with a clustered index to use an allocation order scan to return the data, as that's the fastest way to read all the pages at the leaf-level of an index. In fact, the query plan will show an unordered clustered index scan. Well, guess again. What I didn't remember, and thanks to SQL Server MVP Maciej Pilecki for pointing this out during my talk (even though I didn't believe him at first), is that an allocation order scan can't be used when there's any possibility of the data changing beneath the scan.

Consider an example where an allocation order scan is progressing and a page that the scan has already read then splits - adding a newly allocated page at the end of the allocation-order list of pages. The scan will eventually come to the new page and then re-read some of the rows it has already read - producing duplicates in the scan output - clearly undesirable.

The only time such a scan will be used is when there's no possibility of the data changing (e.g. when the TABLOCK hint is specified, or when the table is in a read-only database) or when its explicitly stated that we don't care (e.g. when the NOLOCK hint is specifed or under READ UNCOMMITTED isolation level). As a further twist, there's a trade-off with setup cost of the allocation order scan against the number of pages that will b read - an allocation order scan will only be used if there's more than 64 pages to be read.

Below I've included a simple script that demonstrates the behavior. For me, the funny thing is that I should have remembered this behavior as it's exactly the same reason why DBCC SHOWCONTIG takes a shared table lock when in the default mode - it's using an allocation order scan and needs to ensure that the data doesn't change and produce duplicates. Oh well - you (re)learn something every day!

-- Drop and recreate the test database

USE master;

DROP DATABASE allocationordertest;

CREATE DATABASE allocationordertest;

USE allocationordertest;

-- Create a simple table that we can fill up quickly, plus a clustered index

CREATE TABLE t1 (c1 INT, c2 VARCHAR (8000));

CREATE CLUSTERED INDEX t1c1 ON t1 (c1);

-- Add some rows, making sure to produce an out-of-order dataset when scanned in allocation order.

DECLARE @a INT;

SELECT @a = 10;

WHILE (@a < 100)

BEGIN

INSERT INTO t1 VALUES (@a, replicate ('a', 5000))

SELECT @a = @a + 1

END;

SELECT @a = 1;

WHILE (@a < 10)

BEGIN

INSERT INTO t1 VALUES (@a, replicate ('a', 5000))

SELECT @a = @a + 1

END;

-- Now try to do an allocation order scan - doesn't work...

SELECT * FROM t1;

-- Until we add the right conditions.

SELECT * FROM t1 WITH (TABLOCK);

Fragmentation (part 4): what are heaps?

Paul Randal - MSFT — Tue, 19 Sep 2006 04:18:00 GMT

Ok - really catching up with the various blog post series I started back in June/July - time to bang out the next few posts in the series on index fragmentation. Remember I'm starting from first principles and covering

What are records?
What are pages?
What are extents?
What is a heap?
What is a clustered index?
What is a non-clustered index?

before getting into the details of fragmentation.

So what is a heap?

A heap is the simplest storage arrangement and is an unordered table (or you could think of it as a collection of unordered pages).
This means that rows will be inserted into the heap anywhere where there is space (don't let this statement confuse you - for a table that has nothing but inserts, records will always be appended to the last allocated page).
A heap consists entirely of data pages - as there is no b-tree, there are no index pages.
As a heap is unordered, the data pages are not linked in any way. (Ah - there are some exceptions. In SQL Server 2000 and 2005 (and maybe in 7.0 - I don't remember), the sysfiles1 table is a linked-heap. This table contains the locations of the files comprising the database and so for safety's sake we want the pages to be linked, but not with the complexity of having a b-tree because we read the pages directly at startup time. In SQL Server 2005, some other system tables are also linked-heaps).
If you want to do a singleton lookup from a heap, the whole heap has to be scanned to find the row(s) matching the search predicate.
Doing a select (*) of the contents of a heap will not guarantee to return the rows in the order they were inserted (as it did in 6.5 when all heaps were linked as sysfiles1 is and the space from singleton deletes was not reclaimed). This is because the pages are accessed in allocation order.
Non-clustered index records contain the physical RID (record ID or record locator) of the corresponding heap record.

Heaps have one interesting feature - forwarded records. If a record needs to be updated, the updated record size is greater than the current record size, and there is no space on the page to fit the new record in then we have two choices:

move the record to a new page and change all the non-clustered index records that point to it to point to the new location of the record
move the record to a new page and leave a forwarding record in the original location to point to the new location

Guess which approach we took? Right, #2. Approach #1 has enormous performance implications because of the extra IOs and logging involved. The forwarding record has the physical location of the new record and the forwarded record has a back-pointer to the forwarding record (so that if its deleted, the forwarding record can be deleted too).

This is one drawback of using heaps - all the extra space that's "wasted" with the forwarding/forwarded records. Another drawback is that when scanning through the heap, forwarding records have to followed immediately (as opposed to ignoring them and just reading the forwarded records when they're encountered). This is to vastly reduce the possiblity of read anomalies (such as non-repeatable reads or missed rows if a row moves before the scan point during a scan).

And that's it. Next up is clustered indexes...

Don't try this at home kids... (data recovery using DBCC PAGE)

Paul Randal - MSFT — Wed, 09 Aug 2006 22:41:00 GMT

Time to cough up a real-life data recovery scenario to temporarily stop those people who badger me relentlessly for scenarios...

We have a situation currently with a customer who managed to delete all their data during a SAN reconfig and then found out that their tape backups hadn't been working properly (the same sad story I've heard a thousand times). They did have a primary filegroup backup plus a backup of a secondary filegroup from a different point. This is on SQL Server 2005.

The backups are restored, but the storage engine metadata for the table with the critical data in is messed up - sys.allocunits somehow has the wrong allocation unit IDs, first page, and root pages. So, what the hell can you do?

Easy (and fun if you're twisted like me).

We need to find the following info for the data allocation unit:

the page ID of the head of the IAM chain for the allocation unit
the root page of the clustered index
the first page of the clustered index

And there's a twist, the table has a column of type image so we also need to find the allocation unit ID and page ID of the head of the IAM chain for the LOB allocation unit.

The easiest way to do this is to scan through the data files, doing DBCC PAGEs of every page and storing the results in a table for later analysis. How? Ah, there's something I haven't told you about DBCC PAGE - it has a WITH TABLERESULTS option.

So, setup a simple script to INSERT/EXEC a DBCC PAGE of every page with option 0 (for speed, otherwise it has to take the time to go through and crack and format every record on every page) and save the results to a table.

To find the root page of the clustered index, query the table for the page with m_type = 2 (an index page) and the highest m_level. The level number increases from the leaf of the b-tree to the root page, so the page with the highest level is the root.

To find the first page, crack open the first record on the root page, and extract the page ID of the child page that the index record points to. This is the left-hand edge of the next level lower in the b-tree. Continue doing this until you find the page with m_level = 0. (Note that on SQL Server 2000, you'll need to look for m_level = 0 AND m_type = 1 - this is because pages at the leaf level and the level above in a SQL Server 2000 clustered index all have m_level = 0).

To find the head of the IAM chain is a little trickier. You need to find any page with m_type = 10 and following the m_prevPage links until its NULL. Then do an option 3 dump of the page and ensure that the sequenceNumber in the IAM page header is 0 (remember that an IAM chain is ordered by the sequence numbers in the IAM pages).

Cool - we've done the clustered index.

Now for the LOB allocation unit. Crack any row at the leaf level of the clustered index. You'll see that one of the columns is a text pointer, containing a page ID and slot number of the text fragment in the LOB allocation unit. Do a DBCC PAGE of that page and now you know the allocation unit ID of the LOB allocation unit and can use the methodology above to find the head of its IAM chain.

Now all you have to do is alter the hidden and unbindable sys.allocunits table to contain the correct data... Who's buying the beer? :-)

Poking about with DBCC PAGE (Part 1 of ?)

Paul Randal - MSFT — Wed, 09 Aug 2006 04:52:00 GMT

(It's been a while since I last posted - summer fun's been in the way obviously and I've been busy picking up some new and exciting (for me) SQL skills, partially to help with four upcoming TechEds I'll be doing in China and Hong Kong. The posting frequency should increase towards the end of the week but in the meantime, this one's a special request.)

I've been getting lots of requests for more info about how to use DBCC PAGE since my Paul-tells-all post a while back and I had been planning to wait until I'd posted more about the internals of DBCC CHECKDB and moved on to various examples of corruptions before starting to use DBCC PAGE again in the blog. However, a friend made me an offer I couldn't refuse and so I've caved in and I'm starting the series now :-)

I'm going to start with cracking a record from scratch. Yes, I know that option 3 of DBCC PAGE will do it for you - but how does it do it? Sometimes you may not have table metadata and so DBCC PAGE won't be able to use option 3 - in that case you'll need to crack it yourself. Also, there are plenty of curious people who like to be immersed in the bits and bytes and besides, cracking records manually is fun (if you're twisted as well as curious like I am :-)

I've created an example database, dbccpagetest, on SQL Server 2005 which I've zipped up and attached to this post. This way you can all use the same data and pages as me without having to go through the tedium of working out where pages are using the PFS pages (as I explained in the initial DBCC PAGE post).

You won't be able to attach it to SQL Server 2000 but you can always download SQL Server 2005 Express Edition and play with it on that. Unzip the files and attached them using the CREATE DATABASE ... FOR ATTACH syntax:

CREATE DATABASE dbccpagetest

ON (FILENAME='C:\Program Files\Microsoft SQL Server\MSSQL.1\MSSQL\Data\dbccpagetest.mdf')

FOR ATTACH;

Note that if the filenames haven't changed since they were created, you only need to specify the primary file in the CREATE DATABASE statement. Very cool.

The database has a single table with a couple of rows, created using the following T-SQL:

CREATE TABLE example (

destination VARCHAR(100),

activity VARCHAR(100),

duration INT);

INSERT INTO example VALUES ('Banff', 'sightseeing', 5);

INSERT INTO example VALUES ('Chicago', 'sailing', 4);

The data page is page (1:152). Dumping that page using option 3 will give output including:

Slot 0 Offset 0x60 Length 33

Record Type = PRIMARY_RECORD Record Attributes = NULL_BITMAP VARIABLE_COLUMNS

Memory Dump @0x44DFC060

00000000: 30000800 05000000 0300f802 00160021 †0..............!

00000010: 0042616e 66667369 67687473 6565696e †.Banffsightseein

00000020: 67†††††††††††††††††††††††††††††††††††g

Slot 0 Column 0 Offset 0x11 Length 5

destination = Banff

Slot 0 Column 1 Offset 0x16 Length 11

activity = sightseeing

Slot 0 Column 2 Offset 0x4 Length 4

duration = 5

Previously (that word always makes me think of Twin Peaks - 'Previously in Twin Peaks...', but I digress...) I'd posted about the structure of a record - here it is again:

record header
- 4 bytes long
- two bytes of record metadata (record type)
- two bytes pointing forward in the record to the NULL bitmap
fixed length portion of the record, containing the columns storing data types that have fixed lengths (e.g. bigint, char(10), datetime)
NULL bitmap
- two bytes for count of columns in the record
- variable number of bytes to store one bit per column in the record, regardless of whether the column is nullable or not
- this allows an optimization when reading columns that are NULL (see here for more info)
variable-length column offset array
- two bytes for the count of variable-length columns
- two bytes per variable length column, giving the offset to the start of the column valu
versioning tag
- this is in SQL Server 2005 only
- this is a 14-byte structure that contains a timestamp plus a pointer into the version store in tempdb

So, let's go through the record from the dump fragment above and figure out what everything means. I'm not going to give all the excruciating detail on every possible value of every byte, but will give you enough to be able to crack records yourself (if you're twisted like me...)

Byte 0 is the TagA byte of the record metadata. Its 0x30, which corresponds to 0x10 (bit 4) and 0x20 (bit 5). Bit 4 means the record has a NULL bitmap and bit 5 means the record has variable length columns. If 0x40 (bit 6) was also set, that would indicate that the record has a versioning tag. If 0x80 (bit 7) was also set, that would indicate that byte 1 has a value in it.

Bits 1-3 of byte 0 give the record type. The possible values are:

0 = primary record
- a data record in a heap that hasn't been forwarded or a data record at the leaf level of a clustered index.
1 = forwarded record
- a data record in a heap that's been updated and was too large to fit in-place on its original page and so has been moved to another page. A forwarding record is left in its place and points to the new location of the record. This is done to avoid having to update any non-clustered index records that point back directly to the original physical location of the record.
2 = forwarding record
- sometimes also called a forwarding stub.
3 = index record
- an index record in the tree of a clustered index, or in any level of a non-clustered index.
4 = blob fragment
5 = ghost index record
6 = ghost data record
- these two are self-explanatory
7 = ghost version record
- a special 15-byte record containing a single byte record header plus a 14-byte versioning tag that is used in some circumstances (like ghosting a versioned blob record)

In our example, none of these bits are set which means the record is a primary record. If the record was an index record, byte 0 would have the value 0x36. Remember that the record type starts on bit 1, not bit 0, and so the record type value from the enumeration above needs to be shifted left a bit (multiplied by two) to get its value in the byte.

Byte 1 is the TagB byte of the record metadata. It can either be 0x00 or 0x01. If it is 0x01, that means the record type is ghost forwarded record. In this case its 0x00, which is what we expect given the TagA byte value.

Bytes 2 and 3 are the offset of the NULL bitmap in the record. This is 0x0008 (DBCC PAGE presents multi-byte values in hex dumps as least-significant byte first). This means that there's a 4-byte fixed length portion of the record starting at byte 4. We expect this because we know the table schema.

Bytes 4 to 7 are the fixed length portion. Again, because we know the table schema, we know to interpret these bytes as a 4-byte integer. Without that knowledge, you'd have to guess. The value therefore is 0x00000005, which is what we'd expect to see as the value of the duration column.

Bytes 8 and 9 are the count of columns in the record. This is 0x0003 which is correct. Given that there are only 3 columns, the NULL bitmap of one bit per column will fit in a single byte.

Byte 10 is the NULL bitmap. Hey - it's value is 0xF8 - what's up with that? Convert it to binary and what do you get? 11111000 This makes perfect sense - bits 0-2 represent columns 1-3 and they're all 0, meaning the columns aren't NULL. Bits 3-7 represent non-existent columns and they're set to 1 for clarity. So the 0xF8 value makes sense - phew.

Bytes 11 and 12 are the count of variable length columns in the record. That value is 0x0002, which we again know to be correct. This means there will be two two-byte entries in the variable length column offset array. These will be bytes 13-14 and 15-16, having values of 0x0016 and 0x0021 respectively.

Now, the explanation that the variable length column offset array stores the offsets to the start of the column value is over-simplified. The entries actually point to the start of the following column value - this is done so that we know how long each column is (they are variable length after all). Notice that the offset of the first variable length column value isn't stored - it doesn't need to be because by definition it must begin right after the last offset in the variable length column offset array.

So, the final offset is bytes 15 and 16, which means the offset of the start of the first variable length column must be byte 17 (or 0x11 in hex), which agrees with the DBCC PAGE dump. The offset of the second variable length column is 0x0016, so the first value is from byte 17 to byte 21 inclusive. This value is 0x42616E6666. We know from the table metadata that this is the first varchar column, destination. Checking our handy ASCII conversion table we find that this translates to 'Banff'.

Using similar logic, the second value is from byte 22 to byte 32 inclusive and has the value 'sightseeing'. Both of these match the data we're expecting.

And that's it. We've just cracked a record from scratch. Now you try it with the second row in the table. In future posts I'll start making things more complicated (maybe trace into the version store or add in some off-row LOB values).

Hopefully this will satisfy all of those who've been waiting for more info on using DBCC PAGE and now I'm looking forward to getting my side of the bargain from my friend...

Under the covers: GAM, SGAM, and PFS pages

Paul Randal - MSFT — Sun, 09 Jul 2006 00:22:00 GMT

(Been a week or so since the last post but I haven't burnt out with blogging yet - I was on vacation over the July 4th weekend and totally offline in and around a small town called Pullman in south-eastern Washington.)

In a previous post I described extents, and in another previous post a while back I described how the extents and pages that are allocated to an IAM chain are tracked in IAM pages. What I didn't describe is how the allocation status of individual pages is tracked, or how the global allocation bitmaps work - that's the subject of this post.

This is the last post that lays the groundwork to be able to discuss allocation checks in CHECKDB - the subject of the following post - and various corruption scenarios (yes Kimberly, I'm going to get to scenarios...)

Bear in mind that everything below is exactly the same in SQL Server 2000 and 2005.

GAM pages

GAM stands for Global Allocation Map. If you remember from before, database data files are split up into GAM intervals (don't get confused - they're not split physically, just conceptually). A GAM interval is equivalent to the amount of space that the bitmaps in GAM, SGAM, and IAM pages track - 64000 extents or almost 4GB. These bitmaps are the same size in each of these three page types and have one bit per extent, but they mean different things in each of the different allocation pages.

One thing to note, at the start of every GAM interval is a GAM extent which contains the global allocation pages that track that GAM interval. This GAM extent cannot be used for any regular page allocations.

The bits in the GAM bitmap have the following semantics:

bit = 1: the extent is available for allocation (you could think of it as currently allocated to the GAM page)
bit = 0: the extent is already allocated for use

These semantics are the same for mixed and dedicated/uniform extents.

SGAM pages

I remember last year having an email discussion about what the 'S' stands for in SGAM. Various names have been used over the years inside and outside Microsoft but the official name that Books Online uses is Shared Global Allocation Map. To be honest, we always just call them 'es-gams' and never spell it out.

As I said above, the SGAM bitmap is exactly the same as the GAM bitmap in structure and the interval it covers, but the semantics of the bits are different:

bit = 1: the extent is a mixed extent and has at least one unallocated page available for use
bit = 0: the extent is either dedicated or is a mixed extent with no unallocated pages (essentially the same situation given that the SGAM is used to find mixed extents with unallocated pages)

GAM, SGAM and IAM pages
So, taking the GAM, SGAM and IAM pages together (remember that in the IAM bitmap, the bit is set if the extent is allocated to the IAM chain/allocation unit), the various combinations of bits are:

GAM	SGAM	Any IAM	Comments
0	0	0	Mixed extent with all pages allocated
0	0	1	Dedicated extent (must be allocated to only a single IAM page)
0	1	0	Mixed extent with >= 1 unallocated page
0	1	1	Invalid state
1	0	0	Unallocated extent
1	0	1	Invalid state
1	1	0	Invalid state
1	1	1	Invalid state

You can see that only 4 of the 8 possible bit combinations for any particular extent are valid. Anything else constitutes a corruption of some sort and can lead to all kinds of horrible situations - more on some of these in later posts.

(But I can't resist - 'how do these corruptions happen?' I'm sure someone is asking. Every database page is 8KB, which is really 16 512-byte disk segments. Imagine a flaky IO system writing some random data into one of the disk segments of a GAM page and causing multiple IAM pages to think they have the same extents allocated...)

PFS pages

PFS stands for Page Free Space, but the PFS page tracks much more than that. As well as GAM intervals, every database file is also split (conceptually) into PFS intervals. A PFS interval is 8088 pages, or about 64MB. A PFS page doesn't have a bitmap - it has a byte-map, with one byte for each page in the PFS interval (not including itself).

The bits in each byte are encoded to mean the following:

bits 0-2: how much free space is on the page
- 0x00 is empty
- 0x01 is 1 to 50% full
- 0x02 is 51 to 80% full
- 0x03 is 81 to 95% full
- 0x04 is 96 to 100% full
bit 3 (0x08): is there one or more ghost records on the page?
bit 4 (0x10): is the page an IAM page?
bit 5 (0x20): is the page a mixed-page?
bit 6 (0x40): is the page allocated?

For instance, an IAM page will have a PFS byte value of 0x70 (allocated + IAM page + mixed page). You can examine PFS pages using DBCC PAGE (the instructions in that post use a PFS page as an example).

Free space is only tracked for pages storing LOB values (i.e. text/image in SQL Server 2000, plus varchar(max)/varbinary(max)/XML and row-overflow data in SQL Server 2005) and heap data pages. This is because these are the only pages that store unordered data and so insertions can occur anywhere there's space. For indexes, there's an explicit ordering so there's no choice in the insertion point.

The point at which a PFS byte is reset is not intuitive. Just yesterday I was helping a couple of MVPs with an issue and one of the questions was (paraphrasing) "This page has a PFS byte value of 0x04 - how can it be full when its not allocated?"

The answer is that PFS bytes are not fully reset until the page is reallocated. On deallocation, the only bit in the PFS byte that's changed is the allocation status bit - this makes it very easy to rollback a deallocation.

Here's an example. Using a database with a simple table with one row. A DBCC PAGE of the IAM page includes:

PFS (1:1) = 0x70 IAM_PG MIXED_EXT ALLOCATED 0_PCT_FULL

If I run the following:

BEGIN TRANSACTION

DROP TABLE T1

And then do the DBCC PAGE again, the output now includes:

PFS (1:1) = 0x30 IAM_PG MIXED_EXT 0_PCT_FULL

And if I rollback then transaction, the DBCC PAGE output reverts to:

PFS (1:1) = 0x70 IAM_PG MIXED_EXT ALLOCATED 0_PCT_FULL

Now that these three pages have been discussed, we're free to explore allocation checks and corruptions and I'm free to go and have breakfast!

(This is just adding the blog to Technorati - Technorati Profile)

Why is the NULL bitmap in a record an optimization?

Paul Randal - MSFT — Thu, 29 Jun 2006 04:42:00 GMT

I've had some questions sent in comments and I wanted to reply to some of them using a post so others who aren't subscribed to the comment sections can see the answers.

Wesley Backelant said:

I was wondering what exactly you mean by "this allows an optimization when reading columns that are NULL". Could you elaborate on that?

Firstly, having a null bitmap removes the need for storing special 'NULL' values for fixed-length datatypes. Without the null bitmap, how can you tell whether a column is NULL?

That's easy for a variable-length column - just check the length. If it's zero, then the column is NULL.

[Edit] Ryan Stonecipher (the dev responsible for DBCC) pointed out that I'd forgotten the case of empty strings - thanks Ryan. In this case, an empty string also has a zero length so for varchar columns you'd need to use the fixed-length solution described below.

It's not so easy for fixed-length columns, which, as their name suggests, have a fixed-length so that trick doesn't work. The only solution is to define a special 'NULL' value, which limits the effective range of the datatype being stored.

Secondly, it saves CPU cycles. If there was no NULL bitmap, then there are extra instructions executed for fixed- and variable-length columns.

For fixed-length:

read in the stored column value (possibly taking a cpu data cache miss)
load the pre-defined NULL value for that datatype (possibly taking a cpu data cache miss, but only for the first read in the case of a multiple row select)
do a comparison between the two values

For variable-length:

calculate the offset of the variable length array
read the number of variable length columns (possibly taking a cpu data cache miss)
calculate the position in the variable length offset array to read
read the column offset from it (possibly taking a cpu data cache miss)
read the next one too (possibly taking another cpu data cache miss, if the offset in step 4 was on the boundary of a cache line size)
compare them to see if they're the same

But with a NULL bitmap, all you have to do is:

read the NULL bitmap offset (possibly taking a cpu data cache miss)
calculate the additional offset of the NULL bit you want to read
read it (possibly taking a cpu data cache miss)

So, its about even for a lookup of a single fixed-length column, but for variable-length columns, and for multiple row selects, there's a clear advantage to having the NULL bitmap.

Hope this makes my comment a bit clearer.

Fragmentation (part 3): What are extents?

Paul Randal - MSFT — Wed, 28 Jun 2006 20:52:00 GMT

(Ok - another flight - another blog post. This time its Boston back to Seattle. The three of us who'd come over for the training course upgraded to 1st Class on Alaska for the flight back (great value at $100 for a 6-hour flight) and so there's oodles of room for laptops, newspapers, and long legs)

In the previous posts I explained about database pages - their structure and some page types. Now I'd like to explain how we group pages into units called extents.

An extent is a group of eight physically consecutive pages in a database data file. Extents are always aligned on 64KB boundaries (i.e. 8-page boundaries), starting at the beginning of each data file. Extents, and all their properties, are exactly the same in SQL Server 2000 and 2005.

There are two types of extents: mixed extents and dedicated (or uniform) extents.

Mixed extents

The first 8 pages that are allocated to any IAM chain (either for an index in SQL Server 2000 or allocation unit in SQL Server 2005) are single-page allocations, which we called mixed pages. This means that only a single page is allocated to the IAM chain at a time, rather than a whole extent. The rationale behind this is to allow very small tables to take up the minimum amount of space.

These mixed pages are allocated from mixed extents that are not allocated to any particular IAM chain. The extent is tracked as being allocated globally, so no IAM chain can allocate it.

As the mixed extent is not allocated to any particular IAM chain, this means that it may hold pages allocated to 8 separate IAM chains. IAM pages themselves are always mixed pages, regardless of how many pages have been allocated to an IAM chain. This means a mixed extent may hold a variety of page types too,including IAM, data, index, or text pages.

Two interesting facts:

once an IAM chain has passed the 8-page threshold and switched to allocating dedicated extents, it will never go back to allocating mixed pages again.
for the purposes of fragmentation, we completely ignore mixed pages and extents as there are so few of them in each IAM chain and it complicates the various algorithms involved.

Dedicated/Uniform extents

Once the magic 8-page threshold is passed, all further allocations are from dedicated extents. This means that an extent at a time is allocated to an IAM chain (and marked as such in one of the IAM pages in the IAM chain). This is also tracked globally.

All pages from a dedicated extent must be allocated to the same IAM chain, and they will all be the same type except in the case of clustered indexes, where there could be a mixture of data pages (from the leaf level) and index pages (from the upper b-tree levels).

Two more interesting facts:

just because an extent is allocated to an IAM chain, that doesn't mean that all the pages are. The pages are allocated as needed, so initially only one page will be allocated. There are some exceptions to this rule, including during an offline index build operation, but I'm not going to go into the algorithm details.
dedicated extents can be deallocated from an IAM chain if all the pages in the extent become deallocated.

You may ask "how do you know which pages are allocated in an extent?" and "how do you track the global allocation state of extents, especially mixed extents?". The answers are by using the PFS pages, and by using the GAM + SGAM pages respectively. I'll cover these in the next post.

(Passing over Montana now at 8pm - pretty clear skies and the sun's at an angle to throw the landscape into sharp relief - very cool. First class is great - actual crockery rather than plastic plates, cups and food...)

Under the covers: IAM chains and allocation units in SQL Server 2005

Paul Randal - MSFT — Mon, 26 Jun 2006 09:01:00 GMT

(I'm sitting here in Seattle airport at 7am on Sunday waiting to catch the same flight to Boston that I caught two weeks ago. Instead of TechEd, this time I'm going to a training course at MIT. I'd enjoy the air travel a lot more with a bigger gap in between the flights. At least I got an aisle seat and another 5000 air miles...)

Anyway, time for part two of the discussion of IAM chains.

Yesterday I explained what IAM chains are and how they correspond to indexes in SQL Server 2000. A table can have a heap or clustered index, plus up to 249 non-clustered indexes, plus storage for LOB values (commonly called the text index). This means that in SQL Server 2000, there can be a maximum of 251 IAM chains per table.

In SQL Server 2005, IAM chains and IAM pages are exactly the same, but what they correspond to is different. A table can now have up to 750000 IAM chains! Wow! What on earth did we do?...

There are now three things that IAM chains map space allocations for:

heaps and b-trees (a b-tree is the internal structure used to store an index)
LOB data
row-overflow data

and we now call these units of space allocation tracking allocation units. The internal names for these three types of allocation unit are (respectively):

hobt allocation unit (Heap Or B-Tree, pronounced 'hobbit', yes, as in Lord Of The Rings)
LOB allocation unit
SLOB allocation unit (Small-LOB)

[Edit]And the external names are, respectively:

IN_ROW_DATA allocation unit
LOB_DATA allocation unit
ROW_OVERFLOW_DATA allocation unit

Let's look at three new features in SQL Server 2005 that made these changes are necessary and boosted the number of potential IAM chains per table.

Included Columns
This is the ability for non-clustered indexes to include non-key columns at the leaf-level. This is useful for three reasons:

it allows a non-clustered index to truly cover a query where the query results include more than 16 columns or the combination of column lengths in the query results is greater than 900 bytes (remember that a non-clustered index key is limited to 16 columns and 900 bytes)
it allows columns to be include in the non-clustered index that have data types that cannot be part of an index key (e.g. text or XML)
it allows a non-clustered index to cover a query without having to have all the columns included in the index key. As the index key is included in rows at all levels of the b-tree, this allows the index to be smaller.

An example of space saving: imagine a 100 million row index, with a key length of 900 bytes, but only the first two integer keys are really needed as the index key, the other 4 fixed-length columns could be included.

With the 900 byte key, 8 rows can fit per database page (i.e. the fanout is 8). This means there will be 12500000 pages at the leaf level, 1562500 pages at the next level up in the b-tree and so on, giving a total of 12500000 + 1562500 + 195313 + 24415 + 3052 + 382 + 48 + 6 + 1 = 14285717 pages (including 1785717 to store the upper levels of the b-tree).

If we go with the included columns method then the key size shrinks to 8 bytes, and with the row overhead we can get the row length in the upper levels of the b-tree down to 15 bytes (giving a fanout of approx. 537). Note that the fanout at the leaf-level is still going to be 8, because the amount of data stored in each row at the leaf-level is the same. So, this means there will be 12500000 pages at the leaf level, 23278 pages at the next level up and so on, giving a total of 12500000 + 23278 + 44 + 1 = 12523323 pages (including 23323 to store the upper levels of the b-tree). Compared to the full-size 900-byte key, this is a 12% saving of 1762394 pages, or 13.6GB! Obviously this is a contrived case but you can see how the savings can occur.

Anyway, I digress. The main reason for adding this feature it to enable true covering queries. A covering query is one where the query optimizer knows it can get all the query results from the non-clustered index and so the query can be satisfied without having to incur the extra IOs of looking up data in the base table - a significant performance saving.

Now that non-clustered indexes can have included columns, and those columns can be LOB data types. This means that having a single LOB allocation unit (as in the case of the single text index in SQL Server 2000) isn't possible any more because each index may have its own set of LOBs. Now, you may ask why we didn't just have a single set of LOBs with multiple references from various indexes plus the base table. We considered that but it would have made things a lot more complicated.

So, with this new feature, each index needs two allocation units - one for the data or index rows (the hobt allocation unit) and one for any LOB data.

Large Rows

One of the things that has plagued schema designers for a long time is the 8060 byte limit on table row sizes so we've removed this restriction in SQL Server 2005. The way we've done this is to allow variable-length columns (e.g. varchar, sqlvariant) to get pushed off-row when the row size gets too big to fit on a single page.

But where do these column values get pushed to? We effectively turn them into mini LOB columns. The column value in the row is replaced with a 16-byte pointer to the off-row column value, which is stored as if its a LOB value in a seperate allocation unit - the row-overflow (or SLOB) allocation unit. These values are stored in text pages in exactly the same way as regular LOB values are, just using a seperate allocation unit. The SLOB allocation unit is only created when the first column value is pushed off-row.

This feature works for non-clustered indexes too - if you consider the ability to have included columns in non-clustered indexes then you could easily have non-clustered index rows that won't fit on a page. It would have been short-sighted of us to get rid of the 900-byte limit and replace it with an 8060-byte limit by not extending the row-overflow feature to non-clustered indexes too.

Now with the addition of this new feature, each index can have up to three allocation units - hobt, LOB, and SLOB. Even with this, that only makes a maximum of 750 IAM chains per table (remember an IAM chain now maps the storage allocations for an allocation unit, so 250 indexes * 3 allocation units = 750 IAM chains). But I mentioned 750 thousand IAM chains per table earlier - where do all the rest come from?

Partitioning

This is what gives us the 1000x multiplier. As you may already know, partitioning is the new feature that allows tables and indexes to be split into a series of ranges, with each range stored separately (most commonly in seperate filegroups). Partitioning is a topic for a separate post.

If each range or partition of the table or index is stored seperately, then each is going to need its own hobt allocation unit. Of course, the LOB values associated with each partition need to be stored with it, and so each partition also needs a LOB allocation unit. Also, the row-overflow feature is per-row, and so rows in each partition will overflow into SLOB allocation units just as for un-partitioned tables and indexes. Thus each partition of a table or index can have up to three allocation units (and hence three IAM chains).

Still, where does that 1000 come in? Each table or index can have up to 1000 partitions. This gives us 250 indexes x 1000 partitions x 3 allocation units = 750000 IAM chains. Realistically this probably won't happen, but it is possible.

Now you know some more of how things are organized in SQL Server 2000 and SQL Server 2005 and hopefully this will help understanding some of my future posts.

Ah - here comes the breakfast cart...

Fragmentation (part 2): What are pages?

Paul Randal - MSFT — Mon, 26 Jun 2006 04:56:00 GMT

(Boston continues its reputation - with me at least - for great seafood - calamari and pan-seared fresh halibut this evening. Yum! I was tempted to get on the T - Boston's subway system - and see what's happening downtown but with no jacket I'd get wet. So, instead I'll be boring and geeky and write another blog post - you guys are spoiled today.)

Hopefully I'm getting close to resolving the chicken-and-egg problem of being able to discuss topics without having to promise further posts to explain terms.

In the previous post in this series I explained what records are. Pages exist to store records. A database page is an 8192-byte (8KB) chunk of a database data file. They are aligned on 8KB boundaries within the data files, starting at byte-offset 0 in the file.

A page has a structure as shown below (all pages have the same basic structure):

Page header

The 96-byte page header is the same for every page. You can see an example of the fields that are stored within a page header in the last post on DBCC PAGE. When considering fragmentation, the most important fields are:

Next page pointer
- The pages in each level of the index are joined in a doubly-linked list according to the logical order (as defined by the index keys) of the index. The next page pointer does not necessarily point to the immediately adjacent physical page in the file (fragmentation...
- As these are doubly-linked lists, there is also a previous page pointer.
Page level
- The various algorithms I'll describe later have to know whether the page being examined is from the leaf-level of the index or not (i.e. page level = 0). Page level increases from the leaf-level to the root page at the top of the b-tree (except in clustered indexes in SQL Server 2000 where there are two level=0 levels - the leaf-level and the level above it - this is just for simplicity :-) How do you tell them apart? You need to look at the page type too - data pages are at the leaf-level, index pages in the level above)
Free space count
- This shows how much free space is available in the page. This is used to calculate the average page density - the less free space, the higher the page density.

Other interesting fields in the page header are:

object ID and index ID
- in SQL Server 2000, these correspond to the actual relationl object ID and index ID to which the page is allocated.
- in SQL Server 2005, this mapping no longer exists. The two IDs on each page correspond (through a calculation) to the allocation unit ID of the allocation unit the page is allocated to. A series of metadata lookups are necessary to determine the parent object and index.
page type
- There are a bunch of different page types, most of which I'll be explaining more about in subsequent posts over the next few weeks. Some interesting ones are (well, interesting to me anyway):
  - data page (holds data records)
  - index page (holds non-clustered index leaf records and non-leaf records from clustered and non-clustered indexes)
  - IAM page (holds allocation information about extents within a fixed 4GB GAM interval that are allocated to an index or allocation unit, in SQL Server 2000 and 2005 respectively. IAM = Index Allocation Map.)
  - GAM and SGAM pages (holds global allocation information about extents in a GAM interval. GAM = Global Allocation Map. SGAM = Shared GAM.)
  - PFS page (holds allocation and free space information about pages within a PFS interval - approx 64MB. PFS = Page Free Space.)
  - text page (actually two types that hold leaf and intermediates nodes in text trees)
  - sort page (holds sort records being used in active sort operations)
  - differential bitmap page (holds information about which extents in a GAM interval have changed since the last full or differential backup)
  - bulk-changed map page (holds information about which extents in a GAM interval have changed while in bulklogged mode since the last backup. This is what allows you to switch to bulk-logged mode for bulk-loads and index rebuilds without worrying about breaking a backup chain.)
  - boot page (holds information about the database - one page used per database)
  - fileheader page (holds information about the file - one page per file)
slot count
- This lists the number of occupied record slots on the page (including ghost records).
- It can also be described as giving the number of entries in the slot array.
ghost record count
- self-explanatory
page ID
- a page's ID is made up of the file ID and the page position within the file and is always written as (file:page)

Slot array

It is a very common misconception that records within a page are always stored in logical order. This is not true. There is another misconception that all the free-space in a page is always maintained in one contiguous chunk. This also is not true. (Yes, the image above shows the free space in one chunk and that very often is the case for pages that are being filled gradually.)

If a record is deleted from a page, everything remaining on the page is not suddenly compacted - inserters pay the cost of compaction when its necessary, not deleters.

Consider a completely full page - this means that record deletions cause free space holes within the page. If a new record needs to be inserted onto the page, and one of the holes is big enough to squeeze the record into, why go to the bother of comapcting it? Just stick the record in and carry on.

Ah - but hold on a minute, what if the record should logically have come at the end of all other records on the page, but we've just inserted it in the middle - doesn't that screw things up somewhat?

No, because the slot array is ordered and get reshuffled as records are inserted and deleted from pages. As long as the first slot array entry points to the logically first record on the page, everythings fine. Each slot entry is just a two-byte pointer into the page - so its far more efficient to manipulate the slot array than it is to manipulate a bunch of records on the page. Only when we know there's enough free space contained within the page to fit in a record, but its spread about the page do we compact the records on the page to make the free space into a contiguous chunk.

One interesting fact is that the slot array grows backwards from the end of the page, so the free space is squeezed from the top by new rows, and from the bottom by the slot array.

Records

You know what these are already - we talked about them last time!

Pages are the basic unit of IO that SQL server uses. The buffer pool maps individual pages and torn-page protection and page checksums are implemented per-page. Fragmentation is mostly concerned with pages, and the efficiency of being able to read them in logical order (as defined by their next page pointers).

They can be allocated individually or in extents, which we'll cover next time...

Under the covers: IAM chains in SQL Server 2000

Paul Randal - MSFT — Sat, 24 Jun 2006 17:24:00 GMT

I've reached a point in my various ramblings where I need to do some glossary work. First up is an explanation of IAM chains. This will be in two parts, detailing how they're used in SQL Server 2000 and then in 2005. (Probably tomorrow as the sun's shining and I want to play with my tractor once the time of day - 7.30am now - becomes a little more socially acceptable for a large diesel engine to be running.)

As with so many of the explanations of database structure internals, there's the chicken-and-egg problem unless I'm standing in front of you with a big whiteboard, so I'll refer to things that haven't been defined yet. We'll get there eventually, I promise.

An IAM (Index Allocation Map) page tracks approximately 4GB worth of space in a single file, aligned on a 4GB boundary. These 4GB chunks are called 'GAM intervals' (another post!). The IAM page tracks what space within that specific GAM interval belongs to a single entity (I'm chosing my words carefully here and not using any word that has SQL Server connotations like 'object'). I'll get to the structure of an IAM page below.

Now, as an IAM page can only track the space for a single GAM interval in a single file, if the database has multiple files, or some files are more then 4GB, and the entity has space allocated from multiple files or multiple GAM intervals within a file, then you can see how multiple IAM pages are needed for each entity to track all the space that its using.

This is where an IAM chain comes in. It's a linked-list of IAM pages that track the space allocated to a single entity. The linked-list is not sorted at all - IAM pages are appended to it in the order that they're needed. The IAM pages within the list are numbered, again, in the order that they were appended to the list.

Definition of 'entity' - what uses an IAM chain? This is vastly different in SQL Server 2000 and 2005, hence the need for two posts.

In SQL Server 2000, a single IAM chain is used for each:

heap or clustered index

a table can only have one or the other, not both
these have index IDs of 0 and 1 respectively

non-clustered index

these have index IDs from 2 to 250 (i.e. you can only(!) have 249 of them)

text storage

for LOB columns in the heap or clustered index
sometimes called the 'text index'
these have a fixedindex ID of 255

This is very simple, right? I usually generalize and say that in SQL Server 2000, there's one IAM chain per index (which fits nicely if you remember that IAM stands for Index Allocation Map).

How exactly does each IAM page track space usage within the GAM interval that it maps to? Each IAM page has two records, an IAM page header and a bitmap.

The IAM page header tracks metadata about the IAM chain, including:

the GAM interval that the IAM page maps
the sequence number of the IAM page within the IAM chain

this increases by one for each page added to the chain
the pages are linked using their previous and next page pointers and the linked-list must be in strict sequence number order

the object and index IDs of the index that the chain maps

all the IAM pages in the IAM chain (and in fact all pages allocated to the index) have the same object and index ID stamped in their page headers.

It also contains the single-page array. This is only used in the first IAM page in the chain and tracks single pages that have been allocated to the index, instead of extents (an extent is a group of 8 contiguous pages - I'll cover pages and extents more fully next week in Fragmentation parts 2 and 3 - it's that chicken-and-egg thing again).

The bitmap occupies the rest of the IAM page and has a bit for each extent in the GAM interval The bit is set if the extent is allocated to the index, and clear if it is not. Obviously two IAM pages that map the same GAM interval for different indexes cannot both have the same bit set (sounds like a job for CHECKDB!)

You can look at the contents of an IAM page using DBCC PAGE. Use the instructions in this post to work out what pages are IAM pages and have a look.

Some random things to note about IAM chains and pages:

they are not self-referential (they do not track the space taken up by themselves)
if some data is deleted, leading to space deallocation, leading to a GAM interval no longer having any space allocated for an index, the IAM page is not deleted
the only operations that delete pages from the IAM chain are TRUNCATE TABLE, DROP TABLE, and certain repairs in DBCC CHECKDB
DBCC INDEXDEFRAG makes a copy of the IAM chain for an index and sorts it by file and GAM interval to allow it to work out what the next physical extent/page in each file is
the IAM chain repairs are far-and-away the most complicated

In the next post, I'll explain how SQL Server 2005 has changed things.

(I notice that some previous posts get rated 4/5 and some 5/5 - I'd love to know what it is that's missing from posts that get rated 4/5 so I can make them more useful/enjoyable - drop me a line...)

Fragmentation (part 1): What are records?

Paul Randal - MSFT — Fri, 23 Jun 2006 18:41:00 GMT

This blogging thing sucks you in, doesn't it? Not content with having an ongoing series on disaster recovery and CHECKDB (with another 6 and 25 more posts planned respectively), I'm starting a new series on fragmentation. This will begin from first principles and work up, in approximately 18 posts over the next few months. The first few posts could be skipped by some people and will cover:

What are records?
What are pages?
What are extents?
What is a heap?
What is a clustered index?
What is a non-clustered index?

Bear with me as I build up the terminology. You could also check out Kalen's excellent book Inside SQL Server 2000 for details on these topics. (Her upcoming volume on the Storage Engine for Inside SQL Server 2005 should be out sometime this summer - buy it!).

The idea for this series came to mind last Friday at TechEd when I spent about 3 hours repeating a deck on fragmentation I gave to the North Texas SSUG in April 2004, people kept wandering by and stopping to listen. If that many people are interested in this stuff, it should make good blog material, and it also seems to be frequently misunderstood. The level of sophistication here ranges from not having any idea what fragmentation is up to defragging or rebuilding all indexes every night. I didn't see anyone there who knew just when it was worth removing fragmentation so I decided to explain here. The first 6 posts will also be useful as background for the CHECKDB internals series I'm doing.

So, what are records? At the simplest level, a record is the physical storage associated with a table or index row. Of course, it gets much more complicated than that...

Data records

These store table rows in a heap, or in the leaf level of a clustered index. They store all the columns of the table row that will fit in a single 8Kb page.
Data records are stored on data pages.
If any columns are for LOB data types (text, ntext, image, and the new LOB types in SQL Server 2005 - varchar(max), nvarchar(max), varbinary(max), XML), then there's a pointer stored in the data record which points to a text record on a different page (the root of a loose tree that stores the LOB value). Exceptions to this are when schema has been set to store LOB columns 'in-row' when possible. This is when a LOB value is small enough to fit within the confines of the data page that holds the data record, and so is stored in the same data record. This is a performance benefit as selecting the LOB column does not require an extra IO to read the text record.
In SQL Server 2005, non-LOB variable length columns (e.g. varchar, sqlvariant) may also be stored 'off-row' as part of the new capability (called row-overflow) of having table rows longer than 8060 bytes. In this case the storage format is the same as for LOB values - a pointer in the data record pointing to a text record.
I'll go into how the columns are laid out differently between heaps and clustered indexes in parts 4 and 5. There are also some twists in the record types for heaps that I'll cover in part 4.

Index records

There are two types of index records (which differ only in what columns they store):

Those that store non-clustered index rows
Those that comprise the b-tree that make up clustered and non-clustered indexes.

I'll explain more about the differences between these in parts 5 and 6 - it can be quite complicated (especially the differences between SQL Server 2000 and 2005) and is worth doing in separate posts.
Index records are stored on index pages.
Index records typically do not contain all the column values in a table (although some do - called covering indexes).
In SQL Server 2005, non-clustered index records can include LOB values as included columns (with the storage details exactly the same as for data records) and also can have row-overflow data that is pushed off-row (again, in exactly the same way as for data records).

Text records

There are various types of text records that comprise the tree structure that stores LOB values, stored on two types of text page.
An in-depth explanation of the various types and how they hang together is beyond the scope of this post and discussion.
They are also used to store variable-length column values that have been pushed out of data or index records as part of the row-overflow capability.

Ghost records

These are records that have been logically deleted but not physically deleted from the leaf level of an index.
The reasons for this are complicated, but basically having ghost records simplfies key-range locking and transaction rollback.
The record is marked with a bit that indicates it's a ghost record and cannot be physically deleted until the transaction that caused it to be ghosted commits. Once this is done, it is deleted by an asynchronous background proces (called the ghost-cleanup task) or it is converted back to a real record by an insert of a record with the exact same set of keys.
Ghost records will be mentioned later in the series when I discuss page compaction.

Other record types

There are also records that are used to store various allocation bitmaps, intermediate results of sort operations, and file and database metadata (e.g. in the per-file fileheader page and database boot page).
These are not relevant to a discussion of fragmentation so won't be considered further here.

Record structure

All records have the same structure, regardless of their type and use, but the number and type of columns will be different. For instance, a data record from a table with a complex schema may have hundreds of columns of various types whereas an allocation bitmap record will have a single column, filling up the whole page.

The record structure isn't relevant to a discussion on fragmentation but is for CHECKDB internals, so here it is:

record header

4 bytes long
two bytes of record metadata (record type)
two bytes pointing forward in the record to the NULL bitmap

fixed length portion of the record, containing the columns storing data types that have fixed lengths (e.g. bigint, char(10), datetime)
NULL bitmap

two bytes for count of columns in the record
variable number of bytes to store one bit per column in the record, regardless of whether the column is nullable or not (this is different and simpler than SQL Server 2000 which had one bit per nullable column only)
this allows an optimization when reading columns that are NULL

variable-length column offset array

two bytes for the count of variable-length columns
two bytes per variable length column, giving the offset to the start of the column value

versioning tag

this is in SQL Server 2005 only
this is a 14-byte structure that contains a timestamp plus a pointer into the version store in tempdb

If you have any questions on this stuff - put them in the comments of drop me an email.

Next time - what are pages?

How to use DBCC PAGE

Paul Randal - MSFT — Sat, 10 Jun 2006 21:14:00 GMT

Yes, finally I come clean and tell all. It's an open secret that there's an undocumented DBCC command called DBCC PAGE that you can use to look at the contents of database pages. I've recommended in forum postings that people use it and Product Support also asks customers to use it during various investigations. Bottom line - it's there, and its extremely well tested (we use it extensively internally in literally thousands of tests). Bear in mind, however, that it is undocumented and thus unsupported - you won't get any help using if from Product Support. Can you use it on production systems? I don't see any reason why not but you should be wary, as with any undocumented command, procedure or trace flag.

Why am I doing this? I get asked this so much from people that are curious and I'd like to do some posts on interpreting CHECKDB results, which is a little hard unless you use DBCC PAGE.

So what's the syntax?

dbcc page ( {'dbname' | dbid}, filenum, pagenum [, printopt={0|1|2|3} ])

The filenum and pagenum parameters are taken from the page IDs that come from various system tables and appear in DBCC or other system error messages. A page ID of, say, (1:354) has filenum = 1 and pagenum = 354.

The printopt parameter has the following meanings:

0 - print just the page header
1 - page header plus per-row hex dumps and a dump of the page slot array (unless its a page that doesn't have one, like allocation bitmaps)
2 - page header plus whole page hex dump
3 - page header plus detailed per-row interpretation

The per-row interpretation work for all page types, including allocation bitmaps.

By default, the output is sent to the errorlog. If you want the output to come back to your current connection, turn on trace flag 3604.

How do you find a page to dump? There are some easy ones - the allocation bitmaps. For example, lets dump the first PFS page in the database, just looking at its header:

DBCC PAGE (master, 1, 1, 0);

PAGE: (1:1)

BUFFER:

BUF @0x02BB582C

bpage = 0x03772000 bhash = 0x00000000 bpageno = (1:1)

bdbid = 1 breferences = 1 bUse1 = 42182

bstat = 0xc0000b blog = 0x21598979 bnext = 0x00000000

PAGE HEADER:

Page @0x03772000

m_pageId = (1:1) m_headerVersion = 1 m_type = 11

m_typeFlagBits = 0x3 m_level = 0 m_flagBits = 0x0

m_objId (AllocUnitId.idObj) = 99 m_indexId (AllocUnitId.idInd) = 0 Metadata: AllocUnitId = 6488064

Metadata: PartitionId = 0 Metadata: IndexId = 0 Metadata: ObjectId = 99

m_prevPage = (0:0) m_nextPage = (0:0) pminlen = 0

m_slotCnt = 1 m_freeCnt = 2 m_freeData = 8188

m_reservedCnt = 0 m_lsn = (199:344:5) m_xactReserved = 0

m_xdesId = (0:0) m_ghostRecCnt = 0 m_tornBits = 351018853

Allocation Status

GAM (1:2) = ALLOCATED SGAM (1:3) = NOT ALLOCATED PFS (1:1) = 0x44 ALLOCATED 100_PCT_FULL

DIFF (1:6) = CHANGED ML (1:7) = NOT MIN_LOGGED

DBCC execution completed. If DBCC printed error messages, contact your system administrator.

There are a bunch of things on this that are beyond the scope of this post to explain - I'll get to them in future posts. One thing you'll notice is that there's some interpretation of the object and index IDs that are stamped on the page. This is because in SQL Server 2005, these are derived from the allocation unit ID, not the actual object and index ID. Simplistically, this is because of partitioning - each index can now have multiple b-trees, and hence multiple IAM chains. Each IAM chain is called an allocation unit. In the next post I'll go into some detail on how this is structured. Chicken-and-egg problem again, or a teaser to keep reading the blog :-)

Now how about if we ask for the detailed dump of the PFS page?

DBCC PAGE (master, 1, 1, 3);

PAGE: (1:1)

BUFFER:

BUF @0x02BB582C

bpage = 0x03772000 bhash = 0x00000000 bpageno = (1:1)

bdbid = 1 breferences = 1 bUse1 = 42182

bstat = 0xc0000b blog = 0x21598979 bnext = 0x00000000

PAGE HEADER:

Page @0x03772000

m_pageId = (1:1) m_headerVersion = 1 m_type = 11

m_typeFlagBits = 0x3 m_level = 0 m_flagBits = 0x0

m_objId (AllocUnitId.idObj) = 99 m_indexId (AllocUnitId.idInd) = 0 Metadata: AllocUnitId = 6488064

Metadata: PartitionId = 0 Metadata: IndexId = 0 Metadata: ObjectId = 99

m_prevPage = (0:0) m_nextPage = (0:0) pminlen = 0

m_slotCnt = 1 m_freeCnt = 2 m_freeData = 8188

m_reservedCnt = 0 m_lsn = (199:344:5) m_xactReserved = 0

m_xdesId = (0:0) m_ghostRecCnt = 0 m_tornBits = 351018853

Allocation Status

GAM (1:2) = ALLOCATED SGAM (1:3) = NOT ALLOCATED PFS (1:1) = 0x44 ALLOCATED 100_PCT_FULL

DIFF (1:6) = CHANGED ML (1:7) = NOT MIN_LOGGED

PFS: Page Alloc Status @0x4414C000

(1:0) - (1:3) = ALLOCATED 100_PCT_FULL

(1:4) - (1:5) = NOT ALLOCATED 0_PCT_FULL

(1:6) - (1:7) = ALLOCATED 100_PCT_FULL

(1:8) - = ALLOCATED 0_PCT_FULL Mixed Ext

(1:9) - (1:10) = ALLOCATED 100_PCT_FULL Mixed Ext

(1:11) - = ALLOCATED 0_PCT_FULL Mixed Ext

(1:12) - = ALLOCATED 0_PCT_FULL IAM Page Mixed Ext

DBCC execution completed. If DBCC printed error messages, contact your system administrator.

How about being able to dump some pages from a user table? Firstly, you have to work out which pages comprise the table and index. Here's one way to do this (without having to divulge any more undocumented commands :-)

create an empty database
do a type 3 page dump of the first page PFS page in the database (1:1) using DBCC PAGE
creating a simple heap and insert a few rows into it
do another PFS dump like in step 2 and you'll see some more pages have been allocated - these are the ones being used to store your new table. One of them will be the IAM page and another will be a data page.

Find some pages to dump and play about with DBCC PAGE. Experiment with adding indexes and LOB columns to see what different kinds of pages are created and look at the linkages between them. I'll go into what the various parts of the output mean in another post - in the meantime, if there's anything in particular you want to know, add a comment and I'll reply.

Have fun!