SQL Server Storage Engine : General

CHECKSUM and Tempdb

Sunil Agarwal — Mon, 24 Mar 2008 00:52:00 GMT

You may recall that starting with SQL Server 2005, you have an option available to enable CHECKSUM on the user databases. For details, please refer to http://blogs.msdn.com/sqlserverstorageengine/archive/2006/06/29/Enabling-CHECKSUM-in-SQL2005.aspx. In fact, any new database created in SQL2005 has CHECKSUM enabled automatically but it does not happen to databases that are upgraded from previous versions of SQL Server. For the upgraded databases from SQL Server versions earlier than SQL Server2005, you will need to enable CHECKSUM explicitly using ALTER DATABASE command. Enabling CHECKSUM is critical and it allows SQL Server to detect the corruption in the IO path (e.g. a disk malfunction) when the page is read as part of query or when you run DBCC CHECKDB command. While this has been very useful to our customers, there was one missing link and that was that SQL Server did not allow enabling CHECKSUM on tempdb. This provided a window where a page corrupted due to mis-behaving disk found its way into user database even if you had enabled CHECKSUM on the databse. Here is one such scenario

Scenario:: bulk import the data into a temp table for staging purposes and then move it to the user database. The user database has CHECKSUM enabled, so the new page, when written to the disk, will have checksum computed. But guess what is missing? If the tempdb disk corrupts the pages in tempdb, SQL Server will have no way of knowing that the page was corrupted and it will go to user database without detection. Yes, when the page is subsequently read, depending upon what the corruption was, the SQL Server may detect it or may not. For example if a bit flip happened for the integer value, it will go undetected.

With CHECKSUM available on tempdb starting with SQL2008, you can finally close this window. You can use the following command

ALTER DATASE tempdb set PAGE_VERIFY CHECKSUM

For new installs of SQL Server 2008, all tempdbs will have CHECKSUM enabled by default. You can always disable it using ALTER DATABASE command but we don't recommend it. For databases upgraded to SQL Server 2008, you will need to explicitly enable CHECKSUM on the tempdb. We measured the performance impact of enabling CHECKSUM in tempdb and the impact is very low (<2% of CPU) which is similar to what you would expect in user database. Since the CHECKSUM is only computed when page is written to the disk, the added point is that there is no 'checkpoint' in tempdb, so a page in tempdb is written to disk ONLY under memory pressure. So you may not see as many CHECKSUM calculations in tempdb.

Note, this is not available in CTP-6 but will be available in RTM bits.

Thanks

Sunil

ok - really the last blog post - Mondays, Mystic, and a Museum

Paul Randal - MSFT — Thu, 23 Aug 2007 21:42:00 GMT

My new blog's not active until September 1st but I wanted to blog about something cool - and totally unrelated to work. Over the weekend Kimberly and I were in Connecticut for the birthday party pig-roast for Carl Franklin (of DotNetRocks fame) along with our good friends Richard Campbell - also from DotNetRocks - and his wife Stacy and Michele Leroux Bustamante. Apart from the great party, we got to be the live audience for the latest recording of their comedy show Mondays in the very cool studio where DotNetRocks is recorded and produced. This is one of the funniest shows I've ever heard - Mark Miller had us crying with laughter. I heartily encourage you to listen - parental advisory though - not for the sensitive or easily offended :-) The link to the show is here - all I can say is BANANAS!!!

While we were there we also went to Mystic Seaport and the US Navy Submarine Force Museum - both excellent and well worth visiting, especially if you're a ships-and-submarines nut like me.

Cheers!

Splits and unions...

Paul Randal - MSFT — Tue, 31 Jul 2007 00:45:00 GMT

Hey folks,

This is my last blog post on the SQL Server Storage Engine blog as I've made the decision to leave Microsoft and join Kimberly running her company, SQLskills.com. This lets me indulge my passion for teaching and consulting full-time and lets me work with my best friend. My last day at Microsoft is August 31st and from September 1st I'll be blogging at http://www.sqlskills.com/blogs/paul/. I'd like to thank all of you for reading my posts here, commenting and emailing me, and I really hope you'll continue to read this blog, as well as my new one :-)

More importantly, yesterday (Sunday, July 29th 2007), Kimberly Tripp and I were married at The Woodmark Hotel in Kirkland, WA, with around 100 friends and family in attendance. Below is a photo of us courtesy of our fabulous photographer, John Mitchell.

Here's a link to his blog with some great pictures of the BBQ we threw on Friday at our house.

Thanks and best regards to you all!

Would you like to join the Storage Engine PM team?

Paul Randal - MSFT — Sat, 12 May 2007 17:16:00 GMT

We have some job openings on the Storage Engine Program Management team - if you're looking for a great career opportunity helping to drive the product you love using, check them out.

What does a Program Manager do? This blog entry by Steven Sinofsky (one of our Senior VPs) sums it up nicely.

Here are links to some of our job descriptions on the Microsoft careers website:

If any of these look interesting to you then send an email to sethuk@microsoft.com with your resume/CV attached and we'll take it from there.

Look forward to hearing from you!

What's the difference between database version and database compatibility level?

Paul Randal - MSFT — Fri, 27 Apr 2007 02:54:00 GMT

I had a question this week from someone who'd heard me say at SQL Connections (paraphrasing) "database compatibility level is mostly about query parsing" and was having trouble trying to forcibly attach a 2005 or 7.0 database to a 2000 server.

His confusion is between database compatibility level and database version. Here's a quick explanation of the difference.

Database version

The database version is a number stamped in the boot page of a database that indicates the SQL Server version of the most recent SQL Server instance the database was attached to. The database version number does not equal the SQL Server version. For example, doing the following:

SELECT @@version;

on one SQL Server instance on my laptop returns:

which I think is an intermediate SP2 build - not bothering to check. However, the database version is 611. You can't see this except using undocumented DBCC commands (which I'm not blogging - yet) but if you attach a database from an earlier version of SQL Server, you'll see these numbers in the error log as SQL Server reports what upgrade steps its doing.

[Edit: I stand corrected. Thanks to Trayce Jordan at Microsoft for pointing out that the database version will be returned using the following T-SQL:

USE master;

SELECT DatabaseProperty ('dbccpagetest', 'version');

/Edit]

Some things to note about database version:

SQL Server is not up-level compatible. You cannot attach a database that was created on (or has been upgraded to) SQL Server 2005 to any earlier version of SQL Server (also true for trying to attach a 2000 database to 7.0, and so on).
You cannot attach a database that was created on an earlier version without going through the proper upgrade procedures. Forcibly attaching a database using various hacky methods will result in all kinds of weird errors, and possibly crashes.

Database compatibility level

The database compatibility level determines how certain database behaviors work. For instance, in 90 compatibility, you need to use the OUTER JOIN syntax to do an outer join, whereas in earlier compatibility levels, you can use '*=' and '=*'. Contrary to popular myth, all of the behavioral differences ARE documented - in the Books Online section for sp_dbcmptlevel - the SP used to set the compatibility level.

There are 5 supported compatibility levels support by SQL Server 2005:

60 = SQL Server 6.0

65 = SQL Server 6.5

70 = SQL Server 7.0

80 = SQL Server 2000

90 = SQL Server 2005

You can see the compatibility level of all databases by doing:

SELECT name AS 'DB Name',

compatibility_level AS 'Compatibility Level'

FROM master.sys.databases;

Some things to note about compatibility levels:

A database created on SQL Server 2005 will have a default compatibility level of 90, unless the model database has a different compatibility level, in which case the new database inherits the compatibility level of model.
New features may work under older compatibility levels but beware of SET options.
An upgraded database retains its compatibility level. For example, a database that was created on SQL Server 2000, and didn't have its compatibility level altered, will stay in 80 compatibility level when its upgraded to SQL Server 2005.

Summary

This was just a quick - and by no means comprehensive - explanation of the difference between the two terms. Basically, there's no relationship between them.

Hope this helps!

SP2 Maintenance Plan bugs fixed

Paul Randal - MSFT — Fri, 13 Apr 2007 03:31:00 GMT

From the tools team here in Redmond:

We recently posted updates to address an issue in two SP2 Maintenance Plan
tasks. The KB article describes the symptoms as follows:

FIX: The Check Database Integrity task and the Execute T-SQL Statement task
in a maintenance plan may lose database context in certain circumstances in
SQL Server 2005

In Microsoft SQL Server 2005 Service Pack 2 (SP2), the following tasks in a
maintenance plan may lose database context in certain circumstances:
. Check Database Integrity
. Execute T-SQL Statement
For example, if a Check Database Integrity task is configured to check
integrity on multiple databases, the task repeatedly checks the default
database.

For the Execute T-SQL Statement task, this problem occurs if the following
conditions are true:
. The Execute T-SQL Statement task contains multiple batches.
. The batches are separated by the GO statement.
. No batches use the USE statement to specify the database.
. The statements in each batch do not contain the database name.
In this scenario, those statements in the second batch and in succeeding
batches will be executed against the default database.

This fix is shipped in two different packages:
. The GDR release is available via http://support.microsoft.com/kb/934458
. The QFE release is available via http://support.microsoft.com/kb/934459

What is the difference between a GDR and QFE release? GDRs are a new means
of providing support. GDR fixes are reserved for those issues identified by
SQL Server support as important enough to install on every instance of SQL
Server. QFEs are used for the majority of fixes where the effects of the
problem are not widespread or severe enough to warrant a GDR.

GDRs and QFEs are associated with certain build numbers. Build numbers 3043
through 3149 are reserved for GDR releases. Build numbers 3150 and up are
reserved for QFEs. By reserving lower build numbers for GDR releases, users
can continue to take advantage of GDR fixes, without having to install QFEs,
which aggregate all the fixes shipped since the most recent service pack.
Once a QFE is installed, GDRs will no longer update your system, because the
build number is too low, and you have to install QFEs to pick up new fixes.

Why is this fix a GDR? This fix was promoted to GDR because users that
relied on the Check Database Integrity task to validate their databases were
not informed of database integrity issues that they expected to be detected.
The fix also addresses issues in the Execute T-SQL task, but, since these
issues typically lead to script failure, duly noted by SQL Server, the T-SQL
issues alone were not enough to justify the GDR.

Those users who now, or in the future, rely on the Check Database Integrity
Task or the Execute T-SQL task in a maintenance plan should install the GDR.
Others may skip the GDR.

New whitepaper on Online Indexing Operations

Paul Randal - MSFT — Thu, 15 Mar 2007 16:56:00 GMT

A great new whitepaper has just been published on TechNet - Online Indexing Operation in SQL Server 2005. It gives a detailed explanation of the online index build process plus guidelines and best practices for implementing online indexing in production. Well worth reading.

While I'm on the subject, some other whitepapers around the Storage Engine that are well worth reading too:

Enjoy

Can you INCLUDE columns of type TEXT, NTEXT, or IMAGE?

Paul Randal - MSFT — Thu, 15 Mar 2007 00:31:00 GMT

I was sent a question today that seems like it could be something that many people get confused about – why can’t an NTEXT column be specified in the INCLUDE list of a CREATE INDEX statement in SQL 2005?

(In one of my internals posts from last year there’s a long explanation of included columns – its basically a way to have true covering indexes to save IOs by not having to go back to the base table once an index row has been found).

The answer to the question is that columns of the legacy large-value data types (TEXT, NTEXT, and IMAGE) cannot be used in the INCLUDE list. You’ll need to change the schema to use one of the new large-value data types (VARCHAR(MAX), NVARCHAR(MAX), VARBINARY(MAX), and XML).

The new types are stored in almost exactly the same way as the legacy types, but the internals of how they are processed by the server are different so they behave like regular data types (in that they work with most T-SQL intrinsic functions and can be used as included columns).

For a more detailed list of possibilities and restrictions on included columns, see the BOL topic Index with Included Columns (look for ‘included columns’ in the BOL Index).

How to create fake statistics for testing query plans

Paul Randal - MSFT — Tue, 13 Mar 2007 04:44:00 GMT

This is very cool - an undocumented option to UPDATE STATISTICS that allows you to fool the server into thinking a table has more rows or pages than it actually has - very useful for testing. See this blog post on the old Query Optimizer team blog. Thanks to Tony Rogerson for pointing it out.

Sometimes life throws you a curve-ball...

Paul Randal - MSFT — Sun, 11 Mar 2007 10:08:00 GMT

Many apologies to those who were hoping to see my presentation on corruption detection and recovery at SQL Server Open World in Denmark last week. An emergency came up after the class Kimberly and I taught in England and we ended up flying a *lot* further than we'd planned - basically all the rest of the way round the world. Anyway, more details on Kimberly's blog here.

As for the England event, we got a ton of ideas for blog post topics - look for these over the next few months spread over both blogs. I've also had lots of email responses to the last few posts - I'll reply/post when I'm back in the US (I'm typing this while en-route home, we're in Narita/Tokyo...)

Thanks

CHECKDB, User Groups and .NET Rocks!

Paul Randal - MSFT — Wed, 28 Feb 2007 17:38:00 GMT

A few things to cover with the first cup of coffee this morning before doing a meatier post or two later today.

First off - I've had a bunch of good responses to my CHECKDB runtime survey and I've got data on 66 databases ranging from 10.5GB up to 2.8TB. I'd like to get some more responses, and I have a lot more DVDs to send anywhere in the world so please take some time to respond. Some of the answers have been eye-opening and I'll blog on at least one facet of these later today. Depending on your answers and needs, we may be able to setup a 1-1 con-call to discuss issues around corruption detection and recovery.

Next up is user groups. As part of the same feedback program on CHECKDB, I've been asking around internal Microsoft aliases to get in touch with customers and out of that has come plans to do remote presentations at user groups around the world. Do you run a SQL Server user group and would be interested in a remote presentation from me or one of my team in the Storage Engine? If so, drop me a line at prandal@microsoft.com

Finally, a couple of weeks back I did a .NET Rocks! interview with Richard and Carl who I met at TechEd in Barcelona last year. It'll be released on their website March 6th and I talk a lot about CHECKDB, checksums, emergency mode repair and some of the daft things people do. We also ambush a special SQL community guest :-)

SQL Server 2005 Best Practices Analyzer CTP released

Paul Randal - MSFT — Thu, 22 Feb 2007 23:44:00 GMT

Folks,

The much-anticipated SQL Server 2005 Best Practices Analyzer has been released for on the Microsoft Download Center. There are a ton of things in there, including some checks around backup and CHECKDB frequency.

SQL Server 2005 Best Practices Analyzer (February 2007 CTP):
http://www.microsoft.com/downloads/details.aspx?FamilyId=DA0531E4-E94C-4991-82FA-F0E3FBD05E63&displaylang=en

Wow - over 40 blogs from the SQL Product Team

Paul Randal - MSFT — Wed, 14 Feb 2007 21:47:00 GMT

Seeing the new blog from the doc team inspired me to ping the extended SQL team to see what active blogs there are. Here's the list I've compiled of active (or "I'm working on content right now") blogs run by members of the SQL team - grouped by area and in no particular order. Lots of cool stuff here...

General SQL Server

Samples team - code samples, mainly for SQL Server 2005
ISV team
UE team
Compact Edition team
SQL Express team
Performance team
Customer Advisory team
Support team
Support team - another PSS blog, mostly Escalation Services
Umachandar Jayachandran - tips and tricks
Matt Hollingsworth - HA
Euan Garden - SQL, VS, ...

Data Programmability

Data Programmability team
ADO.NET team - replaced the Data Access blog
SQL Protocols team
XML team
Protocols team
Angel Saenz-Badillos - JDBC / Lin

Storage Engine

This one (obviously... :-)
Slava Oks - SQLOS / memory management
Saurabh Singhal - SQLOS
Laurentiu Cristofor - Security
Raul Garcia - Security

Service Broker

Roger Wolter - SOA, data management
Rushi Desai - Service Broker
Remus Rusanu - Service Broker

Relational Engine

Query Processor team (replaces the old Query Optimizer team blog)
Programmability & API Development team
CLR Integration team
Denis Ruckebusch - XML datatype
Lyudmila Fokina - various including index builds (in Russian)
Michael Rys - XML, XQuery
Craig Freedman - query processing, execution and plans

Analysis Services / Data Mining

Jamie MacLennan

Reporting Services

John Gallardo
Bob Meyers
Brian Welcker
Lukasz Pawlowski - RS, Notification Services

Sync Services / Replication

Synchronizer

SSIS / DTS

Manageability / Tools

Manageability team
Michiel Wories - SMO / WMI samples

How is Books Online put together?

Paul Randal - MSFT — Wed, 14 Feb 2007 20:49:00 GMT

One of the Technical Writers in the SQL Product Group, Buck Woody, has started a blog about SQL documentation. His first post is an overview of how Books Online is created. Check it out at http://blogs.msdn.com/buckwoody/archive/2007/02/14/sql-server-books-online.aspx.

How to move databases using detach/attach functions?

Paul Randal - MSFT — Wed, 14 Feb 2007 16:39:00 GMT

Quickie post this morning. I was in a review of customer support cases yesterday lunchtime and noticed a trend of cases where people get into trouble moving databases around. There is a good KB article (http://support.microsoft.com/kb/224071/) which explains how to do this for user databases, master, model, tempdb and msdb. It also contains a link to a SQL Server 2005 hotfix for the problem where after detaching a databases that resides on network-attached storage, it can't be reattached.

Well worth reading to avoid downtime and a support call.