Microsoft SQL Server Development Customer Advisory Team : Manageability & Serviceability

Remember to move DTS2000 packages when upgrading msdb from 2005 to 2008

tkejser — Fri, 20 Mar 2009 14:06:00 GMT

Some customers prefer to upgrade to SQL Server 2008 by detaching all databases from their SQL Server 2005 instance and then reattaching them to a completely new SQL Server 2008 instance.

If you choose this upgrade strategy, you should be aware that you cannot detach an msdb database from a SQL Server 2005 instance and reattach it to a SQL Server 2008 instance. If you try, you will get an error message. Backup/restore will not work for msdb in this upgrade scenario either.

The msdb database may contain objects that are relevant to your installation – for example agent jobs and DTS 2000 packages. When upgrading, you should make sure you move all relevant objects in msdb. You can use tasks in Integration Services to move most of these objects. However, DTS 2000 packages cannot be moved using the Integration Services tasks. Instead, you must manually move the contents sysdtspackage.

The CSS team has a great blog entry describing how to move DTS 2000 packages to a SQL Server 2008 instance: How to copy DTS 2000 packages between servers (and from SQL 2000 to SQL 2005 and SQL 2008)

New Version of Partition Management Utility Available

stuarto — Tue, 24 Feb 2009 00:19:00 GMT

I’m happy to announce availability of an updated version of the SQL Server Partition Management tool – now offering full support for SQL Server 2008 as well as improvements in handling SQL Server 2005. The tool is available along with source code at http://www.codeplex.com/SQLPartitionMgmt. If you’re not already familiar with the tool, it provides a set of commands (at the Command Line or via Powershell) to create a staging table on-demand (including all appropriate indexes and constraints) based on a specific partitioned table and particular partition of interest.

By calling this executable, with parameters, from maintenance scripts or SSIS packages, DBAs can avoid having to ‘hard code’ table and index definition scripts for staging tables. The tool eliminates the challenge of keeping such scripts in synch with changes to partition tables’ columns or indexes. It also provides a fast, single-command shortcut for the operation of quickly deleting all data from a partition.

The latest version supports new features in SQL Server 2008 such as filtered indexes, new data types, and partition-aligned indexed views. Enjoy!

-- Stuart Ozer / SQLCAT

SQLCAT Toolbox

Sanjay Mishra — Thu, 21 Feb 2008 21:58:00 GMT

SQLCAT members have published several tools on Codeplex and other places. They are all consolidated here at SQLCAT.com.

http://sqlcat.com/toolbox/default.aspx

New Partition Management Tool Available on Codeplex

stuarto — Mon, 28 Jan 2008 23:02:00 GMT

As anyone working with partitioned tables knows by now, the process of creating stand-alone ‘staging’ tables to support switching data into or out of a partition can be a management challenge, especially if the design of the partition table or its indices change over time. A staging table for new data to be switched-in must include check constraints that match the partition boundaries of the target, and include all columns, constraints and indexes appearing in the partition table. Often, the code to create the staging tables resides in a set of manually created management scripts that must remain carefully synchronized to any changes made in the partition table design. When columns are added, column characteristics change (such as nullability or length) or index definitions are tweaked, the management script creating the staging tables must also be updated. If management scripts fall out of synch, often we discover it only while in the middle of a critical month-end partition switch process, raising the risk of unplanned extension or delay of the maintenance window.

I’ve developed a simple, shared source Partition Management Tool for SQL Server 2005 to make the process of creating the staging tables for partition management nearly foolproof. At runtime, the definition of all columns, indexes, and constraints for the partition table are read using the SMO object model, and a perfect clone table matching a specific partition is created.

The tool also provides an option for creating a staging table without indexes, and then for building indexes later after the table is populated – for applications that prefer to bulk load into a new partition without indexes present. It also includes a quick, transparent method to very quickly and simply clear the contents of a single partition within a partitioned table – a task that you may want to perform when replacing the complete contents of one partition.

Also rather than requiring you to have knowledge of the specific partition number that you are referencing, you may indicate the appropriate partition by passing a value of the partitioning column that falls within the desired partition range when invoking the tool.

The Partition Management Tool runs from the command line, so unfortunately it cannot be used from within SQL Server stored procedures or batches. That’s because SMO-based CLR routines cannot run in-proc in SQL Server 2005. But the tool can be invoked from within SSIS (as an Execute Process task), so that runtime-dependent package variables can be integrated and the script can become part of an automated sliding window scenario orchestrated by SSIS. When invoking from SSIS, the command line arguments of the tool can be populated by expressions within SSIS.

You can download the tool and associated C# source code from SQL Server 2005 Partition Management Tool .

Stuart Ozer

New Best Practices Whitepaper published: "Best Practices for Migrating Non-Unicode Data Types to Unicode"

Sanjay Mishra — Tue, 22 Jan 2008 02:18:00 GMT

The database system in a company that is growing internationally must support multilingual characters in tables with Unicode data types. Existing databases that support only non-Unicode information must be migrated from non-Unicode data type to Unicode data type. This paper summarizes best practices for migrating database systems from non-Unicode to Unicode data type.

http://download.microsoft.com/download/d/9/4/d948f981-926e-40fa-a026-5bfcf076d9b9/SQL_bestpract_MigrationToUnicode.docx

SQL Server 2005 Best Practices Analyzer (January 2008) -- Now Available!

Paul Mestemaker — Sat, 19 Jan 2008 00:36:00 GMT

It's a new year and, with it, we come bearing gifts! We have a pretty significant update to SQL Server 2005 Best Practices Analyzer. It contains many new and updated rules for Analysis Services, a few important rules for the Relational Engine, a couple bug fixes for the UI and command line tools. And all of these rules have rich documentation telling you what need to know. ......(read more)

New Best Practices Whitepaper Published: The Impact of Changing Collations and of Changing Data Types from Non-Unicode to Unicode

Sanjay Mishra — Fri, 07 Dec 2007 02:07:00 GMT

When a business grows internationally, its database system must support multilingual characters in tables. Businesses that previously needed to manage only non-Unicode information must now support database tables that include non-Unicode data types such as CHAR or VARCHAR. As the business grows internationally, it must alter its database system to support Unicode data types. It may also need databases that have different collations across several servers.

http://download.microsoft.com/download/d/9/4/d948f981-926e-40fa-a026-5bfcf076d9b9/SQL_bestpract_CollationChange.docx

New Best Practices Article

carl.rabeler — Wed, 12 Sep 2007 22:10:00 GMT

Scale-Out Querying with Analysis Services Using SAN Snapshots

Renaming Olap Databases Issues within Analysis Services

denny.lee — Tue, 31 Jul 2007 05:06:00 GMT

If you are view the XMLA script that is generated for an Analysis Services database, you will notice that there are both ID and NAME attributes. The reason Analysis Services has this differentiation is so one could rename an Olap database (which you could not do officially within Analysis Services 2000). Database name defaults to its ID and the ID cannot be modified. You can, on the other hand, change the database name. This way end-users could have a different name to the Olap database irrelevant of what its ID was. For example, you could have the ID of the Olap database as "Foodmart" while the name of the database could be "Wade's Groceries". The file system itself, i.e. in the c:\Program Files\Microsoft SQL Server\MSSQL.2\Data folder has a Master.vmp and a folder name (e.g. Foodmart) that still refers to the ID. That is, the folder name and any ID references are to Foodmart irrelevant of whether the database name is "Foodmart", "Wade's Groceries", or "Sally's Deli".

Scenario

A scenario discovered by our customers had noticed an inconsistency with the renaming scenario as per below.

1. Sync with the Foodmart database

2. Rename the Foodmart database to Foodmart_1:

You will notice while the database is now Foodmart_1, the folder name for the database is still called Foodmart - just like the ID.

3. Import the definition of the Foodmart_1 into Visual Studio and make the changes you would like to make (e.g. new dimensions, changes in measure definitions, etc.)

4. Attempt to deploy this Foodmart_1 database back to the original server and you get an error that it already exists. You do not have the option to overwrite it. The error is in the form of:

Error 1 Errors in the metadata manager. The database with the name of 'foodmart_1' already exists in the 'server_name' server. 0 0

5. Try to deploy the database back as Foodmart and you get the following warning.

The 'foodmart' database already exists on the 'server_name' server. If you proceed with deployment, the database will be overwritten.

Would you like to continue?

Now you can overwrite the database, but you'll have the db named "foodmart" as opposed to what you wanted - foodmart_1.

Discussion

Basically, exemplified in this scenario is that there is a small bug within Visual Studio where it does not differentiate between the ID and the NAME. To work around this problem, you can rename the Foodmart_1 database back to Foodmart before importing it back into Visual Studio. Once you're done with your modifications, you can then re-deploy it back to your server as Foodmart, and then rename it back to Foodmart_1.

Often people will rename a database because they plan to version the databases. For example, over a particular period you have three versions of the same database (e.g. new dimensions, new measures, etc.). In this example, your database is Blah_V2 for a previous version and Blah_V3 for the current version. But you do not want users to need to switch to a different database name each time a new version is created so you want to call it Blah each and every time. To do this, you can build and deploy your database with the ID and NAME of blah_v3. Once you deploy it, you can rename the original "Blah" database back to blah_v2 and rename the current one to "Blah". Please note, there can be issues with this approach because your front-end applications will need its metadata to be refreshed when switching from the V2 to V3 due to any of the changes made.

-------------------

Author: Denny Lee

Reviewed by: Kevin Cox, Lubor Kollar, Baya Pavliashvili (baya_baya@hotmail.com), and Nicholas Birke (nick@birke.ws).

SQL Server 2005 Transactional Replication: Benefit of using SubscriptionStreams for low bandwidth, high latency environments

Sanjay Mishra — Tue, 08 May 2007 00:00:00 GMT

Author: Bren Newman

Introduction

When using Microsoft SQL Server 2005 Transactional Replication, the Log Reader Agent reads the log, constructs SQL statements for published articles and posts it to the Distribution database; the Distribution Agent reads the Distribution database, delivers and applies the "package" to the Subscriber. Often the Publisher, Distributor and Subscribers are on fast networks resulting in very low total delivery latencies. However on slower networks such as in a geographically distributed environment, the connection speeds are often of low bandwidth with high network latencies - this obviously has a negative effect on total delivery time. Network latency (easily seen when using the ping command) has a substantial negative performance impact and using the new -SubscriptionStreams can provide drastic performance increases.

-SubscriptionStreams NN

By default (and all previous editions of Microsoft SQL Server) the Distribution Agent uses as single stream to deliver a transaction from the distribution database to the subscriber database. In Microsoft SQL Server 2005 this default behavior can be overridden by specifying the -SubscriptionStreams NN parameter, where NN can be 0 (for non-SQL Server Subscribers or Peer-to-Peer Transactional subscriptions) to 64; where 64 would be a pretty insane number of threads especially if there are multiple Distribution Agents concurrently executing on the same server. The parameter refers to the number of connections allowed per Distribution Agent used to apply batches of changes in parallel to a Subscriber. It's important to note that even though transactions are parallelized, transactional consistency is maintained. This is achieved by cleverly hashing and "bucketizing" the primary key and re-assembly before committal at the subscriber. However, as noted in BOL (ms-help://MS.SQLCC.v9/MS.SQLSVR.v9.en/rpldata9/html/67084a67-43ff-4065-987a-3b16d1841565.htm), if one of the connections fails to execute or commit, all connections will abort the current batch, and the Distribution Agent will use a single stream to retry the failed batches. Before this retry phase completes, there can be temporary transactional inconsistencies at the Subscriber. After the failed batches are successfully committed, the Subscriber is brought back to a state of transactional consistency.

The requirement

A very large industry-leading online business required data to be replicated from busy OLTP servers via the internet to geographically distributed offices around the world with low latency and be able to recover/catch-up from loss of connection for up to 24 hours. The "last-mile" local connection speeds varied but were generally of poor quality (i.e. high latency) and low speed/bandwidth. The average office (i.e. subscriber) connected at around 512kbps with average latencies of between 200-600 milliseconds. The Publisher (i.e. OLTP server) and remote Distributor are located on a fast local network and the Distributor server is a dual-core four socket processor.

The test

The OLTP publisher was "primed" with hundreds of transactions each consisting of approximately 5 to 10 commands per transaction. The Logreader Agent then moved the replicated transactions into the Distribution database. The Distribution Agent was then started and the duration and throughput measured. The network connection between the Distributor and the subscriber was throttled using Network Throttling software (which interfaced directly with network card). Network bandwidth of 512kbps and 128kbps with latencies of 300 milliseconds was tested.

Test results/findings

On a throttled network with 512kbps bandwidth, 300 millisecond latency and a default of only 1 distribution stream; a throughput of 6.86 commands per second was distributed to the subscriber. When increasing the number of streams to 8; a throughput of 40.34 commands per second was attained - 488% improvement. And when further increasing the number of streams to 16; a throughput of 56.11 commands per second was attained - 718% overall improvement and 39% improvement over 8 streams.

When the network bandwidth was reduced to only 128kps but the network latency kept at 300 milliseconds; 1 distribution stream delivered 5.9 commands per second, 8 distribution streams delivered 31.54 and 16 distribution streams delivered 28.01 commands per second. Chart number 1 shows these results.

Although it's important to notice the hugely positive throughput benefit of using SubscriptionStreams, it's important to note the impact of reduced bandwidth versus increased network latency. The results indicate that network latency has greater negative effect than reduced bandwidth - due to the longer response/acknowledge time for each transaction to be delivered across the network. It's equally important to note that for this test, with very low bandwidth (with high network latency), 16 subscription streams did not improve the delivery rate (in fact, there was a slight decrease). This indicates that bandwidth utilization has been exceeded and there is a negative cost adding more managed parallel threads.

As the requirement included the ability to recover/catch-up from a loss of connection for up to 24 hours, Chart number 2 shows the effect of the positive performance improvement measured in hours. Over a slow network, without Microsoft SQL Server 2005 Subscription Streams, it would have taken up to 11 hours to catch-up whereas with the feature less than 2 hours is possible. This assumes 24 hours worth of transactions or 240,000 commands waiting to be distributed due to network connection outage (or other scenarios).

Conclusion

The SubscriptionStreams feature in SQL Server 2005 Transactional Replication can have a highly positive effect on performance, especially over a slower networks; however it must be cautioned that the more threads/connections are opened and if an "error" occurs during reassembly and or committal of the transactions at the subscriber, the number of streams are reduced to 1 and at a later stage increased back to the configured value. The constant toggling between the two states can be costly. It is highly recommended that the optimal number of streams must be carefully tested for each application and environment before rollout.

Even on a high bandwidth network with low network latency (e.g. 1gigabit and 0 millisecond latency), configuring the Distribution Agent to use SubscriptionStreams can provide significant performance benefits - as illustrated in Chart 3. Clearly this is a feature worth testing and possibly deploying.

Check out the exciting new SQL Server Best Practices website.....

thomasda — Sat, 09 Dec 2006 01:31:00 GMT

http://www.microsoft.com/technet/prodtechnol/sql/bestpractice/default.mspx

Get the real-world guidelines, expert tips, and rock-solid guidance to take your SQL Server implementation to the next level. Drawing on the extensive experience and expertise from respected developers and engineers at Microsoft as they walk you through the specifics on solving particularly difficult issues.

Managing Schema Changes (Part 2)

thomasda — Fri, 31 Mar 2006 19:10:00 GMT

SQLCAT Blog: Managing Schema Changes (Part 2)

Question: What are best practices for managing schema changes in SQL Server 2005? What are the fastest, least intrusive techniques for high availability?

Answer: In Part 1, we outlined the behavior of SQL Server 2005 schema changes. In Part 2, we will look at best practices techniques to de-couple the batch updates required (see “Managing Schema Changes Part 1”) for data type and most length changes. These techniques allow us to make schema changes quickly and with least impact to performance and concurrency for large tables like the Billion row scenario.

Scenario: Let’s assume we have a 5 character ZipCode column that we want to increase to 10 character format e.g. 12345-0001. We can change the ZipCode length from char(5) to char (10) using:

alter table alter column ZipCode Char(10)

However, this will incur a concurrency crippling batch update (as seen in Profiler) to change the column length in our Billion row table scenario. If you’d like to capture this in Profiler, see “Managing Schema Changes (Part 1).

Best Practice

The use of a NULL property (excluding DEFAULT WITH VALUES of course) is a best practice insofar as columns can be added quickly. In the case where you do need to set a value to the column, the NULL property also gives us the flexibility of de-coupling the update from the alter table statement.

An approach to accomplish this:

Add a char(10) column, NewZip
Set piecemeal batch size
Populate NewZip using an explicit conversion of ZipCode
Drop old column ZipCode.
Rename NewZip to ZipCode (optional)

Step 1: Add new column

alter table MyTable

add NewZip char(10) NULL

Upon the fast completion of this alter table (about 1 second in my tests), NewZip contains NULLs for all rows in the existing table. You may need to populate NewZip with the values from another column or other values. In step 2, SET ROWCOUNT is used to minimize the concurrency impact of write locks by limiting the number of rows that are affected at one time. An alternative to SET ROWCOUNT is a WHERE clause that specifies certain key ranges, limiting the number of rows affected at one time.

Step 2: Set Batch Size for update

--- (2) set piecemeal batch size

set nocount on

SET ROWCOUNT 1000

Next, the piecemeal update is repeatedly done until there are no NULL values in NewZip. This example sets all rows using the char(5) ZipCode column. Alternately, you can selectively set the value of an added column using a lookup or join.

Step 3: De-coupled Piecemeal Update

declare @rowcount int, @batchcount int, @rowsupdated int

select @rowcount = 1,@batchcount = 0, @rowsupdated = 0

while @rowcount > 0

begin

begin tran

update MyTable

set NewZip = ZipCode

where NewZip is NULL

set @rowcount = @@rowcount

select @rowsupdated = @rowsupdated + @rowcount

if @rowcount > 0 select @batchcount = @batchcount + 1

commit

end

Step 4: drop the char(5) column ZipCode (optional)

alter table MyTable

drop column ZipCode

Step 5: Rename the NewZip column to ZipCode (optional)

exec sp_rename 'MyTable.NewZip','ZipCode','COLUMN'

--- Observe the data type for ZipCode is now char(10)

sp_help MyTable

Trigger for ZipCode

What happens if a user changes a customer ZipCode AFTER the NewZip update? If the user application updates ZipCode char(5) during the piecemeal update process, NewZip may have already been updated with the old zip code value, and thus is out of synch with ZipCode. An update trigger can be used to keep ZipCode and NewZip in synch during the piecemeal update process. If you intend to drop the old char(5) ZipCode (in Step 4), and then rename NewZip to ZipCode (in Step 5), the update trigger will no longer be needed after the piecemeal update.

Explicit data type conversion

Some data type changes require explicit conversion. That is, they cannot be implicitly converted using alter table alter column. For example, an explicit conversion is required to change a datetime datatype to an int or numeric(8) YYYYMMDD format. This scenario can be handled using a convert (or datepart) function in exactly the same manner as the piecemeal update scenario above.

Dropping columns

Dropping columns is a schema only operation. There are no concurrency issues unless you move the object to reclaim space from the dropped column.

Unique and Primary Key Constraints

Adding Unique and Primary Key constraints involves reads, sorts, and writes. By default UNIQUE constraints add nonclustered indexes while PRIMARY KEY adds a clustered index. Obvious performance and concurrency issues would occur when you add a PRIMARY KEY (and default CLUSTERED INDEX) to a large table that already has UNIQUE (and nonclustered index) constraints. This is because adding a Primary Key (and clustered index) would rebuild the nonclustered indexes replacing RIDs (row IDs) with Primary Keys.

Alter table, create and alter index includes ONLINE options for managing unique and primary key constraints and indexes. Where possible, for best performance, highest concurrency and availability, ONLINE options should be used for managing indexes. Otherwise, it would wise to postpone these operations to batch maintenance windows.

Conclusions and best practices:

SQL Server 2005 does support schema changes however some schema changes are faster than others. The fastest schema changes occur when:

Adding column with NULL property
Adding column with NULL property and DEFAULT
Changing NOT NULL to NULL property
Adding DEFAULT constraint
Dropping CHECK or DEFAULT constraint

A schema change best practice for performance and concurrency of large tables includes adding columns with NULL properties. In the case where you need to set a value to the newly added column, starting with a NULL property provides the flexibility where the update can be de-coupled from the alter table statement. Far preferable to a batch update that can significantly affect concurrency on large tables; a piecemeal update can be used to limit the number of rows updated at one time. One technique to limit the number of rows is SET ROWCOUNT. This greatly reduces (or effectively eliminates) the blocking impact of the piecemeal update. If it is important to disallow NULL values after the piecemeal update completes, the application would be have to enforce this.

The inclusion of CHECK constraints requires a read of the entire table to enforce the CHECK constraint. The accompanying table scan can impact write activity for a large table.

Other schema changes present performance and concurrency challenges for large tables due to accompanying batch update operations. Those include:

Adding NULL column with DEFAULT and WITH VALUES clauses
Adding NOT NULL column
Changing NULL to NOT NULL property

Using the Piecemeal Update best practice with NULL columns allows us to avoid the undesirable associated activities occurring with some schema changes such as unintended reads and writes, enabling us to utilize the fastest, least intrusive techniques for high concurrency and availability.

Tom Davidson

SQL Server Customer Advisory Team

Managing Schema Changes (Part 1)

thomasda — Thu, 02 Mar 2006 00:22:00 GMT

Question: What are best practices for managing schema changes in SQL Server 2005? What are the fastest, least intrusive techniques for high availability?

Answer: A common requirement as business requirements evolve over time is managing schema changes. While SQL Server 2005 supports the changing of schema (e.g. adding columns and constraints, changing column data types and properties, and dropping columns and constraints), in Managing Schema Changes (Part 1) we will focus on performance and concurrency issues that should be considered for large tables in typical scenarios. In Managing Schema Changes (Part 2) we will cover additional best practice techniques to maximize performance and concurrency.

Let’s assume we have a Billion row table which is quite common these days. Schema changes have to be completed in the fastest, least intrusive manner with high availability.

Some schema changes include implicit reads and writes. These operations, extending the duration of the schema change transaction, can reduce concurrency on large tables by taking shared locks for reads and exclusive locks for writes. For example, adding a CHECK constraint requires a scan to enforce the CHECK constraint. Implicit updates are performed for schema changes involving NOT NULLS, base data type changes or column length changes. These implicit reads and writes can be seen either via Profiler statement level tracing or by using SET STATISTICS IO ON.

To trace these extra operations in Profiler, select the following

TSQL event, select SQL:StmtStarting and SQL:StmtCompleted.
Stored Procedure event, select SP:StmtStarting and SP:StmtCompleted

Adding columns – alter table MyTable add NewColumn……

#	ColumnProperty	Column Constraints	Profiler Observations (StmtStarting & StmtCompleted)	Performance & Concurrency impact
1	NULL	N/A alter table MyTable add MySSN numeric(9,0) NULL		Fastest. Best performance and concurrency. Metadata only change.
2	NULL	DEFAULT alter table MyTable add MyDate2 datetime NULL default getdate()		Fastest. Best performance and concurrency Metadata only change.
3	NULL	CHECK alter table MyTable add MyPct3 numeric(4,1) NULL constraint MyChk3 check (MyPct3 between 0 and 100)	Samples stats, then reads entire table to enforce check constraint.	Faster than NOT NULL, but does do reads for check constraint. In large tables, the read activity for the check constraint could cause blocking for writes.
4	NULL	CHECK, DEFAULT alter table MyTable add MyInt4 int NULL default 0 constraint myChk4 check (MyInt4 between 0 and 100)	Samples stats, then reads entire table to enforce check constraint.	Faster than NOT NULL, but does do reads for check constraint. In large tables, the read activity for the check constraint could cause block writes.
5	NULL	DEFAULT WITH VALUES alter table MyTable add MyInt5 int NULL default 0 with values	Updates entire table with default value UPDATE [Northwind].[dbo].[MyTable] SET [MyInt5] = DEFAULT	Slower than NULL & DEFAULT (see #2) because the WITH VALUES clause does an update of all rows in existing table. In large tables, the batch update could cause concurrency issues.
6	NULL	CHECK, DEFAULT WITH VALUES alter table MyTable add MyInt6 int NULL default 0 with values constraint myChk6 check (MyInt6 between 0 and 100)	Updates entire table for default value, samples stats, then reads entire table to enforce check constraint. UPDATE [Northwind].[dbo].[MyTable] SET [MyInt6] = DEFAULT	Slowest because WITH VALUES clause requires updates with default values & reads for check constraint. There could be significant concurrency issues in large tables due to the batch update for the default value and read activity for check constraint.
7	NOT NULL	DEFAULT alter table MyTable add MyDate7 datetime NOT NULL constraint myDft7 default getdate()	NOT NULL requires update of entire table for default value UPDATE [Northwind].[dbo].[MyTable] SET [MyDate7] = DEFAULT	Slower than NULL (see #2) because NOT NULL requires updates for default values. Possible concurrency issues with large tables since batch update holds exclusive locks.
8	NOT NULL	CHECK, DEFAULT alter table MyTable add MyInt8 int NOT NULL constraint myDft8 default 0 constraint myChk8 check (MyInt8 between 0 and 100)	Updates entire table for default value, samples stats, then reads entire table to enforce check constraint. UPDATE [Northwind].[dbo].[MyTable] SET [MyInt8] = DEFAULT	Slowest due to updates for default values & reads for check constraint There could be significant concurrency issues in large tables due to the batch update for the default value and read activity for check constraint.

The fastest way to add a new column to a large table (with or without the DEFAULT constraint) is to use the NULL property (see example #1 and #2 above). Although there is a DEFAULT constraint in example #2, MyDate will contain NULLs for all rows.

Including a CHECK constraint on a large table will cause a read of the entire table to enforce the CHECK constraint (see examples #3, #4, #6 and #8). Note that while examples #3 (MyPct3) and #4 (MyInt4) contain NULL values for the newly added columns, a NULL value will not fail the CHECK constraint. NULL just simply means that we don’t know the value. However if the value IS known, the CHECK constraint forces the known value to comply with the check requirement.

Batch updates occur with examples #5 through #8. If the WITH VALUES clause is added to NULL and DEFAULT (see example #5 and #6), all rows will be updated with the DEFAULT value. Adding a column with NOT NULL property (see examples #7 and #8) to an existing table requires a DEFAULT constraint. The NOT NULL / DEFAULT combination causes a batch update to occur. Net, batch updates hold exclusive locks and reduce concurrency, falling short of our performance and availability goals.

In addition to the writes, NULL example #6 and NOT NULL #8 add CHECK constraints that require reads to enforce the CHECK, making these the worst of the bunch from a concurrency, performance and availability point of view.

Changing column datatypes, default and check constraints, NULL properties

You can change a column’s base datatype, or change its length (for numeric includes scale and precision) using the alter table alter column statement. In these cases, performance and concurrency is affected by the accompanying batch update. If a NULL property is changed to NOT NULL, a batch update also occurs. The exception is when changing a NOT NULL property to NULL. This is a fast metadata only operation.

Changing column property, type & length, adding column constraint – alter table MyTable alter column

Change		Profiler Observations
Property Change	NOT NULL to NULL alter table MyTable alter column MyDate7 datetime NULL		Fast, metadata only change
Property Change	NULL to NOT NULL alter table MyTable alter column MyDate7 datetime NOT NULL	Performs batch update UPDATE [Northwind].[dbo].[MyTable] SET [MyDate7] = [MyDate7]	In a large table, concurrency issues could occur due to the accompanying batch update.
Add Default	ALTER TABLE MyTable ADD CONSTRAINT MyDft DEFAULT 0 FOR MySSN		Fast, metadata only change
Drop Default	ALTER TABLE MyTable DROP CONSTRAINT MyDft		Fast, metadata only change
Add Check	ALTER TABLE MyTable ADD CONSTRAINT MyDtChk Check (MyDate2 <= getdate())	Samples stats, then reads entire table to enforce check constraint.	In a large table, concurrency issues could occur due to the accompanying reads for the check constraint.
Drop Check	ALTER TABLE MyTable DROP CONSTRAINT MyDtChk		Fast, schema only change
Increase Length	NUMERIC(9) to (11) ALTER TABLE MyTable ALTER COLUMN MySSN numeric(11,0)	Performs update UPDATE [Northwind].[dbo].[MyTable] SET [MySSN] = [MySSN]	In a large table, concurrency issues could occur due to the accompanying batch update
Data type Change	NUMERIC to INT ALTER TABLE MyTable ALTER COLUMN MySSN int	Performs update UPDATE [Northwind].[dbo].[MyTable] SET [MySSN] = [MySSN]	In a large table, concurrency issues could occur due to the accompanying batch update
Data type Change	INT to VARCHAR(9) ALTER TABLE MyTable ALTER COLUMN MySSN varchar(9)	Performs update UPDATE [Northwind].[dbo].[MyTable] SET [MySSN] = [MySSN]	In a large table, concurrency issues could occur due to the accompanying batch update
Increase Length	Varchar(9) to Varchar(15) ALTER TABLE MyTable ALTER COLUMN MySSN varchar(15)		Fast, metadata only change
Decrease Length	Varchar(15) to Varchar(10) ALTER TABLE MyTable ALTER COLUMN MySSN varchar(10)	Performs update UPDATE [Northwind].[dbo].[MyTable] SET [MySSN] = [MySSN]	In a large table, concurrency issues could occur due to the accompanying batch update
Data type change	Varchar(15) to Char(15) ALTER TABLE MyTable ALTER COLUMN MySSN char(15)	Performs update UPDATE [Northwind].[dbo].[MyTable] SET [MySSN] = [MySSN]	In a large table, concurrency issues could occur due to the accompanying batch update
Increase Length	Char(15) to char(20) ALTER TABLE MyTable ALTER COLUMN MySSN char(20)	Performs update UPDATE [Northwind].[dbo].[MyTable] SET [MySSN] = [MySSN]	In a large table, concurrency issues could occur due to the accompanying batch update

Dropping columns

Dropping columns is a schema only operation. There are no concurrency issues unless you move the object to reclaim space from the dropped column.

Unique and Primary Key Constraints

Conclusions and best practices:

We can effectively manage schema changes in SQL Server 2005 by understanding its behavior. This insight helps us avoid the undesirable associated activities occurring with some schema changes such as unintended reads and writes, enabling us to utilize the fastest, least intrusive techniques for high concurrency and availability.

The fastest schema changes occur when:

Adding a column with NULL property
Adding a column with NULL property and DEFAULT
Changing NOT NULL to NULL property
Adding DEFAULT constraint
Dropping CHECK or DEFAULT constraint

The slowest schema changes have accompanying read and write operations such as:

CHECK constraints require a read of the entire table to enforce the CHECK.
Batch updates occur when adding a NULL column in combination with DEFAULT and WITH VALUES clauses
Batch updates occur when adding NOT NULL column (to provide Default value)
Batch updates occur when changing NULL to NOT NULL property
Batch updates occur when column datatypes are changed or when lengths change. The only exception is increasing the length of varchar.

In conclusion, a schema change best practice for performance and concurrency of large tables includes adding columns with NULL properties. If it’s important to disallow NULL values, the application would have to do this. In “Managing Schema Changes (Part 2)”, we’ll see how concurrency and performance can be maximized by de-coupling the batch update of a newly added column using the NULL property. The batch update is then replaced with a piecemeal update for high concurrency and performance. This technique can also be used to change the type or length of columns without incurring the concurrency crippling batch update.

Tom Davidson

SQL Server Customer Advisory Team

How can SQL Server 2005 help me evaluate and manage indexes?

thomasda — Mon, 12 Dec 2005 18:43:00 GMT

SQLCAT Blog:

Question: How can SQL Server 2005 help me evaluate and manage indexes?

(1) How can I find out whether my indexes are useful? How are they used?

(2) Do I have any tables or indexes that are not used (or rarely)

(3) What is the cost of index maintenance vs. its benefit?

(4) Do I have hot spots & index contention?

(5) Could I benefit from more (or less) indexes?

Answer:

SQL Server 2005 Dynamic Management Views (DMVs) are important insofar as they expose changing server state information that typically spans many sessions, many transactions, and many requests. DMVs provide a level of transparency that was not available in SQL Server 2000 and can be used for diagnostics, memory and process tuning, and monitoring. The SQL Server engine tracks detailed resource history in DMVs that can be queriable with SELECT statements but are not persisted to disk. Thus DMVs reflect activity since the last SQL Server recycle.

Since indexes provide an alternative to a table scan, and because DMVs expose index usage counters, we can compare the cost vs. the benefit of indexes. This comparison will include the maintenance cost of keeping the index up to date, versus the benefit of reads e.g. when an index can be used in lieu of a table scan. Keep in mind that an update or delete operation involves both a read which is first required to determine whether a row qualifies for the update operation, and a write if a row is found to qualify. In an insert operation, only writes will be performed on all indexes. Consequently, in an insert-intensive workload, writes will exceed reads. In an update-intensive (updates and deletes) workload, read and write counts are generally close assuming there are not a lot of ‘records not found’. In read-intensive workloads, read counts will exceed write counts. Referential constraints such as foreign keys require other read activity (for inserts, updates, and deletes) to ensure referential integrity is maintained.

(1) How can I find out whether my indexes are useful? How are they used?

First, we will determine whether indexes are ‘useful’. DDL is used to create objects (such as indexes) and update the catalog. Creating the index does not constitute ‘use’ of the index, and thus the index will not be reflected in the index DMVs until the index is actually used. When an index is used by a Select, Insert, Update, or Delete, its use is captured by sys.dm_db_index_usage_stats. If you have run a representative workload, all useful indexes will have been recorded in sys.dm_db_index_usage_stats. Thus, any index not found in sys.dm_db_index_usage_stats is unused by the workload (since the last re-cycle of SQL Server). Unused indexes can be found as follows:

(2) Do I have any tables or indexes that are not used (or rarely used)?

------ unused tables & indexes. Tables have index_id’s of either 0 = Heap table or 1 = Clustered Index

Declare @dbid int

Select @dbid = db_id('Northwind')

Select objectname=object_name(i.object_id)

, indexname=i.name, i.index_id

from sys.indexes i, sys.objects o

where objectproperty(o.object_id,'IsUserTable') = 1

and i.index_id NOT IN (select s.index_id

from sys.dm_db_index_usage_stats s

where s.object_id=i.object_id and

i.index_id=s.index_id and

database_id = @dbid )

and o.object_id = i.object_id

order by objectname,i.index_id,indexname asc

Rarely used indexes will appear in sys.dm_db_index_usage_stats just like heavily used indexes. To find rarely used indexes, you look at columns such as user_seeks, user_scans, user_lookups, and user_updates.

--- rarely used indexes appear first

declare @dbid int

select @dbid = db_id()

select objectname=object_name(s.object_id), s.object_id, indexname=i.name, i.index_id

, user_seeks, user_scans, user_lookups, user_updates

from sys.dm_db_index_usage_stats s,

sys.indexes i

where database_id = @dbid and objectproperty(s.object_id,'IsUserTable') = 1

and i.object_id = s.object_id

and i.index_id = s.index_id

order by (user_seeks + user_scans + user_lookups + user_updates) asc

(3) What is the cost of index maintenance vs. its benefit?

If a table is heavily updated and also has indexes that are rarely used, the cost of maintaining the indexes could exceed the benefits. To compare the cost and benefit, you can use the table valued function sys.dm_db_index_operational_stats as follows:

--- sys.dm_db_index_operational_stats

declare @dbid int

select @dbid = db_id()

select objectname=object_name(s.object_id), indexname=i.name, i.index_id

, reads=range_scan_count + singleton_lookup_count

, 'leaf_writes'=leaf_insert_count+leaf_update_count+ leaf_delete_count

, 'leaf_page_splits' = leaf_allocation_count

, 'nonleaf_writes'=nonleaf_insert_count + nonleaf_update_count + nonleaf_delete_count

, 'nonleaf_page_splits' = nonleaf_allocation_count

from sys.dm_db_index_operational_stats (@dbid,NULL,NULL,NULL) s,

sys.indexes i

where objectproperty(s.object_id,'IsUserTable') = 1

and i.object_id = s.object_id

and i.index_id = s.index_id

order by reads desc, leaf_writes, nonleaf_writes

--- sys.dm_db_index_usage_stats

select objectname=object_name(s.object_id), indexname=i.name, i.index_id

,reads=user_seeks + user_scans + user_lookups

,writes = user_updates

from sys.dm_db_index_usage_stats s,

sys.indexes i

where objectproperty(s.object_id,'IsUserTable') = 1

and s.object_id = i.object_id

and i.index_id = s.index_id

and s.database_id = @dbid

order by reads desc

The difference between sys.dm_db_index_usage_stats and sys.dm_db_index_operational_stats is as follows. Sys.dm_db_index_usage_stats counts each access as 1, whereas sys.dm_db_index_operational_stats counts depending on the operation, pages or rows.

(4) Do I have hot spots & index contention?

Index contention (e.g. waits for locks) can be seen in sys.dm_db_index_operational_stats. Columns such as row_lock_count, row_lock_wait_count, row_lock_wait_in_ms, page_lock_count, page_lock_wait_count, page_lock_wait_in_ms, page_latch_wait_count, page_latch_wait_in_ms, pageio_latch_wait_count, pageio_latch_wait_in_ms detail lock and latch contention in terms of waits. You can determine the average blocking and lock waits by comparing waits to counts as follows:

declare @dbid int

select @dbid = db_id()

Select dbid=database_id, objectname=object_name(s.object_id)

, indexname=i.name, i.index_id --, partition_number

, row_lock_count, row_lock_wait_count

, [block %]=cast (100.0 * row_lock_wait_count / (1 + row_lock_count) as numeric(15,2))

, row_lock_wait_in_ms

, [avg row lock waits in ms]=cast (1.0 * row_lock_wait_in_ms / (1 + row_lock_wait_count) as numeric(15,2))

from sys.dm_db_index_operational_stats (@dbid, NULL, NULL, NULL) s, sys.indexes i

where objectproperty(s.object_id,'IsUserTable') = 1

and i.object_id = s.object_id

and i.index_id = s.index_id

order by row_lock_wait_count desc

The following report shows blocks in the [Order Details] table, index OrdersOrder_Details. While blocks occur less than 2 percent of the time, when they do occur, the average block time is 15.7 seconds.

It would be important to track this down using the SQL Profiler Blocked Process Report. You can set the Blocked Process Threshold to 15 using sp_configure ‘Blocked Process Threshold’,15. Afterwards, you can run a trace to capture blocks over 15 seconds.

The Profiler trace will include the blocked and blocking process. The advantage of tracing for long blocks is the blocked and blocking details can be saved in the trace file and can be analyzed long after the block disappears. Historically, you can see the common causes of blocks. In this case the blocked process is the stored procedure NewCustOrder. The blocking process is the stored procedure UpdCustOrderShippedDate.

The caveat with Profiler Trace of Blocked Process Report is that in the case of stored procedures, you cannot see the actual statement within the stored procedure that is blocked. You can capture the actual blocked statement of a stored procedure in realtime (as it is occuring) using the following:

create proc sp_block_info

select t1.resource_type as [lock type]

,db_name(resource_database_id) as [database]

,t1.resource_associated_entity_id as [blk object]

,t1.request_mode as [lock req] --- lock requested

,t1.request_session_id as [waiter sid] --- spid of waiter

,t2.wait_duration_ms as [wait time]

,(select text from sys.dm_exec_requests as r --- get sql for waiter

cross apply sys.dm_exec_sql_text(r.sql_handle)

where r.session_id = t1.request_session_id) as waiter_batch

,(select substring(qt.text,r.statement_start_offset/2,

(case when r.statement_end_offset = -1

then len(convert(nvarchar(max), qt.text)) * 2

else r.statement_end_offset end - r.statement_start_offset)/2)

from sys.dm_exec_requests as r

cross apply sys.dm_exec_sql_text(r.sql_handle) as qt

where r.session_id = t1.request_session_id) as waiter_stmt --- statement blocked

,t2.blocking_session_id as [blocker sid] -- spid of blocker

,(select text from sys.sysprocesses as p --- get sql for blocker

cross apply sys.dm_exec_sql_text(p.sql_handle)

where p.spid = t2.blocking_session_id) as blocker_stmt

from

sys.dm_tran_locks as t1,

sys.dm_os_waiting_tasks as t2

where

t1.lock_owner_address = t2.resource_address

exec sp_block_info

(5) Could I benefit from more (or less) indexes?

Remembering that indexes involve both a maintenance cost and a read benefit, the overall index cost benefit can be determined by comparing reads and writes. Reading an index allows us to avoid table scans however they do require maintenance to be kept up-to-date. While it is easy to identify the fringe cases where indexes are not used, and the rarely used cases, in the final analysis, index cost benefit is somewhat subjective. The reason is the number of reads and writes are highly dependent on the workload and frequency. In addition, qualitative factors beyond the number of reads and writes can include a highly important monthly management report or quarterly VP report in which the maintenance cost is of secondary concern.

Writes of all indexes are performed for inserts, but there are no associated reads (unless there are referential constraints). Besides select statements, reads are performed for updates and deletes, writes are performed if rows qualify. OLTP workloads have lots of small transactions, frequently combining select, insert, update and delete operations. Data Warehouse activity is typically separated into batch windows having a high concentation of write activity, followed by an on-line window of read activity.

SQL Statement	Read	Write
Select	Yes	No
Insert	No	Yes, all indexes
Update	Yes	Yes, if row qualifies
Delete	Yes	Yes, if row qualifies

In general, you want to keep indexes to a funtional minimum in a high transaction OLTP environment due to high transaction throughput combined with the cost of index maintenance and potential for blocking. In contrast, you pay for index maintenance once during the batch window when updates occur for a data warehouse. Thus, data warehouses tend to have more indexes to benefit its read-intensive on-line users.

In conclusion, an important new feature of SQL Server 2005 includes Dynamic Management Views (DMVs). DMVs provide a level of transparency that was not available in SQL Server 2000 and can be used for diagnostics, memory and process tuning, and monitoring. DMVs can be useful in answering practical questions such as index usage, cost benefit of indexes, and index hot spots. Finally, DMVs are queriable with SELECT statements but are not persisted to disk. Thus they reflect changing server state information since the last SQL Server recycle.

How can SQL Server 2005 OnLine Piecemeal Restore improve availability?

thomasda — Thu, 27 Oct 2005 11:21:00 GMT

Question 1: How can SQL Server 2005 OnLine Piecemeal Restore improve availability?

Question 2: Can I recover a single object such as a table or specific partition(s) from a partitioned table?

Answer:

The question is how to get the greatest flexibility, manageability, availability and recoverability for large databases. Anytime you need to restore a database to a specific point-in-time, all filegroups in that database must be restored to the same point-in-time. Restoring an entire multi-terabyte database to a point-in-time could result in a big outage, an especially severe penalty if the data to be restored is relatively small compared to the entire database. While SQL Server 2000 provided filegroup restores, all filegroups must be available for the database to be accessible.

To provide greater flexibility for large DB scenarios, a new feature of SQL Server 2005 called OnLine Piecemeal Restore allows us to use specific filegroup backups to create a smaller work database containing only the data we wish to restore. By placing tables (including individual partitions of partitioned tables) on specific filegroups, we can devise a granular backup and restore strategy where part of a database such as a table or even a single partition of a partitioned table, can be restored quickly. We can also roll transactions forward to a specific point-in-time with transaction logs using any of the following: a specific time, a transaction mark, or a log sequence number (LSN).

The example database, ProdDB has the following secondary filegroups:

· Customers: FGCustomers

· Current year orders (2005): FGOrders

· Historical orders (2000-2004): FGOrderHistory.

ProdDB contains three large user tables: Customers, Orders and OrderDetails. The Customers table is placed on filegroup FGCustomers. The table Orders is partitioned by year. The current years’ orders (2005) are placed on FGOrders, while years 2000-2004 are placed on filegroup FGOrderHistory. The layout of ProdDB is as follows:

Primary filegroup

FGCustomers (RW)

FGOrders (RW)

FGOrderHistory (RO)

Contains system tables only, no user tables

Customers

Customers data

Orders & OrderDetails

2005 Orders partition

OrderDetails

OrderHistory

2000-2004 Orders partitions

Customers

The combination of filegroup backups and rolling transaction logs forward provide recoverability to a point-in-time. Assume that a batch job has resulted in incorrect 2005 data and that you must perform a partial filegroup restore of FGOrders (shaded in gray) to recover the data.

Steps for partial online filegroup restore to a point-in-time:

Backup ProdDB’s filegroups primary, FGCustomers, FGOrders, FGOrderHistory
Batch jobs run (assume batch job messes up 2005 data……)
Backup tail of transaction log (assuming you need to roll forward to point-in-time in the log)
Perform piecemeal restore. Use ProdDB primary & specific filegroup (FGOrders) backup to create a new work database called PartialDB. Roll forward transaction log to point in time (just before bad batch job). The 2005 partition of Orders was recovered from the FGOrders backup.
Delete bad data in ProdDB.
Copy restored partition or table from work database PartialDB to ProdDB

Example of filegroup & transaction log backups, bad batch job, and online piecemeal restore

1. Backup ProdDB filegroups & transaction log:

BACKUP DATABASE ProdDB

FILEGROUP = 'PRIMARY'

to disk = 'C:\ProdDB_FG_PRIMARY_SQL.BAK' with init

-- SQL FGCustomers FILEGROUP backup

BACKUP DATABASE ProdDB

FILEGROUP = 'FGCustomers'

to disk = 'C:\ProdDB_FG_FGCustomers_SQL.BAK' with init

-- SQL FGOrders FILEGROUP backup

BACKUP DATABASE ProdDB

FILEGROUP = 'FGOrders'

to disk = 'C:\ProdDB_FG_FGOrders_SQL.BAK' with init

-- SQL FGOrderHistory FILEGROUP backup

BACKUP DATABASE ProdDB

FILEGROUP = 'FGOrderHistory'

to disk = 'C:\ProdDB_FG_FGOrderHistory_SQL.BAK' with init

-- SQL log backup

BACKUP log ProdDB

to disk = 'C:\ProdDB_LOG_SQL1.BAK' with init

2. Assume a batch job messes up some 2005 data

-- assume a batch job wrongly updates or deletes some 2005 data

DELETE from [ProdDB]..Orders

where OrderID in (select top 50 OrderID from [ProdDB]..Orders where OrderDate > '2005-01-30')

3. Backup tail of log:

-- Backup tail of transaction log (assuming you need to roll forward to point-in-time in the log)

BACKUP log ProdDB

to disk = 'C:\ProdDB_LOG_SQL2.BAK' with init

4. OnLine Piecemeal Restore creates work database PartialDB:

-- PartialDB will be created from ProdDB backups.

-- MOVE used to create files for PartialDB

RESTORE DATABASE PartialDB

from disk = 'C:\ProdDB_FG_PRIMARY_SQL.BAK'

WITH NORECOVERY, REPLACE, PARTIAL – partial:not all FG will be restored

,MOVE 'ProdDB_data' to 'C:\Program Files\Microsoft SQL Server\MSSQL.1\MSSQL\DATA\PartialDBPri1dat.mdf'

,MOVE 'SProdDB_log' to 'C:\Program Files\Microsoft SQL Server\MSSQL.1\MSSQL\DATA\PartialDBlog.ldf'

-- restore only the latest 2005 data, oldest partitions on FGOrdersHistory not restored

RESTORE DATABASE PartialDB

from disk = 'C:\ProdDB_FG_FGOrders_SQL.BAK'

WITH NORECOVERY

,MOVE 'FGOrders_file1' to 'C:\Program Files\Microsoft SQL Server\MSSQL.1\MSSQL\DATA\PartialDBOrdFi1dt.ndf'

-- restore first log up to point-in-time. You can also restore to a transaction mark or LSN.

RESTORE log PartialDB

from disk = 'C:\ProdDB_LOG_SQL1.BAK'

WITH NORECOVERY, STOPAT = 'Oct 26, 2005 11:00 PM'

-- restore second log up to point-in-time. You can also restore to a transaction mark or LSN.

RESTORE log PartialDB

from disk = 'C:\ProdDB_LOG_SQL2.BAK'

WITH RECOVERY, STOPAT = 'Oct 26, 2005 11:00 PM'

-- NOTE: Piecemeal restore. Part of PartialDB is recovered & online.

select file_id, name, physical_name, state, state_desc

from sys.database_files

Note if you choose to use a transaction mark to establish the point-in-time to recover to, remember that the transaction must be writable, e.g. it must contain an insert, update, or delete within the transaction for the transaction mark to be written to the log. A transaction mark with select statements only is not a writeable transaction.

Note that the newly created work database PartialDB contains only 3 files from 2 filegroups. A backup of the Primary filegroup restores the files ProdDB_data (contains system catalog tables) and SProdDB_log which is the transaction log. The secondary filegroup FGOrders is used to recreate FGOrders_file1. FGOrders contains only the 2005 Orders data.

File_id	Name	Physical_name	State	State_Desc
1	ProdDB_data	C:\Program Files\Microsoft SQL Server\MSSQL.1\MSSQL\DATA\PartialDBPri1dat.mdf	0	ONLINE
2	SProdDB_log	C:\Program Files\Microsoft SQL Server\MSSQL.1\MSSQL\DATA\PartialDBlog.ldf	0	ONLINE
3	FGCustomers_file1	C:\Program Files\Microsoft SQL Server\MSSQL.1\MSSQL\DATA\ProdDBCustFi1dt.ndf	3	RECOVERY_PENDING
4	FGOrders_file1	C:\Program Files\Microsoft SQL Server\MSSQL.1\MSSQL\DATA\PartialDBOrdFi1dt.ndf	0	ONLINE
5	FGOrderHistory_file1	C:\Program Files\Microsoft SQL Server\MSSQL.1\MSSQL\DATA\ProdDBOrdHistFi1dt.ndf	3	RECOVERY_PENDING

PartialDB special notes:

The primary filegroup contains the system tables that define objects for the database. As a best practice and to expedite the restore process, it is suggested that user objects are placed on secondary filegroups, not the primary filegroup. Once the primary filegroup is restored, the system tables contain the metadata for all objects in the database, although you may not be actually restoring all objects.

While you can select data from any ONLINE filegroup, remember that some objects such as a partitioned table, can span filegroups. If some but not all partitions of a partitioned table (such as in the example above) have been recovered, an unqualified SELECT * FROM Orders, or any other object that has not been recovered will likewise fail as follows:

Msg 679, Level 16, State 1, Line 1

One of the partitions of index '' for table 'dbo.Orders'(partition ID 72057594038452224) resides on a filegroup that cannot be accessed because it is offline, restoring, or defunct. This may limit the query result.

5. Delete bad data in ProdDB.

Before restoring the data in ProdDB , you must remove the bad data in ProdDB using various techniques such as

a. drop table (if you plan on using select into to recreate the object)

b. truncate table to remove all rows or delete to remove specific rows using a WHERE clause

c. switch out (for partitioned tables). A partition must be empty (using switch out) before you can switch the new data in. Once switched out, you can drop the bad data.

6. Copy partition or table from PartialDB to ProdDB.

Use copy techniques to restore the data in ProdDB. Recreate indexes on the copied object, add any constraints, etc. Some copy alternatives to consider:

a. select * into ProdDB..OrderDetails from PartialDB..OrderDetails (if you are restoring the entire table, drop table in ProdDB first)

b. insert into ProdDB..Orders select * from PartialDB..Orders where OrderDate >= '2005-01-30'

c. create work table on correct filegroup, copy data, switch in (for partitioned tables). Note: The definition of the work source table and target partitioned table must match exactly. The target partition must be empty. The source table (used in the switch in) must be on the same filegroup as the partition.

If you restore an object to a specific point-in-time, you may have to consider the effects on referential integrity (RI). You may have to perform RI checks to ensure relationships are intact.

Summary:

In conclusion, SQL Server 2005 provides a flexible filegroup backup strategy that should be especially helpful in large database scenarios. By placing tables (including individual partitions of partitioned tables) on specific filegroups, we can devise a granular backup and restore strategy where we can restore part of a database quickly to a smaller work database. Using the techniques described above, we can recover entire tables or specific table partitions. Combined with transaction marks, we can restore to a specific point-in-time. After the object has been restored to a work database, the last step is to copy it to the production database. After the copy, the work database can be dropped.

Tom Davidson

SQL Server Customer Advisory Team