SQL Premier Field Engineering

Index Maintenance across all Databases in an Instance

Menzo Steinhorst — Thu, 30 Jul 2009 11:48:00 GMT

I needed to make a job for a customer that rebuilds all indexes in an instance that have a fragmentation of more than 10%. Usually we use a maintenance plan for rebuilding indexes. Although that is a fine approach is has one disadvantage: it rebuilds all indexes in a database regardless of the level of fragmentation. I do not like that because it takes up resources that are not necessary. It also takes far longer to complete than my job.

As a basis I used the code from the BOL on the sys.dm_db_index_physical_stats page. Here you can find the following piece of code:

-- Ensure a USE <databasename> statement has been executed first.

SET NOCOUNT ON;

DECLARE @objectid int;

DECLARE @indexid int;

DECLARE @partitioncount bigint;

DECLARE @schemaname nvarchar(130);

DECLARE @objectname nvarchar(130);

DECLARE @indexname nvarchar(130);

DECLARE @partitionnum bigint;

DECLARE @partitions bigint;

DECLARE @frag float;

DECLARE @command nvarchar(4000);

-- Conditionally select tables and indexes from the sys.dm_db_index_physical_stats function

-- and convert object and index IDs to names.

SELECT

object_id AS objectid,

index_id AS indexid,

partition_number AS partitionnum,

avg_fragmentation_in_percent AS frag

INTO #work_to_do

FROM sys.dm_db_index_physical_stats (DB_ID(), NULL, NULL , NULL, 'LIMITED')

WHERE avg_fragmentation_in_percent > 10.0 AND index_id > 0;

-- Declare the cursor for the list of partitions to be processed.

DECLARE partitions CURSOR FOR SELECT * FROM #work_to_do;

-- Open the cursor.

OPEN partitions;

-- Loop through the partitions.

WHILE (1=1)

BEGIN;

FETCH NEXT

FROM partitions

INTO @objectid, @indexid, @partitionnum, @frag;

IF @@FETCH_STATUS < 0 BREAK;

SELECT @objectname = QUOTENAME(o.name), @schemaname = QUOTENAME(s.name)

FROM sys.objects AS o

JOIN sys.schemas as s ON s.schema_id = o.schema_id

WHERE o.object_id = @objectid;

SELECT @indexname = QUOTENAME(name)

FROM sys.indexes

WHERE object_id = @objectid AND index_id = @indexid;

SELECT @partitioncount = count (*)

FROM sys.partitions

WHERE object_id = @objectid AND index_id = @indexid;

-- 30 is an arbitrary decision point at which to switch between reorganizing and rebuilding.

IF @frag < 30.0

SET @command = N'ALTER INDEX ' + @indexname + N' ON ' + @schemaname + N'.' + @objectname + N' REORGANIZE';

IF @frag >= 30.0

SET @command = N'ALTER INDEX ' + @indexname + N' ON ' + @schemaname + N'.' + @objectname + N' REBUILD';

IF @partitioncount > 1

SET @command = @command + N' PARTITION=' + CAST(@partitionnum AS nvarchar(10));

EXEC (@command);

PRINT N'Executed: ' + @command;

END;

-- Close and deallocate the cursor.

CLOSE partitions;

DEALLOCATE partitions;

-- Drop the temporary table.

DROP TABLE #work_to_do;

This code runs in only one database so I needed to add a cursor to run through all databases on the instance. The code above also differentiates partitions in an index. I didn't need that, so I stripped it out. I als stripped out the reorganize code. On an enterprise edition I prefer online rebuilds.

I came up with the following code:

-- Index rebuild online on all databases on an instance

SET NOCOUNT ON

DECLARE @db as sysname;

DECLARE @db_id as int;

DECLARE @cmd1 as nvarchar(1000);

DECLARE @objectid int;

DECLARE @indexid int;

DECLARE @schemaname nvarchar(130);

DECLARE @objectname nvarchar(130);

DECLARE @indexname nvarchar(130);

DECLARE @partitionnum bigint;

DECLARE @command nvarchar(4000);

DECLARE @errornumber int;

DECLARE DatabaseList CURSOR FAST_FORWARD FOR

SELECT [name], [database_id] FROM sys.databases

WHERE [name] NOT IN ('tempdb','model')

AND state_desc = 'ONLINE'

ORDER BY [name]

OPEN DatabaseList

FETCH NEXT FROM DatabaseList INTO @db, @db_id

WHILE (@@fetch_status = 0)

BEGIN

PRINT N' ';

PRINT N'Database: ' + @db;

-- Conditionally select tables and indexes from the sys.dm_db_index_physical_stats function

-- and convert object and index IDs to names.

EXEC ('USE ' + @db + ';

SELECT p.object_id AS objectid

, p.index_id AS indexid

, o.name AS objectName

, s.name AS schemaName

, i.name AS indexName

INTO ##work_to_do

FROM sys.dm_db_index_physical_stats (' + @DB_ID + ', NULL, NULL , NULL, ''LIMITED'') AS p

INNER JOIN sys.objects as o

ON p.object_id = o.object_id

INNER JOIN sys.schemas as s

ON s.schema_id = o.schema_id

INNER JOIN sys.indexes i

ON p.object_id = i.object_id

AND i.index_id = p.index_id

WHERE p.avg_fragmentation_in_percent > 10.0 AND p.index_id > 0

AND p.index_id < 25000

AND p.page_count > 1000;')

-- Declare the cursor for the list of fragmented indexes to be processed.

DECLARE FragmentedIndexes CURSOR FOR SELECT * FROM ##work_to_do;

-- Open the cursor.

OPEN FragmentedIndexes;

-- Loop through the indexes.

WHILE (1=1)

BEGIN;

FETCH NEXT

FROM FragmentedIndexes

INTO @objectid, @indexid, @objectname, @schemaname, @indexname;

IF @@FETCH_STATUS < 0 BREAK;

-- Check for invalid datatypes for online rebuild

-- If those datatypes are found, do normal rebuild

BEGIN TRY

SET @command = N'USE ' + @db + '; ALTER INDEX ' + @indexname + N' ON ' + @schemaname + N'.' + @objectname + N' REBUILD WITH (ONLINE=ON)';

EXEC (@command);

PRINT N'Executed: ' + @command;

END TRY

BEGIN CATCH

SELECT @errornumber = ERROR_NUMBER()

IF @errornumber = 2725

BEGIN

SET @command = N'USE ' + @db + '; ALTER INDEX ' + @indexname + N' ON ' + @schemaname + N'.' + @objectname + N' REBUILD';

EXEC (@command);

PRINT N'Executed: ' + @command;

END

END CATCH

END;

-- Close and deallocate the cursor.

CLOSE FragmentedIndexes;

DEALLOCATE FragmentedIndexes;

-- Drop the temporary table.

DROP TABLE ##work_to_do;

FETCH NEXT FROM DatabaseList INTO @db, @db_id

END

CLOSE DatabaseList;

DEALLOCATE DatabaseList;

Explanation of the code:

I start with creating a cursor for all databases on the instance:

DECLARE DatabaseList CURSOR FAST_FORWARD FOR

SELECT [name], [database_id] FROM sys.databases

WHERE [name] NOT IN ('tempdb','model')

AND state_desc = 'ONLINE'

ORDER BY [name]

Of course I exclude the tempdb and the model database. On model I don't expect indexes and tempdb is obvious.

After that I open the database cursor and start processing the databases one at the time.

First I collect the fragmented indexes:

EXEC ('USE ' + @db + ';

SELECT p.object_id AS objectid

, p.index_id AS indexid

, o.name AS objectName

, s.name AS schemaName

, i.name AS indexName

INTO ##work_to_do

FROM sys.dm_db_index_physical_stats (' + @DB_ID + ', NULL, NULL , NULL, ''LIMITED'') AS p

INNER JOIN sys.objects as o

ON p.object_id = o.object_id

INNER JOIN sys.schemas as s

ON s.schema_id = o.schema_id

INNER JOIN sys.indexes i

ON p.object_id = i.object_id

AND i.index_id = p.index_id

WHERE p.avg_fragmentation_in_percent > 10.0 AND p.index_id > 0

AND p.index_id < 25000

AND p.page_count > 1000;')

I use sys.dm_db_index_physical_stats for finding the fragmented indexes. In order to get the name, schemaname and objectname I join the sys.dm_db_index_physical_stats with sys.objects, sys.schemas and sys.indexes.

The result I insert in a global temp table. Global because the exec statement runs on another thread so I can't use a local temp table. I can't see it in the rest of the procedure if I did that.

In the where clause I check on fragmentation over 10%. I also check if the index_id is smaller than 25000. If it is over it is either a XML index or a Spatial index.

The last check I do is on page_count of the index. SQL Server will not rebuild indexes that are smaller than 1000 pages(8MB). It will not do seeks on such small indexes, only scans. So fragmentation won't matter.

With the fragmented indexes in the temp table I can build the statement necessary to defragment the index:

BEGIN TRY

SET @command = N'USE ' + @db + '; ALTER INDEX ' + @indexname + N' ON ' + @schemaname + N'.' + @objectname + N' REBUILD WITH (ONLINE=ON)';

EXEC (@command);

PRINT N'Executed: ' + @command;

END TRY

BEGIN CATCH

SELECT @errornumber = ERROR_NUMBER()

IF @errornumber = 2725

BEGIN

SET @command = N'USE ' + @db + '; ALTER INDEX ' + @indexname + N' ON ' + @schemaname + N'.' + @objectname + N' REBUILD';

EXEC (@command);

PRINT N'Executed: ' + @command;

END

END CATCH

Since the customer had an enterprise edition I do the rebuild online. That was necessary since the customer had a 24*7 system. Online rebuilds do have limitations: you can't rebuild an index online if the index contains blob datatypes like text, ntext, xml, varchar(max), nvarchar(max), image, etc….

If those datatypes are used in the underlying table you can't do an online rebuild either. That's why I check for errornumber 2725. If that's returned I build the rebuild statement without the online option.

The rest of the code is cleanup code. We should drop the temp table and close and deallocate the cursors(very important!)

I want to add some more errorhandling later on. The temptable should always be dropped. But that's the only thing missing in the code.

If you have any remarks on my code, I'll be delighted to hear it!

Regards,

Menzo

Reading PerfMon Counters in T-SQL with CLR

Menzo Steinhorst — Wed, 25 Mar 2009 16:08:01 GMT

A collegue of mine asked me if it is possible to read out PerfMon counters that are not in the sys.dm_os_performance_counters DMV. I searched everywhere but the only possibility I could think of is by using a CLR user defined function.

So I wrote an assembly (I'm still a VB programmer, I never bothered to learn C#. And I'm never going to as long as VB.Net exists J ):

The code is fairly simple:

Imports System

Imports System.Data

Imports System.Data.SqlClient

Imports System.Data.SqlTypes

Imports Microsoft.SqlServer.Server

Partial Public Class UserDefinedFunctions

<Microsoft.SqlServer.Server.SqlFunction()> _

Public Shared Function ReadPerfMonCounter(ByVal Category As String, ByVal Counter As String, ByVal Instance As String) As SqlString

'******************************************************************************

'* Disclaimer: I'm a lousy programmer!!!! *

'* ****************************************************************************

Dim sTemp As String

Try

Dim perfCounter As New System.Diagnostics.PerformanceCounter(Category, Counter, Instance, "localhost")

'Somehow it won't return a result on the first NextValue call.

'Never mind: we just call it 2 times :).

sTemp = perfCounter.NextValue().ToString

'And wait a little while, but now she's ready to go.

Threading.Thread.Sleep(1000)

Return New SqlString(perfCounter.NextValue.ToString)

Catch ex As Exception

Return New SqlString(ex.Message)

End Try

End Function

End Class

The only problem I had was the problem that the NextValue method of the perfCounter didn't return anything but 0 on its first call. That's why I call it 2 times now.

Once compiled you can deploy this assembly on your server. If you don't have .Net but want the assembly, just mail me.

I deployed it to the AdventureWorksTest database, which is a copy of the regular AdventureWorks database.

If you deploy it make sure your database has TRUSTWORTHY ON because I didn't sign the assembly. This assembly needs unrestricted access. That's unsafe on SQL 2005.

When you have done all that you can use a select statement like:

SELECT AdventureWorksTest.dbo.ReadPerfMonCounter('PhysicalDisk','% Idle Time','0 C: D: E:');

That's all. Error handling could be better by the way. When you make a typo in your counter you'll wait forever instead of getting an error.

If you don't like the unsafe setting on the assembly you could transfer that part to a class library and reference it from your .NET stored procedure. Plenty opportunities left….

I didn't bother on execution context and all that. On my machine the service account is a local admin. You might need some impersonification code if not.

Regards,

Menzo

Back again!

Menzo Steinhorst — Wed, 25 Mar 2009 13:37:08 GMT

I didn't blog for a long time. For various reasons but the main reason is because I have been quite busy.

So what did I do the last half year?

A lot of Performance Tuning and Optimization Workshops Plus on SQL 2005. This is in my opinion one of the best workshops we offer. There's really good stuff in it on Windows and SQL internals and architecture. I delivered the workshops to both DBA's and developers and both groups were enthusiastic about the workshop.
A lot of SQLRAP's. RAP's are fun to do because you never know what to expect. You see quite a wide array of problems on SQL and it's never the same.
Ramping up on SQL 2008. Although SQL 2008 uses more or less the same engine as SQL 2005 it takes me a lot of time. The new features are not all that easy or could be used (and are!) in complex scenarios.
Ramping up on the SQL Server BI stack. I was proficient in SSIS and SSRS, but I didn't know much about SSAS. That's going to change this year. A lot of our customers use SSAS or want to use it, and I want to be able to help them with that. Since SSAS is completely different from the engine this will take time…
Next to all SQL stuff I do like to spend some time with my family every now and then……

Despite the economic climate, my agenda is still more than full. But no complains about thatJ.

I will blog more in the future.

Menzo

TechEd IT Proffesionals 2008 EMEA Day 2

Menzo Steinhorst — Wed, 05 Nov 2008 00:26:29 GMT

Today I visited 4 presentations:

Next generation analysis services, Gemini
Windows 2008 Clustering
Policy Based Management
Migrating to SQL Server.

The best session was on Policy Based Management by Buck Woody. He made PBM look very simple and showed cool things like using SQL to get values in conditions. And he showed some central management stuff. With central management you can evaluate policies over several instances.

He also demonstrated PBM with powershell. I don't know much about powershell but I should learn it. It looked powerful and with powershell you can evaluate your policies on SQL 2005 and SQL 2000 as well. Partly because not everything is supported but that doesn't matter that much.

The Gemini session was also remarkable. Handling 20.000.000 rows in seconds in Excel was quite impressive. I'm not really into Analysis Services but this looked great.

The migrating session had an interesting session on the SQL Server Migration Assistant Application. They did a migration from Oracle to SQL Server. I had some experience with the tool but they showed features I had never used. PL/SQL conversion is not without problems: build in packages are extensively used by Oracle developers and they are not easily migrated by the tool.

The most important thing I learned in the clustering session was the validation tool. Every time you change something on the cluster: run the validation tool and you get a report whether or not your cluster is still in mint condition. Clustering is a lot simpler in Windows 2008. I do meet a lot of people that are scared of clustering. I hope Windows 2008 makes them more comfortable with clustering.

Tomorrow I don't have to man the booth so I can go to sessions all day.

TechEd IT Proffesionals 2008 EMEA Day 1

Menzo Steinhorst — Wed, 05 Nov 2008 00:04:28 GMT

For the next two weeks I will be attending Teched in the beautiful city of Barcelona. I attended 2 sessions today. The first one was about the high availability tools of SQL Server 2008. It was presented by Andrew Fryer, a technology evangelist of Microsoft UK. I saw a good presentation that emphasized on mirroring but also covered clustering and log-shipping. Personally I prefer clustering over mirroring. With mirroring there is always more administration with logins and other stuff that isn't mirrored. On the other hand: the transparent failover capabilities of mirroring are impressive.

Andrew also covered combinations of mirroring and log-shipping. I did see this at a few client sites and it provides extra availability at a low cost.

The second session was on storage in SQL 2008. It emphasized on the compression features of SQL 2008. He had a demo on back-up compression and he did some compression on tables. What impressed me was that on a partitioned table you can choose different compression methods on all partitions.

There are two levels of compression:

Row level: this one uses variable lengths for fixed length data types.
Page level: this one compresses entire datapages by looking for data that is on the page multiple times. This data is only stored once. In rows where the data originally was the data is replaced by a pointer which takes up less space. Page level compression includes row level by the way. Page level compression will save you more space.

On the downside: this feature is only available in the enterprise and developer edition.

He also touched sparse column support briefly. Personally I like this feature a lot. You can create columns that contain lots of nulls as sparse. SQL will convert all sparse columns on a row to a XML string. But clients don't notice anything and you can save a lot of space.

Tomorrow I'm have to man the Ask The Expert booth in the morning. In the afternoon I'm going to some sessions.

The SQL 2008 Resource Governor

Menzo Steinhorst — Wed, 02 Jul 2008 23:30:44 GMT

The client I work for today asked me to come up with a quick overview of the Resource Governor. I'm in luck these days because I have lots of assignments on SQL 2008. Usually PFE's deal with production environments. More often then not this is still SQL 2000. I think 80% of my assignments are on SQL 2005 and 20% are on SQL 2000.

I have no previous experience with the Resource Governor. I only saw it in action during a presentation of Bob Beauchemin of SQLSkills on TechEd 2007 in Barcelona. Excellent presentation by the way. Presentations of Bob Beauchemin are always excellent.

What is it?

It's a mechanism to limit resource consumption on incoming requests.

How does it work?

It's got 3 components

Resource Pools
- A resource pool represents the physical resources of a server. You can set:
  - MIN and MAX for CPU
  - MIN and MAX for memory
- The sum of the MIN values across all pools cannot exceed 100.
- There's an Internal Pool. SQL Server needs this one for its own processes. You cannot change this pool.
Workload Groups
- In a workload group you can group session requests that are more or less equal to each other. You can group them with a classification stored procedure.
- On the group you can set some advanced settings.
Classifier Function to point logins to workload groups.

To illustrate the pools and groups I took the following screenshot:

Screen clipping taken: 2-7-2008; 22:00

This is the form in which you manage the pools and groups. It's a standard master-detail form. When you select a pool, you get the groups belonging to it in the bottom datagridview. Options for the groups:

Importance	High, medium or low. Is taken into account while scheduling requests
Maximum Requests	Maximum number of simultaneous request in the group
CPU Time	Maximum CPU time a request can use
Memory Grant %	Maximum memory a request can take from the pool
Grant Time-out	Maximum amount of time a query will wait on a resource before it will fail
Degree of Parallelism	DOP for parallel requests

When you got your pools and groups setup you have to create a classifier function:

CREATE FUNCTION [dbo].[rgClassifier]()

RETURNS sysname

WITH SCHEMABINDING

BEGIN

-- Define the return sysname variable for the function

DECLARE @grp_name AS sysname;

SET @grp_name = 'default';

-- Specify the T-SQL statements for mapping session information

-- with Workload Groups defined for the Resource Governor.

IF (IS_SRVROLEMEMBER ('sysadmin') = 1)

SET @grp_name = 'Main';

-- TODO

-- Put additional classification logic here

RETURN @grp_name;

END

Above a very simple example of a classifier function. You can filter requests on roles, usernames and whatever else you can think of. Every new session goes through this procedure and is assigned to a workload group.

That is a very simple explanation of the Resource Governor

Is it useful?

It can be useful on systems with a heavy load and lots of processes that are not that important. With the Resource Governor you can guarantee that your OLTP performance is not influenced by some crap SSIS package that is taking all CPU's up to 100%.

But?

It is quite complex to set up. There are lots of options average DBA's do not use or know of. If you make an error in the setup you could end up with an unresponsive server. It'll take skilled DBA's to set this up and manage properly. I'll hope the training kits of the new certifications will delve into this. Btw, if you mess up your setup: you can always correct things with the Dedicated Administration Account(DAC). That one is on the Internal pool and always has access to resources.

Conclusion

Promising feature, but hard to set up and manage. For mission critical databases very useful. You need to alter your ways of administering an instance. You do not notice any performance problems when a bad query on a low priority workload is taking forever to complete.

In multi-instance environments things get even more complicated since the Resource Governor is not instance aware.

Other Stuff on the Resource Governor

I saw a few samples on Internet with nice Perfmon diagrams, showing the CPU and memory usage of workloads.

The RC0 BOL contains quite a lot of info on the Resource Governor. When you read everything you'll be an expert.

Sync Framework

Menzo Steinhorst — Wed, 02 Jul 2008 12:58:00 GMT

Today I'm working for a customer who wants to implement the Sync Framework. As you probably know the Sync Framework is still beta but RTM is expected somewhere this fall.

The client has a typical problem. They have 7000 field engineers roaming the globe. They need to synchronize a lot of date before and after an assignment. Till now the client had implemented merge replication in SQL 2005 for this. They ran into problems because for merge replication you need to be in the dbo group. Since service engineers are not part of that group they had to edit system stored procedures in order to get this solution to work.

They asked us for a solution. I first tested merge replication in SQL 2008 without success. I heard vaguely about the Sync Framework and decided to give it a try.

The Sync Framework is more .Net then SQL Server. Since I'm not a .Net expert I was a bit worried in the beginning. Luckily implementing the Sync Framework is very easy and doesn't require lots of coding. Here's the graphical explanation in VB.Net(I still don't like C# so I always program in VB.Net):

Create a Windows Forms application in Visual Studio 2008
Add a Local Database Cache to your project.
Fill in the wizard screen.

I connected to a SQL 2008 database so I can use the new change tracking features. This saves your data model because if you don't use change tracking you have to use triggers and tombstone-tables and they will seriously clutter your data model.

The client connection is a SQL Server Compact 3.5. SQL Server Compact is very small(2MB) and uses very little memory; even with pretty large datasets like 10.000 rows. Of course we can't expect miracles of it so loading up a 2GB database is useless. I did try it but performance was not optimal :).

When you press Ok the wizard will generate scripts and alter your development database. The scripts come in handy when you want to deploy your solution to the production database.
Create a dataset of your generated local data cache.

The wizard proceeds with the dataset screen:
When you are done creating the dataset, bring up the data sources view in visual studio. Now drag a table from your dataset and drop it on the empty form.

When you drag a datatable on your form Visual Studio adds all necessary components and code. So without programming you end up with a fully functional form.

I docked the datagridview in the form to make it look better.
Now its time to add syncing. First add a button on the toolbar for executing syncing. In the code of the form you add the click event of the button. To obtain the sync code you go back to the local data cache designer

When you press the Show Code Example link you'll get the code you need. Just paste this code in the click event of the button.

Here's the event:

Private Sub cmdSync_Click(ByVal sender As Object, ByVal e As System.EventArgs) Handles cmdSync.Click

' Call SyncAgent.Synchronize() to initiate the synchronization process.

' Synchronization only updates the local database, not your project's data source.

Dim syncAgent As SampleDataCacheSyncAgent = New SampleDataCacheSyncAgent()

Dim syncStats As Microsoft.Synchronization.Data.SyncStatistics = syncAgent.Synchronize()

' TODO: Reload your project data source from the local database (for example, call the TableAdapter.Fill method).

Me.SyncDataSet.Parameters.Merge(Me.ParametersTableAdapter.GetData())

End Sub

The merge line I added myself. The form is ready now.
Now we move on with the sync class itself to choose what kind of syncing we want to have.

Now you can choose the syncdirection that fits your needs.
The last step is to add conflict management during syncing:

Partial Public Class SampleDataCacheServerSyncProvider

Private Sub SampleDataCacheServerSyncProvider_ApplyChangeFailed(ByVal sender As Object, ByVal e As Microsoft.Synchronization.Data.ApplyChangeFailedEventArgs) Handles Me.ApplyChangeFailed

Dim clientChange As DataTable = e.Conflict.ClientChange

Dim serverChange As DataTable = e.Conflict.ServerChange

e.Action = Microsoft.Synchronization.Data.ApplyAction.RetryWithForceWrite

End Sub

End Class

I choose a simple RetryWithForceWrite here but you can code here what you want.

Conclusion

I build a fully functional sync sample in a few minutes. That's not too bad. Of course this is a simple scenario where complete tables are synchronized. If you want to synchronize on queries you have to write a lot of code yourself. But it is possible. I can think of lots scenarios where this framework could be very useful. Especially with all the smartphones and all that.

Deploying this solution was something else. I ran into lots of problems. There is no installer for Sync Services yet and I didn't succeed in deploying it manually. For now you have to install the complete .Net framework 3.5 SDK and the beta service packs for VS 2008 and .Net 3.5. The installer and the Sync Framework itself are expected to go RTM this fall. So we have to wait a little bit for proper installers.

Table Partitioning Conclusions

Menzo Steinhorst — Tue, 01 Jul 2008 12:14:17 GMT

Today I tested all my code and made changes where I made mistakes. And I made quite a few! My biggest mistake was with the GetDate() function. I made partitions with milliseconds which was not my intention! I changed the code in my last post where necessary. If you still encounter bugs, please contact me.

A positive conclusion is the management studio of SQL 2008. Great tool! I used the scripts of the partitioning wizard quite a lot but the best feature it offered was the debugger. Finally T-SQL developers have T-SQL debugging without installing Visual Studio. Here's a screenshot:

The debugger saved me a lot of time because you immediately know where your bugs are when you can step through your code. No more output messages and all that old-school techniques. Of course intellisense also saves time while coding. I'm impressed by the new management studio.

Btw: you can't use the debugger on a SQL 2005 instance.

The wizards in the SQL 2008 management studio only support standard scenarios. I hope I pointed you in some directions what you can do with partitioning when you want something else. This was quite a complex scenario but didn't require that much code.

The code I produced is not nearly finished. Robust error handling should be added before this is installed on production databases. For me this assignment is over though. The DBA's from the client can take it from here.

My next assignments will be on the Sync framework and on the Resource governor of SQL 2008. My next blogs will be on those subjects.

Table Partitioning Sliding Window Case

Menzo Steinhorst — Mon, 30 Jun 2008 23:27:00 GMT

As I mentioned in my last post I'm currently working on a complex partitioning project for a customer. They need partitioning in order to speed up inserts on the table. Lots of inserts take place on this table. Right now they have around 2 million inserts per day and this will grow quickly to 4 million inserts/day. The table is heavily queried too, so there are a lot of indexes on the table. I know this isn't the most desirable design but that's how it is at the moment and at Premier Field Engineering our projects are usually not long enough to make judgments on architecture and design. Initial testing with partitioning looked promising so now we will implement it.

Let me get started. We start with the partition function.

First we set boundaries.

· They want partitions for each day of the current week. Weeks start on Monday.

· They want partitions for the last four weeks.

· They want partitions for the last 6 months.

· For the rest of the data they want one big partition.

There desired boundaries have some problems:

· Last four weeks do usually not end on the first of the month. Do we add these rows to the last month partition or do we make a partition for this month. I choose the latter because sooner or later this month would become a partition anyway.

· The archive partition is going to be huge! Maybe its better to move it to an archive-table. I have to discuss this with the customer, for now it's ok.

The table is partitioned at the moment with a RANGE RIGHT function. You can do 2 things when you have a table on a partition scheme you don't want:

· You can drop the clustered index and recreate it with the MOVE TO partitionscheme(partitionColumn) command.

· You can drop the table and recreate it on the partition scheme you want.

I choose number 2 because with SSIS you can quickly move data around and recreating a clustered index on a big table takes forever, especially with the MOVE TO directive.

I start with the initial situation in the partition function:

I'll load up the SQL2008 management studio and generate the scripts with the wizard. The wizard is also available when you connect to a SQL 2005 instance.

Here's the script I got:

BEGIN TRANSACTION

CREATE PARTITION FUNCTION [pfEvent](datetime)

AS RANGE LEFT

FOR VALUES (N'2008-07-06T00:00:00'

, N'2008-07-05T00:00:00'

, N'2008-07-04T00:00:00'

, N'2008-07-03T00:00:00'

, N'2008-07-02T00:00:00'

, N'2008-07-01T00:00:00'

, N'2008-06-30T00:00:00'

, N'2008-06-23T00:00:00'

, N'2008-06-16T00:00:00'

, N'2008-06-09T00:00:00'

, N'2008-06-02T00:00:00'

, N'2008-06-01T00:00:00'

, N'2008-05-31T00:00:00'

, N'2008-04-30T00:00:00'

, N'2008-03-31T00:00:00'

, N'2008-02-29T00:00:00'

, N'2008-01-31T00:00:00'

, N'2007-12-31T00:00:00');

CREATE PARTITION SCHEME [psEvent]

AS PARTITION [pfEvent]

TO ([PRIMARY]

, [PRIMARY]

, [ARCHIVE]

, [ARCHIVE]);

CREATE CLUSTERED INDEX [ClusteredIndex_on_psEvent_633504211030911271] ON [dbo].[Event]

(

[EvtDatTim]

) ON [psEvent]([EvtDatTim]);

DROP INDEX [ClusteredIndex_on_psEvent_633504211030911271] ON [dbo].[Event] WITH ( ONLINE = OFF );

COMMIT TRANSACTION

I'll execute the script and the empty recreated table will be partitioned.

Now I can recreate all indexes again and load the data back with SSIS. After that foreign keys can be restored.

Now the table is partitioned I have to implement a mechanism to automate the sliding window. I'm going to implement this with stored procedures that are executed in jobs.

A couple of stored procedures are needed:

1. A stored procedure, run every Sunday, that creates the daily partitions of next week.

2. A stored procedure that merges the daily partitions of last week into 1 week partition. Also run every Sunday.

3. A stored procedure that merges the fifth week partition into the corresponding month partitions. Also run every Sunday.

4. A stored procedure that merges the seventh month partition into the archive partition. For simplicity I run this one every Sunday as well. Otherwise I need a second job.

Off course I could do all this in one stored procedure but I like small procedures. That debugs a lot easier!

1 & 2 are easy, 3 & 4 are a bit more complicated. Let's get going.

1. You can't simply create partitions with GetDate(). You would get the current time down to milliseconds into the boundary range. That's why I created a function to prepare the date:

CREATE FUNCTION MSPartitionPrepareDate

(

@Date SmallDateTime

)

RETURNS SmallDateTime

BEGIN

DECLARE @Result SmallDateTime;

SELECT @Result = CAST(CONVERT(Varchar(23),@Date,101 ) AS SmallDateTime);

RETURN @Result;

END

The procedure that creates partitions for the days of the next week:

CREATE PROCEDURE [dbo].[MSPartitioningAddDays]

BEGIN

DECLARE @SplitDate AS DateTime;

ALTER PARTITION SCHEME psEvent

NEXT USED [PRIMARY];

SELECT @SplitDate = dbo.MSPartitionPrepareDate(GetDate() + 1);

ALTER PARTITION FUNCTION pfEvent()

SPLIT RANGE (@SplitDate);

ALTER PARTITION SCHEME psEvent

NEXT USED [PRIMARY];

SELECT @SplitDate = dbo.MSPartitionPrepareDate(GetDate() + 2);

ALTER PARTITION FUNCTION pfEvent()

SPLIT RANGE (@SplitDate);

ALTER PARTITION SCHEME psEvent

NEXT USED [PRIMARY];

SELECT @SplitDate = dbo.MSPartitionPrepareDate(GetDate() + 3);

ALTER PARTITION FUNCTION pfEvent()

SPLIT RANGE (@SplitDate);

ALTER PARTITION SCHEME psEvent

NEXT USED [PRIMARY];

SELECT @SplitDate = dbo.MSPartitionPrepareDate(GetDate() + 4);

ALTER PARTITION FUNCTION pfEvent()

SPLIT RANGE (@SplitDate);

ALTER PARTITION SCHEME psEvent

NEXT USED [PRIMARY];

SELECT @SplitDate = dbo.MSPartitionPrepareDate(GetDate() + 5);

ALTER PARTITION FUNCTION pfEvent()

SPLIT RANGE (@SplitDate);

ALTER PARTITION SCHEME psEvent

NEXT USED [PRIMARY];

SELECT @SplitDate = dbo.MSPartitionPrepareDate(GetDate() + 6);

ALTER PARTITION FUNCTION pfEvent()

SPLIT RANGE (@SplitDate);

ALTER PARTITION SCHEME psEvent

NEXT USED [PRIMARY];

SELECT @SplitDate = dbo.MSPartitionPrepareDate(GetDate() + 7);

ALTER PARTITION FUNCTION pfEvent()

SPLIT RANGE (@SplitDate);

END

That's not too hard. Very simple and very readable. First you add room on the partition scheme and then you create the new day partition on the function. Do that 7 times and you're done.

2. Now we have to merge the data of the partitions of last week into one week partition. You can do that in 2 ways: Either you merge all days except Sunday, or you switch out all data to a staging table and switch it back in after you split out the days. Since a day is a relatively small partition I merge them directly.

CREATE PROCEDURE [dbo].[MSPartitioningMergeDaysInWeek]

BEGIN

-- Merge the last 7 days partitions in one week partition.

-- Last Monday

EXEC [dbo].[MSPartitioningMergeDay] 1

-- Last Tuesday

EXEC [dbo].[MSPartitioningMergeDay] 2

-- Last Wednesday

EXEC [dbo].[MSPartitioningMergeDay] 3

-- Last Thursday

EXEC [dbo].[MSPartitioningMergeDay] 4

-- Last Friday

EXEC [dbo].[MSPartitioningMergeDay] 5

-- Last Saturday

EXEC [dbo].[MSPartitioningMergeDay] 6

END;

CREATE PROCEDURE [dbo].[MSPartitioningMergeDay]

@MinusDays int

BEGIN

DECLARE @MergeDate AS DateTime;

SELECT @MergeDate = dbo.MSPartitionPrepareDate (GetDate() - @MinusDays);

ALTER PARTITION FUNCTION pfEvent()

MERGE RANGE (@MergeDate) ;

END;

2 simple stored procedures that get this job done. I work from 1 to 6 in order to merge the smallest set. If I work from 6 to 1 the set to merge would be larger after every merge.

3. The simple stuff is done now. After you run stored procedure 2 you have 5 week partitions. The oldest needs to be merged with the already existing month partition. If no partition exists yet one has to be created.

Simple design:

· First find the oldest week partition. GetDate() - 35 will get you there.

· Switch out this partition

· Look if a new month starts between GetDate() - 35 and GetDate() - 28

· If yes create new month partition that ends on GetDate() - 28

· If no alter the existing month partition so that it ends on GetDate() - 28

· Switch partition back in.

Here's the code:

CREATE PROCEDURE [dbo].[MSPartitioningMergeOldestWeekPartition]

BEGIN

DECLARE @PartitionNumber AS int;

DECLARE @Boundary AS DateTime;

DECLARE @DateDiff AS int;

DECLARE @SplitDate AS DateTime;

DECLARE @ErrorNumber AS int;

-- Partition number and boundary value

SELECT @PartitionNumber = boundary_id

,@Boundary = CAST(value AS DateTime)

FROM sys.partition_range_values rv

INNER JOIN sys.partition_functions pf

ON rv.function_Id = pf.function_id

WHERE pf.name = 'pfEvent'

AND CAST(value AS DateTime) = (SELECT CAST(CAST(DATEPART("YEAR",getdate()-35) AS Varchar(4))

+ '-' + Right('0' + Convert(VarChar(2), Month(GetDate()-35)), 2)

+ '-' + Right('0' + Convert(VarChar(2), Day(GetDate()-35)), 2)

+ ' 00:00:00.000' AS DateTime))

-- Switching out.....

ALTER TABLE [dbo].[Event]

SWITCH PARTITION @PartitionNumber

TO Staging_Event_Proto

Print 'Geswitched'

--Getting rid of the now empty partition....

SELECT @SplitDate = dbo.MSPartitionPrepareDate(GetDate() - 35);

ALTER PARTITION FUNCTION pfEvent()

MERGE RANGE (@SplitDate)

SELECT @DateDiff = DATEDIFF("mm",getdate()-35,GETDATE()-28)

IF @DateDiff <> 0

BEGIN

-- Find last day of the month and make partitionboundary of it

ALTER PARTITION SCHEME psEvent

NEXT USED [PRIMARY];

ALTER PARTITION FUNCTION pfEvent()

SPLIT RANGE (DATEADD("dd", -1, CAST(CAST(DATEPART("YYYY" ,Getdate()-30) AS Varchar(4)) + '-' + CAST(DATEPART("mm",Getdate()-28) AS Varchar(2)) + '-01' AS DATETIME)));

-- Starting day of next partition is endday of this partition.

BEGIN TRY

ALTER PARTITION SCHEME psEvent

NEXT USED [PRIMARY];

SELECT @SplitDate = dbo.MSPartitionPrepareDate(GetDate() - 28);

ALTER PARTITION FUNCTION pfEvent()

SPLIT RANGE (@SPlitDate);

END TRY

BEGIN CATCH

-- Partition might be there already

SELECT @ErrorNumber = ERROR_NUMBER();

IF @ErrorNumber <> 7721 -- Errornumber of overlapping partitions

BEGIN

SELECT

ERROR_NUMBER() AS ErrorNumber,

ERROR_SEVERITY() AS ErrorSeverity,

ERROR_STATE() AS ErrorState,

ERROR_PROCEDURE() AS ErrorProcedure,

ERROR_LINE() AS ErrorLine,

ERROR_MESSAGE() AS ErrorMessage;

END

END CATCH

END

-- Switching in....

-- Can't use the switch command since the parititions don't match.

BEGIN TRANSACTION

INSERT INTO [dbo].[Event]

SELECT * FROM dbo.Staging_Event_Proto;

TRUNCATE TABLE dbo.Staging_Event_Proto;

COMMIT TRANSACTION

END;

That's quite a piece of code. I have tested it now and it works. If it doesn't on your machine give me a shout.

4. For the last procedure I take the easy way and just merge the partition. The partition is not queried a lot and it can run in the background without bothering anybody.

Here we go:

CREATE PROCEDURE [dbo].[MSPartitioningMergeOldestMonth]

BEGIN

DECLARE @Boundary DateTime;

-- Boundary value: it's always partion 1

SELECT @Boundary = CAST(value AS DateTime)

FROM sys.partition_range_values rv

INNER JOIN sys.partition_functions pf

ON rv.function_Id = pf.function_id

WHERE pf.name = 'pfEvent'

AND rv.boundary_id = 1

--Merge this partition....

ALTER PARTITION FUNCTION pfEvent()

MERGE RANGE (@Boundary)

END;

So that's it. Now we only have to make a job with a weekly schedule to run it on Sunday and we're done. Tomorrow I'm going to test all this code. This case took me almost 7 hours and a lot of energy :). But it was great fun. I usually don't code all day long anymore but every once and while it's great fun!

Expect a post on testing soon.

I know I'm not the best T-SQL Coder there is: suggestions to optimize the code are therefore welcome.

Table Partitioning Basics(2)

Menzo Steinhorst — Sun, 29 Jun 2008 23:06:36 GMT

In my last blog I looked at SPLIT and MERGE operations on partition functions. SPLIT and MERGE are nice functions but I use them with care. A merge is not the best performing function there is…. And that is completely logical when you think about everything that is happening during execution:

Data needs to be transferred from one partition to another.
When the new partition is on another filegroup data needs to be transferred physically.
All storage aligned indexes(and all indexes need to be storage aligned!) must be updated.

All very time-consuming tasks. Not things you want to do every day. A split is usually an easy operation because you usually add an partition at the end of the range. If you split existing data the same rules apply as to a merge.

So what is a good idea then to move data around in an partitioned table? When you need to move complete partitions the SWITCH operator is a good choice.

ALTER TABLE Test

SWITCH PARTITION 1

TO StagingTable

This statement moves all data in partition 1 to the staging table StageTest. The staging table needs to have exactly the same schema as the partitioned table. It also needs to have the same clustered and non-clustered indexes and the same constraints. It also needs to be empty. This is a bit of a pain but if you don't do this you'll get errors:

Msg 4947, Level 16, State 1, Line 1

ALTER TABLE SWITCH statement failed. There is no identical index in source table Test.dbo.StagingTable' for the index 'IX_Date' in target table 'Test.dbo.PartitionSample'.

Msg 4982, Level 16, State 1, Line 1

ALTER TABLE SWITCH statement failed. Check constraints of source table 'Test.dbo.StagingTable' allow values that are not allowed by range defined by partition 4 on target table 'Test.dbo.PartitionSample'.

Msg 4905, Level 16, State 1, Line 1

ALTER TABLE SWITCH statement failed. The target table 'Test.dbo.StagingTable' must be empty.

Switching out takes seconds instead of hours so the choice is easy.

You can switch in data as well with the following statement:

ALTER TABLE StagingTable

SWITCH TO Test PARTITION 4

TO Test

This is also very fast, especially when the partition is empty.

So concluding: when you want to merge to big partitions you switch out the partition to merge to a staging table. Then you remove the boundary point with the ALTER PARTITION FUNCTION…MERGE command. After that you switch the data back in to the partitioned table.

This week I'm going to implement a complicated sliding window scenario at one of our customers. The scenario includes a partition function with daily partitions for the current week, weekly partitions for the last four weeks and monthly partitions for older data. I'll blog about that later this week.

Table Partitioning Basics

Menzo Steinhorst — Sat, 28 Jun 2008 23:11:00 GMT

Partioning objects in SQL Server starts with defining Partition Functions and Partition Schema's. The functions split your data and the schema's place it somewhere. A function can be used by multiple schema's. You start out by creating a function:

CREATE PARTITION FUNCTION [pfSample](datetime)

AS RANGE LEFT

FOR VALUES (1, 100, 1000);

A function must cover all data: from minus infinity to infinity.

You have to choose between RANGE RIGHT and RANGE LEFT. RANGE LEFT is default if you don't specify it.

RANGE RIGHT:

Partition	1	2	3	4
Values	<1	>= 1 AND < 100	>=100 AND < 1000	>= 1000

RANGE LEFT:

Partition	1	2	3	4
Values	<=1	> 1 AND <= 100	>100 AND <= 1000	> 1000

Things to notice here:

You always have 1 partition more than values you put in the Partition Function.
Range Left/Right does nothing more putting the boundary on a side.

Range Left/Range Right is an important decision to make. Most people favor Range Left. If you use a sliding window scenario Range Left feels more natural because with Range Right your first partition will be empty. Let me clarify this with the following example:

You want to partition a table on a field that contains a date. You want to make partitions for the last four months. The rest of the data you switch out to an archive table. So you have partitions for March, April, May and June. Your first partition on a Range Right will be dates before March. So your first partition will be empty… The same scenario with a Range Left will make your last partition empty. That's not so bad and could even be useful if you didn't slide your window on time.

When you are satisfied with your partition function you can proceed with the partition schema:

CREATE PARTITION SCHEME [psSample]

AS PARTITION [pfSample]

TO ([PRIMARY], [PRIMARY], [PRIMARY], [PRIMARY])

Schema's tell the function where to put the partitions. In this case, on my laptop, I put all partitions on the primary filegroup. In production environments you will put them on different filegroups of course.

Now you have everything in place to partition a table that has a datetime column. In SQL 2008 you can partition an existing table with a wizard. No such thing in SQL 2005. You have to rename your table, recreate a new on on the partition scheme, and move the data back…..Or you just remove the clustered index and recreate that on the partition scheme

ALTER TABLE Test

ADD CONSTRAINT PK_Test

PRIMARY KEY CLUSTERED(TestDate)

ON psSample(TestDate)

Make sure that one of the columns in your clustered key matches the partitioning column otherwise:

Msg 1908, Level 16, State 1, Line 1

Column 'datum' is partitioning column of the index 'PK_Test2'. Partition columns for a unique index must be a subset of the index key.

On tables with lots of rows partitioning the clustered key might take a while. Think of hours on really big tables. Don't try this on peak hours!

Now you have your partitioned table up and running you have a few commands at your disposal to administer the thing.

First of all you can SPLIT or MERGE a partition.

SPLIT adds a new partition. For example:

ALTER PARTITION FUNCTION pfSample()

SPLIT RANGE(2000);

This piece of T-SQL will create an extra partition that contains rows > 2000. If you get the following error you have to alter your Partition Scheme:

Msg 7707, Level 16, State 1, Line 1

The associated partition function 'pfSample' generates more partitions than there are file groups mentioned in the scheme 'psSample'.

The message says it all: you are out of places to put partitions on in your scheme. So let's add one to the scheme:

ALTER PARTITION SCHEMA psSample

NEXT USED [Primary];

Now you have added an extra partition to your scheme. The split will work now.

Sometimes you want to merge two partitions. That you do with the MERGE statement:

ALTER PARTITION FUNCTION pfSample()

MERGE RANGE(1);

When this completes you have one partition less.

That's it for tonight. I'll do one more post on partitioning and then I cover swithing in and switching out staging tables.

Synonyms and other Oracle stuff

Menzo Steinhorst — Thu, 26 Jun 2008 21:50:00 GMT

Every now and then I meet Oracle specialists at customers sites. Sometimes they start a good discussion on the differences between the Oracle engine and the SQL Server engine. I don't mind a good discussion on the differences but sadly lots of Oracle guys compare 11G to SQL Server 6.5. (btw don't get me wrong : Oracle has a great engine, but we definitely closed the gap!)

Two features they always refer to are schema's and synonyms. Schema's are very important in Oracle. Basically Oracle uses schema's like SQL Server uses databases. When you install an Oracle you have 1 database and you can create schema's in there to separate different data. Since SQL 2005 we also have schema's. Nothing will stop you for using them like you did in Oracle. But usually they are used to group related objects in 1 database. I must say I do not see many databases where they are used at all. Too bad, because it's a great way to group objects which share the same security. You only have to give permissions on schema's and not on the separate objects anymore.

Synonyms is another thing. In Oracle you create synonyms to use objects from 1 schema in another schema. In SQL Server you can also create synonyms. Normal practice is to create synonyms in the model database. That way, every new database created after that already has the synonym. After a restart it's even in the tempDB. In SQL Server synonyms make objects available across databases and schema's. Instead of using the 3 part name like AdventureWorks.Person.Address you can use the name of the synonym. For synonyms goes the same as for schema's: I almost never see them in customer databases.

Table Partitioning in SQL2008(2)

Menzo Steinhorst — Thu, 26 Jun 2008 17:47:00 GMT

Today I proceeded with partitioning a large table in SQL 2008. The objective is to build a partition function with a sliding window. Usually you use staging tables to get to this. This customer wants to keep all his data in one table so staging tables cannot be used. That's a pity because then we can't use all the new wizards in SQL 2008.

Since I do want to show you what this wizard looks like here's the screenshot:

You can switch in a staging table or switch out to a staging table. Pretty straightforward. It's not really automating things but merely making administration easier.

For automating you still have to create a stored procedure to do your merging and splitting. And a job to execute it regularly.

Table Partitioning in SQL 2008

Menzo Steinhorst — Wed, 25 Jun 2008 23:26:00 GMT

At the moment I'm working for a client that wants to partition some tables. They need to do that because the tables have grown to immense proportions. One table has 800.000.000 rows. Obviously a lot of inserts are happening on this tables and they hope to improve the performance of these inserts by partitioning the table. At the moment there production servers are running SQL 2005 but we are going to try it on SQL2008 as well. SQL 2008 has some cool gui functions for partitions with sliding windows which you have to program yourself otherwise. Let's have a look:

When you right-click a table you'll see the following menu. In the Storage option you can create and manage partitions. Since we didn't define any partitions yet the Manage Partition option is disabled.

We go for Create Partition. First you'll get an introduction screen but after that you'll get this screen in which you can choose the partitioning column and you can align the indexes. This is great because all this involved some serious scripting in SQL 2005.

Now you'll get some screens where you can choose or create new partition functions and schemes. When you opt for create you'll get this screen in which you can set the boundaries in a graphical manner.

Eventually you'll end up in this screen where you can execute the partition scripts or save them. The nice thing of the scripts is that they also run on SQL 2005. So if you need to do some partitioning on your 2005 server it pays of to install a SQL2008 server somewhere with your database and get some scripts :).

Introduction

Menzo Steinhorst — Tue, 24 Jun 2008 20:57:00 GMT

Hi all,

I'm Menzo Steinhost. I'm a SQL Server PFE based in the Netherlands.

Before I came to Microsoft I worked for Deloitte Consulting. I did lots of different things there ranging from Oracle development to project audits. At some point I picked up SQL Server and I decided to specialize myself in it. I started way back in 1998 with SQL Server 6.5 and have experience with every version since. A few days ago I started a project on SQL 2008 and I'm very enthousiastic about it. I will tell you more about this in my next post.

My main focus is SQL Server performance tuning. But I also do Risk Assesments and other things around SQL Server. I used to be a developer so I'm also interested in typical developer stuff.

I will mainly blog about performance related issues on SQL Server. Performance tuning is a passion of mine and sometimes a rewarding job. You can make spectacular improvements, especially on poor indexed databases.

As a PFE I visit many premier accounts and on every site I see problems that you can prevent very easy. Expect very practical information on this issues on this blog. There are also some neat tools available from Microsoft and third party vendors that might be useful in your day to day life. I will elaborate on these as well.

Enjoy and I hope to hear from you!
Menzo