SQL Server Engine Tips

Converting from hex string to varbinary and vice versa

SQL Server Engine Team — Thu, 03 Jul 2008 01:26:00 GMT

Converting hexadecimal values to varbinary and vice versa is now easier using the XQuery functionality available from SQL Server 2005. The code samples below show how to perform the conversion(s):

-- Convert hexstring value in a variable to varbinary:

declare @hexstring varchar(max);

set @hexstring = 'abcedf012439';

select cast('' as xml).value('xs:hexBinary( substring(sql:variable("@hexstring"), sql:column("t.pos")) )', 'varbinary(max)')

from (select case substring(@hexstring, 1, 2) when '0x' then 3 else 0 end) as t(pos)

-- Convert binary value in a variable to hexstring:

declare @hexbin varbinary(max);

set @hexbin = 0xabcedf012439;

select '0x' + cast('' as xml).value('xs:hexBinary(sql:variable("@hexbin") )', 'varchar(max)');

For more details on XQuery see link below:

http://msdn.microsoft.com/en-us/library/ms189075(SQL.100).aspx

In SQL Server 2008, these conversions are even more easier since we added support directly in the CONVERT built-in function. The code samples below show how to perform the conversion(s):

declare @hexstring varchar(max);

set @hexstring = '0xabcedf012439';

select CONVERT(varbinary(max), @hexstring, 1);

set @hexstring = 'abcedf012439';

select CONVERT(varbinary(max), @hexstring, 2);

declare @hexbin varbinary(max);

set @hexbin = 0xabcedf012439;

select CONVERT(varchar(max), @hexbin, 1), CONVERT(varchar(max), @hexbin, 2);

For more details on the new CONVERT binary styles see link below:

http://msdn.microsoft.com/en-us/library/ms187928(SQL.100).aspx

Converting from Base64 to varbinary and vice versa

SQL Server Engine Team — Tue, 01 Jul 2008 01:25:00 GMT

Converting Base64 values to varbinary and vice versa is now easier using the XQuery functionality available from SQL Server 2005 onwards. The code samples below show how to perform the conversion:

-- Convert Base64 value in a variable to varbinary:

declare @str varchar(20);

set @str = '3qAAAA==';

select cast(N'' as xml).value('xs:base64Binary(sql:variable("@str"))', 'varbinary(20)');

-- Convert binary value in a variable to Base64:

declare @bin varbinary(20);

set @bin = 0xDEA00000;

select cast(N'' as xml).value('xs:base64Binary(xs:hexBinary(sql:variable("@bin")))', 'varchar(20)');

-- Convert varbinary value in a column to Base64:

select top (10) cast(N'' as xml).value('xs:base64Binary(xs:hexBinary(sql:column("qs.sql_handle")))', 'varchar(512)') as sql_handle_base64

into #t

from sys.dm_exec_query_stats as qs;

-- Convert Base64 value in a column to varbinary:

select cast(N'' as xml).value('xs:base64Binary(sql:column("t.sql_handle_base64"))', 'varbinary(20)') as sql_handle

from #t as t;

drop table #t;

For more details on XQuery see link below:

http://msdn.microsoft.com/en-us/library/ms189075(SQL.100).aspx

Differences between ISNULL and COALESCE

SQL Server Engine Team — Fri, 27 Jun 2008 03:55:00 GMT

I came across a question in the SQL Server MVP newsgroup recently about ISNULL and COALESCE usage. COALESCE basically translates to CASE expression and ISNULL is a built-in implemented in the database engine. Both ISNULL and COALESCE can be used to get the same results but there are some differences.

1. Data type determination of the resulting expression - ISNULL uses the first parameter type, COALESCE follows the CASE expression rules and returns type of value with highest precedence

2. The NULLability of result expression is different for ISNULL and COALESCE. ISNULL return value is always considered NOT NULLable (assuming the return value is a non-nullable one) whereas COALESCE is not. So the expressions ISNULL(NULL, 1) and COALESCE(NULL, 1) although equivalent have different NULLability values. This makes a difference if you are using these expressions in computed columns and creating key constraints or making return value of a scalar UDF deterministic so that it can be indexed.

Please note that I am referring to expressions that will alwahys return a non-NULLable value here. Otherwise, you can have ISNULL or COALESCE return NULL value just fine.

3. Validations for ISNULL and COALESCE is also different. For example, NULL value for ISNULL is converted to int whereas for COAELSCE you have to provide a type. Ex:

ISNULL(NULL, NULL) -- is int

COALESCE(NULL, NULL) -- Will throw an error

COALESCE(CAST(NULL as int), NULL) -- it valid and returns int

4. ISNULL takes only 2 parameters whereas COALESCE takes variable number of parameters

5. COALESCE is based on the ANSI SQL standard whereas ISNULL is a proprietary TSQL function

6. You could get different plans for queries using ISNULL & COALESCE if the expressions involve scalar sub-queries. This will make a performance difference and queries with COALESCE often fare worse here. See below repro script:

use tempdb

create table t1 ( i int );

create table t2 ( i int );

create table t3 ( i int );

set showplan_text on;

select isnull((select i from t1 where t1.i = t2.i), (select max(i) from t3))

from t2;

select coalesce((select i from t1 where t1.i = t2.i), (select max(i) from

t3))

from t2;

set showplan_text off;

drop table t1, t2;

Spool operators in query plan...

SQL Server Engine Team — Thu, 30 Aug 2007 19:38:11 GMT

I came across a question in the relationalserver.performance newsgroup where a customer was wondering about the spools seen in a recursive query execution plan. The query is shown below:

USE Northwind;
Go

WITH EmpChart AS
(
SELECT EmployeeId, ReportsTo, 1 AS treelevel
FROM Employees
WHERE (Employees.ReportsTo = 2)
UNION ALL
SELECT e.EmployeeId, e.ReportsTo, treelevel +1
FROM Employees e
JOIN EmpChart ec
ON e.ReportsTo=ec.EmployeeID
)
SELECT * FROM EmpChart;

The plan for the above query shows an index spool and a table spool. They are one and the same. The plan is shown below:

|--Index Spool(WITH STACK)
|--Concatenation
|--Compute Scalar(DEFINE:([Expr1013]=(0)))
| |--Compute Scalar(DEFINE:([Expr1003]=(1)))
| |--Clustered Index Scan(OBJECT:([Northwind].[dbo].[Employees].[PK_Employees]), WHERE:([Northwind].[dbo].[Employees].[ReportsTo]=(2)))
|--Assert(WHERE:(CASE WHEN [Expr1015]>(100) THEN (0) ELSE NULL END))
|--Nested Loops(Inner Join, OUTER REFERENCES:([Expr1015], [Recr1006], [Recr1007], [Recr1008]))
|--Compute Scalar(DEFINE:([Expr1015]=[Expr1014]+(1)))
| |--Table Spool(WITH STACK)
|--Compute Scalar(DEFINE:([Expr1009]=[Recr1008]+(1)))
|--Clustered Index Scan(OBJECT:([Northwind].[dbo].[Employees].[PK_Employees] AS [e]), WHERE:([Northwind].[dbo].[Employees].[ReportsTo] as [e].[ReportsTo]=[Recr1006]))

The index spool is also a lazy spool here meaning rows get inserted into the spool during execution of the recursive part also. Additionally, the rows from the index spool is read using a stack-like mechanism otherwise the recursive part may visit the same rows again. Here is how to read the plan with the recursive query:

1. Start with the anchor / top-most part

The anchor part of the recursive CTE first gets executed and the spool is created with index. Note the stack option also in the spool. This indicates that rows are read in a FIFO manner.

2. Next the recursive part of the query

|--Nested Loops(Inner Join, OUTER
REFERENCES:([Expr1015], [Recr1006], [Recr1007], [Recr1008]))
|--Compute
Scalar(DEFINE:([Expr1015]=[Expr1014]+(1)))
| |--Table Spool(WITH STACK)
|--Compute
Scalar(DEFINE:([Expr1009]=[Recr1008]+(1)))
|--Clustered Index
Scan(OBJECT:([Northwind].[dbo].[Employees].[PK_Employees] AS [e]),
WHERE:([Northwind].[dbo].[Employees].[ReportsTo] as
[e].[ReportsTo]=[Recr1006]))

This is the nested loop join between the spool (created in step #1 for the anchor query) and the recursive part of the query. Since this is eager spool, rows will be populated into the spool also until the recursion is completed or the maximum level is reached. The maximum level check is done using the assert operator above the nested loop join:

|--Assert(WHERE:(CASE WHEN [Expr1015]>(100) THEN (0) ELSE
NULL END))

3. Now, the way to tell which spools are related or the same is to look at the properties of the spool operator in the query plan output. The index spool has a property called NodeId which will referenced by the table spool as PrimaryNodeId property in another part of the plan.

Lastly, SQL Server can also create a plan with an eager spool which can be seen below for the query. In case of eager spool, query execution can continue only after the eager spool has been fully created. This is different from the lazy spool.

select count(distinct ShipVia), count(distinct ShipCountry)
from Orders as o, Customers as c
where o.CustomerID = c.CustomerID;

1. To read, the plan we will start again from the top part which contains the eager spool population.

| |--Table Spool
| |--Hash Match(Inner Join,
HASH:([c].[CustomerID])=([o].[CustomerID]),
RESIDUAL:([Northwind].[dbo].[Customers].[CustomerID] as
[c].[CustomerID]=[Northwind].[dbo].[Orders].[CustomerID] as
[o].[CustomerID]))
| |--Index
Scan(OBJECT:([Northwind].[dbo].[Customers].[Region] AS [c]))
| |--Clustered Index
Scan(OBJECT:([Northwind].[dbo].[Orders].[PK_Orders] AS [o]))

Here you can see the hash join between customers and orders table that populates the eager spool table.

2. The ShipVia distinct count is computed as follows by reading from the spool.

|--Compute
Scalar(DEFINE:([Expr1004]=CONVERT_IMPLICIT(int,[Expr1010],0)))
| |--Stream
Aggregate(DEFINE:([Expr1010]=COUNT([Northwind].[dbo].[Orders].[ShipVia] as
[o].[ShipVia])))
| |--Hash Match(Aggregate, HASH:([o].[ShipVia]),
RESIDUAL:([Northwind].[dbo].[Orders].[ShipVia] as [o].[ShipVia] =
[Northwind].[dbo].[Orders].[ShipVia] as [o].[ShipVia]))
| |--Table Spool

3. Similarly, the ShipCountry distinct count is computed using the same spool. You can see this by looking at the NodeId and PrimaryNodeId properties of the spool operators in the query plan or showplan xml.

|--Compute
Scalar(DEFINE:([Expr1005]=CONVERT_IMPLICIT(int,[Expr1011],0)))
|--Stream
Aggregate(DEFINE:([Expr1011]=COUNT([Northwind].[dbo].[Orders].[ShipCountry]
as [o].[ShipCountry])))
|--Hash Match(Aggregate, HASH:([o].[ShipCountry]),
RESIDUAL:([Northwind].[dbo].[Orders].[ShipCountry] as [o].[ShipCountry] =
[Northwind].[dbo].[Orders].[ShipCountry] as [o].[ShipCountry]))
|--Table Spool

4. Finally, since COUNT() aggregate always returns one row, we just do a nested loop join between the two parts of the tree above to return a row.

|--Nested Loops(Inner Join)
|--Compute
Scalar(DEFINE:([Expr1004]=CONVERT_IMPLICIT(int,[Expr1010],0)))
....
|--Compute
Scalar(DEFINE:([Expr1005]=CONVERT_IMPLICIT(int,[Expr1011],0)))

Hope this helps you read execution plans that contain the various spool operators.

Umachandar Jayachandran

SQL Server 2005 SP2 Re-release and post fixes

SQL Server Engine Team — Wed, 11 Apr 2007 21:21:16 GMT

Bob Ward from PSS has a wonderful blog article that explains the details about the re-release of SQL Server 2005 SP2 and fixes posted later. This is a must read for anyone deploying SQL Server 2005 SP2 to understand the various hotfixes, GDRs and procedures. Please visit his link for more details.

SQL Server 2005 SP2 Re-release and post fixes

-- Umachandar

New MSDN Books Online search functionality

SQL Server Engine Team — Fri, 30 Mar 2007 18:53:12 GMT

Check out the new Books Online search functionality online. The link below provides a scoped search of Books Online that returns a more precise and targeted result set. You can use it to search Books Online content quickly.

http://search.live.com/macros/sql_server_user_education/booksonline

Please direct your feedback to SQLServerUE@hotmail.com.

-- Umachandar

SQL Server 2005 Performance Dashboard Reports

SQL Server Engine Team — Thu, 29 Mar 2007 19:22:43 GMT

The SQL Server 2005 Performance Dashboard Reports are Reporting Services report files designed to be used with the Custom Reports feature introduced in the SQL Server 2005 SP2 release of SQL Server Management Studio. The reports allow a database administrator to quickly identify whether there is a current bottleneck on their system, and if a bottleneck is present, capture additional diagnostic data that may be necessary to resolve the problem. For example, if the system is experiencing waits for disk IO the dashboard allows the user to quickly see which sessions are performing the most IO, what query is running on each session and the query plan for each statement.

Common performance problems that the dashboard reports may help to resolve include:
- CPU bottlenecks (and what queries are consuming the most CPU)
- IO bottlenecks (and what queries are performing the most IO).
- Index recommendations generated by the query optimizer (missing indexes)
- Blocking
- Latch contention

The information captured in the reports is retrieved from SQL Server's dynamic management views. There is no additional tracing or data capture required, which means the information is always available and this is a very inexpensive means of monitoring your server.

Reporting Services is not required to be installed to use the Performance Dashboard Reports.

Check out these reports and experiment with them to troubleshoot your performance problems. You can find the reports at:

SQL Server 2005 Performance Dashboard Reports

Umachandar

OBJECT_NAME enhancement and OBJECT_SCHEMA_NAME addition in SQL Server 2005 SP2

SQL Server Engine Team — Fri, 23 Mar 2007 23:24:46 GMT

SQL Server 2005 SP2 has an important enhancement to OBJECT_NAME metadata function and a new OBJECT_SCHEMA_NAME metadata function. I will first describe the old functionality to give some context and demonstrate how the new features help a lot.

Please note that the examples uses DMVs that looks at metadata in each database or plan cache so performance of the queries depends on your environment, number of objects in the database, auto open/auto close setting of database, size of buffer pool or plan cache etc. So exercise caution if you run the query on a production database and it might be prudent just to restrict it to a specific database to see the results.

The OBJECT_NAME function is typically used to get the name of an object by specifying the object identifier (object_id value). For example, the query below returns the usage stats of indexes in the AdventureWorks database:

select OBJECT_NAME(i.object_id) as object_name, *
from sys.dm_db_index_usage_stats as i
where i.database_id = db_id('AdventureWorks');

Now, the above query will return the correct object name only if the query was run in the AdventureWorks database. This is because the metadata function OBJECT_NAME is database specific and uses information from the catalog tables in the current database. If you run the query in a different database (say master) then the names will either be wrong (because object_id may match a different object in master database) or NULL. So in order to get the correct object name, the query has to be executed in the context of the database you are interested (AdventureWorks in our example).

This DMV also returns information from multiple databases on the server. Let us now consider a scenario where you want to write a single query that returns say the top 5 entries from each database based on the number of user seeks that were performed on the index. The query will look like below:

select *
from (
select *, DENSE_RANK() OVER(PARTITION by database_id ORDER BY user_seeks DESC) as rnk
from sys.dm_db_index_usage_stats
) as i
where i.rnk <= 5;

To modify the query to return the user friendly object names instead of the identifiers, you will have to use dynamic SQL in older versions of SQL Server. This is because the OBJECT_NAME function always worked within the context of the current database. SQL Server 2005 SP2 removes this restriction and we now have an additional optional parameter that specifies the database id. This allows us to write a single query that can retrieve the OBJECT_NAME given the object identifier and database identifier of object in any database in a server. The modified query that lists the entries along with the object names is shown below:

select OBJECT_NAME(i.object_id, i.database_id) as objname, *
from (
select *, DENSE_RANK() OVER(PARTITION by database_id ORDER BY user_seeks DESC) as rnk
from sys.dm_db_index_usage_stats
) as i
where i.rnk <= 5;

Can you see how simple it is now to write queries that involve looking at object metadata in multiple databases? The enhancement to OBJECT_NAME now allows you to write complex queries that look at metadata in different databases without using dynamic SQL.

Lastly to round off the post, SQL Server 2005 has user schema separation and supports the ANSI SQL model of schemas. So it is always better to return object names along with their schema names. The OBJECT_SCHEMA_NAME metadata function can be used to return the schema name of a schema-scoped object like table or view by specifying the object identifier and optionally database identifier. The modified query below shows how to list the schema name also:

select OBJECT_SCHEMA_NAME(i.object_id, i.database_id) as schname
, OBJECT_NAME(i.object_id, i.database_id) as objname
, *
from (
select *, DENSE_RANK() OVER(PARTITION by database_id ORDER BY user_seeks DESC) as rnk
from sys.dm_db_index_usage_stats
) as i
where i.rnk <= 5;

The final query that shows the full object names along with the database in quoted format is shown below. The query lists the top 5 indexes in each database that had the most number of user_seeks performed.

select QUOTENAME(DB_NAME(i.database_id), '"')
+ N'.'
+ QUOTENAME(OBJECT_SCHEMA_NAME(i.object_id, i.database_id), '"')
+ N'.'
+ QUOTENAME(OBJECT_NAME(i.object_id, i.database_id), '"') as full_obj_name, *
from (
select *, DENSE_RANK() OVER(PARTITION by database_id ORDER BY user_seeks DESC) as rnk
from sys.dm_db_index_usage_stats
) as i
where i.rnk <= 5
order by i.database_id, i.rnk;

Below are some more examples on how to use the new functions.

-- Returns top 5 fragmented indexes in each database:

select QUOTENAME(DB_NAME(i.database_id), '"')
+ N'.'
+ QUOTENAME(OBJECT_SCHEMA_NAME(i.object_id, i.database_id), '"')
+ N'.'
+ QUOTENAME(OBJECT_NAME(i.object_id, i.database_id), '"') as full_obj_name
, *
from (
select *, DENSE_RANK() OVER(PARTITION by database_id ORDER BY avg_fragmentation_in_percent DESC) as rnk
from sys.dm_db_index_physical_stats(default, default, default, default, default)
where avg_fragmentation_in_percent > 0
) as i
where i.rnk <= 5
order by i.database_id, i.rnk;

-- Returns top 10 statements (in sql modules) per database that took the most elapsed time:

select coalesce(QUOTENAME(DB_NAME(t.dbid), '"'), '') /* NULL means resource db*/
+ N'.'
+ QUOTENAME(OBJECT_SCHEMA_NAME(t.objectid, t.dbid), '"')
+ N'.'
+ QUOTENAME(OBJECT_NAME(t.objectid, t.dbid), '"') as full_obj_name
, SUBSTRING(t.text, (t.statement_start_offset/2)+1,
((CASE t.statement_end_offset
WHEN -1 THEN DATALENGTH(t.text)
ELSE t.statement_end_offset
END - t.statement_start_offset)/2) + 1) AS statement_text
, t.*
from (
select *, DENSE_RANK() OVER(PARTITION BY t.dbid ORDER BY qs.total_elapsed_time) as rnk
from sys.dm_exec_query_stats as qs
cross apply sys.dm_exec_sql_text(qs.sql_handle) as t
where t.objectid is not null
) as t
where t.rnk <= 10
order by t.dbid, t.rnk;

Hope you find these enhancements in SQL Server 2005 SP2 useful and productive. Feel free to comment if you need more examples or more use cases.

Umachandar

SQL Server Performance Engineering Team Blog

SQL Server Engine Team — Wed, 28 Feb 2007 00:16:15 GMT

I would like to announce the blog from my team - SQL Server Performance Engineering. Feel free to visit our blog at http://blogs.msdn.com/sqlperf for your performance related questions and needs. Use the email link in the home page to share your feedback on performance and topics of interest.

Umachandar Jayachandran

SQL Server 2005 SP2 has been released...

SQL Server Engine Team — Mon, 19 Feb 2007 22:24:28 GMT

SQL Server 2005 Service Pack 2 has been released. You can find more information from the service pack 2 page.

This service pack extends the functionality of SQL Server in many ways. The What's New document highlights the major changes in the Database Engine, Analysis Services, Integration Services and others. The service pack also has performance and manageability enhancements that will be of interest to lot of developers and database administrators.

Please download the latest service pack and test your database applications with it to see the benefits.

Umachandar Jayachandran

Database Applications Profile Survey in http://connect.microsoft.com/sqlserver

SQL Server Engine Team — Sun, 04 Feb 2007 19:16:00 GMT

My team (Data & SQL Storage Performance Team) is conducting a survey on http://connect.microsoft.com/sqlserver to better understand SQL Server workloads. The connect link below will take you directly to the survey:

https://connect.microsoft.com/SQLServer/Survey/Survey.aspx?SurveyID=2226

Please consider providing your workload characteristics and hardware configuration to help us better understand the workloads that run on SQL Server. This is a detailed survey so we are hoping to determine more than what SQM or SQLH2 or other mechanisms offer today. Thanks for participating.

--
Umachandar Jayachandran

SQL Server 2005 SP2 December CTP is available for download

SQL Server Engine Team — Wed, 20 Dec 2006 01:37:25 GMT

SQL Server 2005 Service Pack 2 December CTP is available now for download. You can download it from:

http://www.microsoft.com/downloads/details.aspx?FamilyID=d2da6579-d49c-4b25-8f8a-79d14145500d&DisplayLang=en

Note that the December CTP has been published to the same location as the November CTP so if you come from the SQL Server 2005 CTP Home Page and get to December CTP don't be confused.

Please take this opportunity to test your applications with the new service pack and provide feedback on the new features or bugs that you encounter.

Next week, I will be posting series of articles that highlight some of the new features in SQL Server 2005 SP2 release.

Thanks

Umachandar

Download SQL Server 2005 SP2 CTP

SQL Server Engine Team — Wed, 08 Nov 2006 21:04:00 GMT

SQL Server 2005 Service Pack 2 (SP2) CTP has been released. You can provide feedback on the CTP release via http://connect.microsoft.com/sqlserver. The link for the various downloads is:

http://www.microsoft.com/sql/ctp.mspx

SQL Server 2005 SP2 CTP contains some new features in the database engine (vardecimal data types, object_name function enhancement, logon triggers, Common Criteria Compliance, sys.dm_exec_text_query_plan DMV, plan cache improvements among others). The list of bugs fixed in the release can be found at:

http://support.microsoft.com/default.aspx?scid=kb%3ben-us%3b921896

Please let me know if you have questions regarding the new features. For any issues with setup or bugs in the CTP release please use the connect feedback mechanism or post in the SQL Server MSDN forums.

Disabled anonymous comments...

SQL Server Engine Team — Sat, 04 Nov 2006 18:34:00 GMT

I have disabled anonymous comments due to the inordinate number of spam comments. And since I had email notifications for comments it was pretty bad. Hopefully this is fine.

Top 5 relational data warehouse performance improvements you would like to see in next version of SQL Server

SQL Server Engine Team — Wed, 20 Sep 2006 23:35:00 GMT

Hello Everyone,

We are currently looking at top performance issues for various workloads and how we can improve those in next version of SQL Server. We are collecting feedback from various customer sources. I would like to extend an invitation to readers here for feedback on top 5 relational data warehouse performance improvements you would like to see in SQL Server.

Please bear the following in mind:

1. Briefly describe the relational data warehouse workload or scenario or application. More details you can provide for us the better.

2. Prioritize the list of performance improvements in order of importance or impact to your workload. You can also just highlight the top 5 performance problems with running your relational data warehouse workload in SQL Server

3. Feel free to suggest any feature(s) that will help in improving the performance of the particular data warehouse workload or scenario

I will consolidate the feedback and post summary version of it as a link in this post.

Thanks
Umachandar