SQL Server Engine Tips : Best Practices

Differences between ISNULL and COALESCE

SQL Server Engine Team — Fri, 27 Jun 2008 06:55:00 GMT

I came across a question in the SQL Server MVP newsgroup recently about ISNULL and COALESCE usage. COALESCE basically translates to CASE expression and ISNULL is a built-in implemented in the database engine. Both ISNULL and COALESCE can be used to get the same results but there are some differences.

1. Data type determination of the resulting expression - ISNULL uses the first parameter type, COALESCE follows the CASE expression rules and returns type of value with highest precedence

2. The NULLability of result expression is different for ISNULL and COALESCE. ISNULL return value is always considered NOT NULLable (assuming the return value is a non-nullable one) whereas COALESCE is not. So the expressions ISNULL(NULL, 1) and COALESCE(NULL, 1) although equivalent have different NULLability values. This makes a difference if you are using these expressions in computed columns and creating key constraints or making return value of a scalar UDF deterministic so that it can be indexed.

Please note that I am referring to expressions that will alwahys return a non-NULLable value here. Otherwise, you can have ISNULL or COALESCE return NULL value just fine.

3. Validations for ISNULL and COALESCE is also different. For example, NULL value for ISNULL is converted to int whereas for COAELSCE you have to provide a type. Ex:

ISNULL(NULL, NULL) -- is int

COALESCE(NULL, NULL) -- Will throw an error

COALESCE(CAST(NULL as int), NULL) -- it valid and returns int

4. ISNULL takes only 2 parameters whereas COALESCE takes variable number of parameters

5. COALESCE is based on the ANSI SQL standard whereas ISNULL is a proprietary TSQL function

6. You could get different plans for queries using ISNULL & COALESCE if the expressions involve scalar sub-queries. This will make a performance difference and queries with COALESCE often fare worse here. See below repro script:

use tempdb

create table t1 ( i int );

create table t2 ( i int );

create table t3 ( i int );

set showplan_text on;

select isnull((select i from t1 where t1.i = t2.i), (select max(i) from t3))

from t2;

select coalesce((select i from t1 where t1.i = t2.i), (select max(i) from

t3))

from t2;

set showplan_text off;

drop table t1, t2;

Use of SCHEMABINDING option for TSQL UDFs can improve performance in SQL Server 2005...

SQL Server Engine Team — Tue, 13 Jun 2006 01:04:00 GMT

SQL Server 2005 has new optimization logic to use the SCHEMABINDING option to derive certain properties about the TSQL UDF. This can greatly improve performance of queries that use scalar UDFs in a SELECT statement for example. You can read all about it at the blog entry below from one of our development engineers.

http://blogs.msdn.com/sqlprogrammability/archive/2006/05/12/596424.aspx

Determining optimal MAXDOP setting from TSQL in SQL Server 2005

SQL Server Engine Team — Wed, 14 Sep 2005 22:02:00 GMT

For optimal performance of multi-processor installations, we recommend that the MAXDOP setting remain equal to the number of physical processors that are being used. For example, if the system is configured for two physical processors and four logical processors, MAXDOP should be set to 2. This is documented in the KB article:

http://support.microsoft.com/default.aspx/kb/322385

But there is no easy way to determine the number of physical processors in case of hyper-threaded CPUs for example from TSQL itself. Often you have to resort to using OS utilities or write small program using Win32 API to determine the logical processors in a hyper-threaded configuration or look at the BIOS or processor type.

In SQL Server 2005, there are set of views and table-valued functions that fall under the umbrella of Dynamic Management Views which expose lot of information about SQL Server, memory structures, SQLOS information and so on. More details can be obtained from the "Dynamic Management Views and Functions " topic in Books Online:

ms-help://MS.SQLCC.v9/MS.SQLSVR.v9.en/tsqlref9/html/cf893ecb-0bf6-4cbf-ac00-8a1099e405b1.htm

You can now derive the logical processors information from sys.dm_os_sys_info DMV in SQL Server 2005 easily. For example, the query below gives the optimal MAXDOP setting taking into account number of physical processors.

select case

when cpu_count / hyperthread_ratio > 8 then 8

else cpu_count / hyperthread_ratio

end as optimal_maxdop_setting

from sys.dm_os_sys_info;

This view also contains other interesting information like physical memory of the system, virtual memory available for processes, and number of schedulers. I will post more tips about the various interesting DMVs in SQL Server 2005. If you want to know about any particular area feel free to post a comment and I will post something on that.

[Modified: 20060517]

Recently, I came across a limitation in this system view. The hyperthread_ratio column is > 0 for the multi-core systems too. So there is no way to differentiate between a system with multi-core and hyperthreaded CPUs using the DMV. And since in the case of multi-core processor based systems, the MAXDOP value can be set to the number of CPUs the usefulness of the query is limited. The sure way to know the effect of MAXDOP setting is to test against your workload that involves parallel queries/operations.

Renaming sa account in SQL Server 2005

SQL Server Engine Team — Sat, 27 Aug 2005 23:44:00 GMT

SQL Server 2005 introduces a revamped security model including users and schema separation & more granular grantable permissions. ALTER LOGIN statement in SQL Server 2005 allows you to disable the sa account and rename it. This is a good security measure to take on SQL Servers running in mixed authentication mode for example. The statements to rename and disable sa account is below:

ALTER LOGIN sa DISABLE;

ALTER LOGIN sa WITH NAME = [sys-admin];

Getting SQL Server registry settings via SQLCLR table-valued user-defined function...

SQL Server Engine Team — Sat, 20 Aug 2005 02:47:00 GMT

SQL Server stores several configuration values like data path root, program directory, default domain etc in the registry. These values can be obtained by using the SMO Settings object but you can use it only from the client side. SMO is alo not supported in SQLCLR so it is not possible to leverage the functionality on server-side directly. So if you want to obtain the settings from TSQL then it is difficult to do. This problem comes up quite often in the newsgroups and the solutions using undocumented extended stored procedures are not pretty/advisable.

Anyway to solve this problem, I wrote a SQLCLR table-valued user-defined function (TVF) that accesses the registry settings on the server directly and returns them as a table. This can be used to easily query and get registry settings for any instance on a server. The C# code for the SQLCLR TVF is shown below:

using System;
using System.Collections;
using System.Text;
using Microsoft.Win32;
using System.Data.SqlTypes;
using System.Data.SqlClient;
using Microsoft.SqlServer.Server;

/*=========================================================================================

File:      SqlRegSettings.cs
Summary:   Table-valued function to retrieve registry settings for any
             SQL Server instance.
Parameter: InstanceName - Name of the instance whose registry entries should be retrieved
             Specify default or '' to automatically determine connected instance
             Specify NULL to refer to default instance
             Specify name of instance to get registry entries for specific instance
Date:      May 24, 2005

---------------------------------------------------------------------

THIS CODE AND INFORMATION ARE PROVIDED "AS IS" WITHOUT WARRANTY OF ANY
KIND, EITHER EXPRESSED OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE
IMPLIED WARRANTIES OF MERCHANTABILITY AND/OR FITNESS FOR A
PARTICULAR PURPOSE.

============================================================================================ */

[assembly: System.Security.Permissions.RegistryPermissionAttribute(System.Security.Permissions.SecurityAction.RequestMinimum, Read = @"HKEY_LOCAL_MACHINE\Software\Microsoft\Microsoft SQL Server")]
[assembly:CLSCompliant(true)]
namespace Microsoft.Samples.SqlServer
{
    public sealed class SqlRegSettings
    {
        const string SOFTWARE_KEY_NAME = @"Software";
        const string MICROSOFT_KEY_NAME = @"Microsoft";
        const string SQLSERVER_KEY_NAME = @"Microsoft SQL Server";
        const string DEFAULT_SQLSERVER_KEY_NAME = @"MSSQLServer";
        const string INSTANCES_KEY_NAME = @"Instance Names";
        const string PRODUCT_KEY_NAME = @"SQL";
        const string SETUP_KEY_NAME = @"Setup";
        private static readonly string[] MSSQL_VALUES = new string[11]{
            "AuditLevel", "BackupDirectory", "DefaultCollationName", "DefaultDomain", "DefaultLogin", "FullTextDefaultPath",
            "LoginMode", "Map#", "Map$", "Map_", "SetHostName"
        };
        private static readonly string[] SETUP_VALUES = new string[5] {
            "Collation", "SQLBinRoot", "SQLDataRoot", "SQLPath", "SqlProgramDir"
        };

        SqlRegSettings()
        {
        }

        [SqlFunction(Name = "SqlRegSettings_CS", FillRowMethodName = "FillRegistryRow", DataAccess = DataAccessKind.Read, TableDefinition = "RegistryValueName nvarchar(128), RegistryValue nvarchar(4000)")]
        public static IEnumerable GetSqlRegSettings(SqlString argument)
        {
            string argumentValue = null;
            string instanceName = null;
            RegistryKey instanceRoot = null;
            RegistryKey rk = null;
            Hashtable registryValues = new Hashtable();

            // Determine instance name only if default parameter value was used in TVF call
            // select * from SqlRegSettings(default)
            // And TVF parameter default is specified as ''
            if (!argument.IsNull)
            {
                argumentValue = argument.ToString();
                if (argumentValue.Length == 0)
                {
                    using (SqlConnection conn = new SqlConnection("context connection=true"))
                    {
                        conn.Open();
                        using (SqlCommand cmd = conn.CreateCommand())
                        {
                            cmd.CommandText = "select serverproperty('InstanceName') as InstanceName";
                            instanceName = (string)cmd.ExecuteScalar();
                        }
                        conn.Close();
                    }
                }
                else
                {
                    instanceName = argumentValue;
                }
            }

// Get the registry key based on instance name:
GetInstanceRootKey(instanceName, out instanceRoot);

            // Return a hashtable with the key/value pairs:
            if (instanceRoot != null)
            {
                // Get the default values under MSSQLServer:
                rk = instanceRoot.OpenSubKey(DEFAULT_SQLSERVER_KEY_NAME);
                foreach (string valueName in MSSQL_VALUES)
                {
                    if (rk.GetValue(valueName) != null)
                        registryValues.Add(valueName, rk.GetValue(valueName).ToString());
                }
                rk.Close();
                rk = null;

                // Get the values under Setup:
                rk = instanceRoot.OpenSubKey(SETUP_KEY_NAME);
                foreach (string valueName in SETUP_VALUES)
                {
                    if (rk.GetValue(valueName) != null)
                        registryValues.Add(valueName, rk.GetValue(valueName).ToString());
                }
                rk.Close();
                rk = null;
            }
            return registryValues;
        }

        private static void GetInstanceRootKey(string instanceName, out RegistryKey instanceRoot)
        {
            object registryValue = null;
            string instanceRegistryKey = null;

            // Get the registry key based on instance name:
            if (instanceName == null)
            {
                // use the default registry path
                instanceRoot = Registry.LocalMachine.OpenSubKey(SOFTWARE_KEY_NAME).OpenSubKey(MICROSOFT_KEY_NAME).OpenSubKey(DEFAULT_SQLSERVER_KEY_NAME);
            }
            else
            {
                // use the redirected entries via "Instance Names" key for SQL2005 first:
                instanceRoot = Registry.LocalMachine.OpenSubKey(SOFTWARE_KEY_NAME).OpenSubKey(MICROSOFT_KEY_NAME).OpenSubKey(SQLSERVER_KEY_NAME).OpenSubKey(INSTANCES_KEY_NAME).OpenSubKey(PRODUCT_KEY_NAME);
                if (instanceRoot != null)
                {
                    registryValue = instanceRoot.GetValue(instanceName);
                    instanceRoot.Close();
                    instanceRoot = null;

                    if (registryValue != null)
                    {
                        instanceRegistryKey = registryValue.ToString();
                        instanceRoot = Registry.LocalMachine.OpenSubKey(SOFTWARE_KEY_NAME).OpenSubKey(MICROSOFT_KEY_NAME).OpenSubKey(SQLSERVER_KEY_NAME).OpenSubKey(instanceRegistryKey);
                    }
                    else
                    {
                        // use the key based on older versions
                        instanceRoot = Registry.LocalMachine.OpenSubKey(SOFTWARE_KEY_NAME).OpenSubKey(MICROSOFT_KEY_NAME).OpenSubKey(SQLSERVER_KEY_NAME).OpenSubKey(instanceName);
                    }
                }
            }
        }

        public static void FillRegistryRow(Object obj, out string registryValueName, out string registryValue)
        {
            DictionaryEntry registryItem = (DictionaryEntry)obj;
            registryValueName = registryItem.Key.ToString();
            if (registryItem.Value != null)
            {
                registryValue = registryItem.Value.ToString();
            }
            else
            {
                registryValue = null;
            }
        }
    }
}

The code demonstrates several programming techniques including requesting for required registry permissions, how to determine registry locations for SQL Server 2000/2005 instances etc. Note that the code uses the documented registry key paths for SQL Server 2005 and access to values is restricted to a subset of what is exposed in the SMO Settings class. Below are some highlights about the C# code:

1. Use of RegistryPermissionAttribute to specifically request permission to read only the required registry keys

2. The static member REGISTRY_VALUES contains a list of registry values that are of interest. This can be modified to include additional values

3. The default for the instance name argument is set as '' in the TSQL TVF definition below. This will make the code to determine the instance name automatically by making a context connection and checking for the InstanceName property. Pass NULL to get registry settings for default instance

4. The GetInstanceRootKey sub-routine is used to get the root registry path for a specific instance. This sub-routine contains all the logic to determine the redirected instance keys in case of SQL Server 2000/2005 and default instance path

5. The Registry classes of the .NET framework is used to read the values from the SQL Server registry keys

6. Save the C# code to a file called SqlRegSettings.cs and compile using command-line "csc /t:library SqlRegSettings.cs". Copy the DLL to a path which the SQL Server 2005 instance can access it. The TSQL script below assumes that the DLL was copied to C:\Temp directory on the SQL Server.

The script to create a login that maps to the asymmetric key from DLL is shown below. This is required for the EXTERNAL_ACCESS assembly creation to work.

USE master
GO

CREATE ASYMMETRIC KEY SQLCLRSamplesKey FROM EXECUTABLE FILE = 'D:\Projects\SqlRegSettings\CS\bin\Release\SqlRegSettings.CS.dll'
CREATE LOGIN SQLCLRSamplesLogin FROM ASYMMETRIC KEY SQLCLRSamplesKey
GRANT EXTERNAL ACCESS ASSEMBLY TO SQLCLRSamplesLogin
GO

The script to create the assembly in TSQL and test the TVF is shown below:

USE <yourdb>

-- Cleanup:
if object_id('SqlRegSettings', 'FT') is not null
drop function SqlRegSettings;
if exists(select * from sys.assemblies where name = 'SqlRegSettings')
drop assembly SqlRegSettings;
go

-- Create assembly:
CREATE ASSEMBLY [SqlRegSettings]
FROM 'D:\Projects\SqlRegSettings\CS\bin\Release\SqlRegSettings.CS.dll'
WITH PERMISSION_SET = EXTERNAL_ACCESS
go

-- Create TVF:
CREATE FUNCTION [SqlRegSettings](@InstanceName nvarchar(128) = '')
RETURNS TABLE(RegistryValueName nvarchar(128), RegistryValue nvarchar(4000))
AS EXTERNAL NAME [SqlRegSettings].[Microsoft.Samples.SqlServer.SqlRegSettings].[GetSqlRegSettings];
GO

select * from SqlRegSettings(default)
go

select coalesce(case i.InstanceName when '' then '<AUTOMATICALLY DETERMINED INSTANCE>' else i.InstanceName end, '<DEFAULT INSTANCE>') as InstanceName
     , r.RegistryValueName, r.RegistryValue
from (
    -- shows how to get registry values for various instances:
    -- you can run query from any SQL Server 2005 instance on box to get all the values.
    select ''
union all
    select NULL
union all
    select 'YUKON'
union all
    select 'SHILOH_EN_CS'
     ) as i(InstanceName)
cross apply SqlRegSettings(i.InstanceName) as r
order by i.InstanceName, r.RegistryValueName;
go

You can download the entire Visual Studio solution which has both the C# and VB.NET version of the SQL CLR TVF from http://umachandar.members.winisp.net/files/SqlRegSettings.zip. Note that the TSQL TVF definition created from Visual Studio does not contain the default value for the parameter since there is no way to specify this from the IDE. So the TVF created by using the Visual Studio deploy option will error out if you specify the DEFAULT keyword for the parameter. Also, the names of the SQLCLR TVF are suffixed with _CS and _VB when created from the IDE.

Please feel free to comment on the sample or ask any questions.

Umachandar Jayachandran

Feedback requested on database mirroring...

SQL Server Engine Team — Fri, 29 Jul 2005 07:43:00 GMT

Database mirroring is one of the new high availability features in SQL Server 2005. More details on the database mirroring feature can be found at http://www.microsoft.com/technet/prodtechnol/sql/2005/dbmirror.mspx. Configuring security for database mirroring involves the process of enabling each instance (principal, mirror, and witness) to communicate with each other using the database mirroring endpoint. The security configuration can be done from the SQL Server Management Studio Security Configuration Wizard in Mirror page of database properties. This wizard handles the case when all of the instances are running under the same domain user account for example. If the instances are running under a local user account and/or part of different domains, then certificate based authentication needs to be configured for the endpoints. The steps for configuring database mirroring can be found at http://www.microsoft.com/technet/prodtechnol/sql/2005/dbmirror.mspx#EGAA or the Books Online topics.

To simplfy the process of evaluating this feature overall and especially the configuration of endpoints using certificate based authentication, I have created a set of scripts that will allow you perform the configuration. The scripts can be downloaded from http://umachandar.members.winisp.net/files/MirrorDb.zip. The scripts can also provide as learning aids for powerful scripting techniques using sqlcmd, and configuration of certificates. The zip file contains the following:

mirrordb.cmd - Command script to configure mirroring. The script takes instance names for principal, mirror, and witness & database to mirror. For additional details, please review the usage of the script by running it without any parameters. Only tested on Windows Server 2003 since it uses new CMD extensions but it will work from Windows XP.
Mirror_Init.sql - This script contains the SQLCMD variables that can be used to modify password for certificates, mirroring endpoint name and so on
Mirror_Setup.sql - This script performs the necessary configuration steps for enabling database mirroring
Mirror_Partner_Cleanup.sql - This script performs cleanup of the mirroring confguration on the partner (principal or mirror)
Mirror_Final_Cleanup.sql - This script performs final cleanup of the mirroring configuration on the witness and files created by setup
Mirror_Debug.sql - This script contains some queries that shows how to get database mirroring metadata from catalog views.

To evaluate the database mirroring feature, you can run just the "mirrordb.cmd" script. Run the script without any parameters to see usage examples, pre-requisites and description of the script. Feel free to send feedback on the database mirroring feature using the comments section of the blog entry. For questions regarding the usage of the script or problems, use the contact form in the blog or the comments section.

Thanks for trying out database mirroring feature and hoping to hear from you.

Umachandar Jayachandran

Ordering guarantees in SQL Server...

SQL Server Engine Team — Wed, 20 Jul 2005 21:20:00 GMT

Ordering guarantees of queries in various context is a common source of confusion. For example, a common workaround to make the results from querying a view ordered is to introduce TOP 100 PERCENT and ORDER BY in the view definition. But this however does not guarantee order in the actual results sent to the client since the query optimizer will re-order operations to find more efficient query plans. Note that even though this topic applies to SQL Server 2005 most of the rules are valid for SQL Server 2000 too.

Here are the scenarios that guarantee ordering:

If you have an ORDER BY in the top-most SELECT block in a query, the presentation order of the results honor that ORDER BY request
If you have a TOP in the same SELECT block as an ORDER BY, any TOP computation is performed with respect to that ORDER BY. For example, if there is a TOP 5 and ORDER BY clause then SQL Server picks the TOP 5 rows within a given sort. Note that this does not guarantee that subsequent operations will somehow retain the sort order of a previous operation. The query optimizer re-orders operations to find more efficient query plans
Cursors over queries containing ORDER BY in the top-most scope will navigate in that order
INSERT queries that use SELECT with ORDER BY to populate rows guarantees how identity values are computed but not the order in which the rows are inserted
SQL Server 2005 supports a number of new "sequence functions" like RANK(), ROW_NUMBER() that can be performed in a given order using a OVER clause with ORDER BY
For backwards compatibility reasons, SQL Server provides support for assignments of type SELECT @p = @p + 1 ... ORDER BY at the top-most scope.

Conor Cunningham

Query Optimizer Development Lead

Using ANSI SQL style DECLARE CURSOR statement in UDF...

SQL Server Engine Team — Wed, 22 Jun 2005 07:09:00 GMT

Today I encountered a post in the Microsoft Technical forums where a user was trying to use the ANSI SQL style DECLARE CURSOR statement which will raise error # 1049 in SQL Server. This behavior is true in SQL Server 2000. Here is a code sample that demonstrates the problem:

create function cur_test (@id int)
returns int
as
begin
declare cur_test scroll cursor for select o.id from sysobjects as o;
return 0;
end

This will produce error message:

Msg 1049, Level 15, State 1, Procedure cur_test, Line 5
Mixing old and new syntax to specify cursor options is not allowed.

Now, let us look at the reason for the error message. In TSQL, user-defined functions cannot modify state of the database or persist information between executions. This is one of the reasons to disallow insert/update/delete on permanent tables or dynamic SQL. This also means that UDFs will disallow cursors that are global in nature which happens to be the case with those declared using the ANSI SQL syntax. Note that there is no way to specify cursor as local using the ANSI SQL syntax. So we default all cursors declared within UDFs to be local implicitly. This creates a conflict in case of the DECLARE CURSOR statement using ANSI SQL syntax since that doesn't support the local option. Hence the error message about mixing old and new syntax for cursor options. The workaround is to use TSQL syntax for the DECLARE CURSOR statement to specify LOCAL or leave it out in which case we default to LOCAL anyway. Here is the modified code:

create function cur_test (@id int)
returns int
as
begin
declare cur_test cursor scroll for select o.id from sysobjects as o;
return 0;
end

-- or

create function cur_test (@id int)
returns int
as
begin
declare cur_test cursor local scroll for select o.id from sysobjects as o;
return 0;
end

Umachandar Jayachandran

Performing batched updates...

SQL Server Engine Team — Wed, 01 Jun 2005 03:56:00 GMT

Updating a large table is a common scenario. This is often encountered in cases of applications that performs series of tasks in the background or data warehousing loading scenarios. To reduce locking and logging resources, such update statements are broken down into smaller batches or units of work. This has been traditionally done using SET ROWCOUNT setting in SQL Server. Now, SQL Server 2005 provides you with a simpler construct that the optimizer can understand and use efficiently. TOP clause has been enhanced in SQL Server 2005 to support expressions and can be used now in all DML operations. Let's look at a quick example on how to use TOP with UPDATE:

use tempdb;
go

-- Create some sample data:
select * into bulkupdate from master.sys.all_objects;

-- Show the top 5 rows order by object_id:
select top (5) * from bulkupdate where schema_id = 4 order by object_id;

select CURRENT_TIMESTAMP AS UpdateStart;

-- Update 5 rows at at time until we are done:

while (@@rowcount>0)
update top(5) bulkupdate set schema_id = schema_id + 1 where schema_id = 4;

select CURRENT_TIMESTAMP AS UpdateEnd;

-- Show the updated top 5 rows order by object_id:
select top (5) * from bulkupdate order by object_id;

-- Drop the table:
drop table bulkupdate;
go

In this example, we create a table called "bulkupdate" with some sample data from one of the system tables and update 5 rows at a time for a specific condition. Note that the order in which the rows are updated is not guaranteed with the TOP clause and it depends on the query plan. This example demonstrates how to use TOP clause in UPDATE statements to split an operation that affects large number of rows into smaller manageable units.

Umachandar Jayachandran