Stuart Padley's Web Log

Continued...

stuartpa — Sat, 16 Dec 2006 20:25:00 GMT

An unwanted surprise when changing the SQL Server 2005 Service Account

stuartpa — Wed, 12 Oct 2005 21:13:00 GMT

When changing the Microsoft SQL Server 2005 Service Account you might get back this rather alarming error message box:

---------------------------
WMI Provider Error
---------------------------
The process terminated unexpectedly. [0x8007042b]
---------------------------
OK
---------------------------

This is related to the problem documented in Books On Line on the page titled: 'Start and Restarting Services' (accessible in the index as 'starting SQL Server'). I quote from the paragraph that describes the problem:

"If SQL Server 2005 is shut down with open shared memory or a named pipe connection and is then restarted with the credentials of a different user, it may fail to start. This is normal behavior for connection handles controlled by the operating system. To restart SQL Server, end all SQL Server client processes manually or by restarting the computer. Possible errors include "Server shared memory provider failed to initialize. Error: 0x5" and "Server named pipe provider failed to listen on [ \\.\pipe\sql\query ]. Error: 0x5" (The pipe name may vary.)"

The problem described in the paragraph above manifests itself as the 'The process terminated unexpectedly. [0x8007042b]' error message box when using the SQL Configuration Manager to change the SQL Server (Engine) Service Account. The workaround, as stated, is to close all client processes (connecting to the named pipe) or reboot the server.

The problem is caused by an Operating System limitation which has been removed in Windows Vista. In Windows Vista a change has been made to allow the tear down of a named pipe listener when only the server side connections are closed. In Windows 2003 and earlier, there is a requirement to close down the client side connections as well, and this requirement prevents sqlservr.exe from creating a new properly ACLed named piper listener when the service is restarted with new Service Account credentials.

I hope you will find this post using your favorite search engine should you be panicked by this error message in the wild. I will also push to get a KB article written with this information in it.

A strange problem with 'ADD SIGNATURE' and the new SQLCMD command line utility

stuartpa — Tue, 04 Oct 2005 01:28:00 GMT

This one is a little odd: But, when using the new 'ADD SIGNATURE' DDL in SQL Server 2005, if you are using SQLOLEDB through the SQLNCLI (SQL Native Client) protocol stack, i.e. the stack the new command line utility SQLCMD uses, you will get the following 'Incorrect syntax' error when trying to issue an 'ADD SIGNATURE' command:

C:\>sqlcmd -S .
1> ADD SIGNATURE TO object_signature BY CERTIFICATE my_certificate WITH PASSWORD = 'sd#@@$FDajdlksaj'
2> go
Msg 156, Level 15, State 1, Server <server>, Line 1
Incorrect syntax near the keyword 'ADD'.

No, you are not going mad, the syntax is fine; but, under the covers SQLNCLI is thinking the keyword 'ADD' is a user created stored procedure and is automatically prepending 'exec ' to the command. You can see this with your own eyes by turning on profiler, you will see the following batch getting executed (notice the prepended 'exec '):

exec ADD SIGNATURE TO object_signature BY CERTIFICATE my_certificate WITH PASSWORD = 'sd#@@$FDajdlksaj'

This is a mistake in our SQLOLEDB protocol stack which we hope to fix in SQL Server 2005 Service Pack 1. In the mean time, the workaround is simple, if not a little awkward. You can prepend a ';' to your ADD SIGNATURE DDL i.e.:

C:\>sqlcmd -S .
1> ;ADD SIGNATURE TO object_signature BY CERTIFICATE my_certificate WITH PASSWORD = 'sd#@@$FDajdlksaj'
2> go

The ';' will confuse the parsing going on in SQLOLEDB so that it does not add the 'exec ', so everything will work fine. We are not aware of any other DDL that if affected by this, but if you find any please let me know.

Again, this is only a problem with SQLOLEDB in SQLNCLI (SQL Native Client). OSQL is okay, because this uses ODBC, so is the old Query Analyzer. The new SQL Management Studio is also fine, because this uses the new SQLClient managed stack.

An error starting a SQL Server 2005 User Instance when using Remote Desktop

stuartpa — Tue, 04 Oct 2005 00:50:00 GMT

Late last week, I was using the new 'User Instance' support in SQL Express 2005. I was connecting from home using Remote Desktop to my main office machine, and when trying to connect to SQL Express, I was becoming increasingly frustrated with the following error:

Error: 'Failed to generate a user instance of SQL Server due to a failure in starting the process for the user instance. The connection will be closed.'

It turns out there is a bug in Windows XP Service Pack 2 (on my office machine) which prevents connecting to User Instances. It is documented in this knowledge base article (896613), although the article doesn't give the exact connection error message so I found it hard to find. The article also doesn't mention this is specific to 'user instances', i.e. connecting to the parent instance is fine. It is also SQL Express specific, regardless of whether the version of Express came with Visual Studio or you obtained it from somewhere else (i.e. the SQL Express download page here).

The problem exists if the session you are using was started using Remote Desktop. i.e. you rebooted you office computer from home and logged back on to it (from home) using Remote Desktop. The problem would exist at home over Remote Desktop and it would also still exist in the office if you just 'unlocked' the session which was started from home. You would need to logoff and log back on again on your office machine. Luckily there is a hotfix available for the bug (as described in the article), and it should be included in Windows XP Service Pack 3.

Protecting the SQL Server 2005 'Service Master Key', the root of all encryption

stuartpa — Mon, 26 Sep 2005 02:47:00 GMT

Microsoft SQL Server 2005 introduces a new feature set, known internally as 'Secret Storage'. Secret Storage is the overall umbrella technology for the data encryption facilities in the SQL Server database engine, including the key management story. The key management of a SQL Server 2005 instance is rooted in the Service Master Key (SMK), a 128 bit 3DES key which is encrypted using the DPAPI and the service account credentials. All other keys are protected in someway by the Service Master Key. There is a diagram here of the key management hierarchy, and an excellent post with more details form Laurentiu here.

There is also a second encryption of the Service Master Key encrypted using the 'machine key'. Both encryptions can be viewed using the sys.key_encryptions view in the 'master' database:

SELECT * FROM sys.key_encryptions WHERE key_id = 102

key_id thumbprint    crypt_type crypt_type_desc             crypt_property
102    0x01          ESKM       ENCRYPTION BY MASTER KEY    0xFFFFFFFFEC00000...
102    0x03          ESKM       ENCRYPTION BY MASTER KEY    0xFFFFFFFFEC00000...

The key internally recognized with the 0x01 thumbprint is an encryption of the Service Master Key using the (SQL Engine) service account credentials. The key internally recognized with the 0x03 thumbprint is an encryption of the Service Master Key using the machine key credentials.

The Service Master Key is loaded into memory at database start up time. The Service Master Key is first decrypted using the 0x01 key and service account credentials, if that fails, it is decrypted using the 0x03 key and local machine key credentials, and in this failure case, we then replace the original encryption of the Service Master Key using the service account credentials with a new encryption using the new service account credentials.

The second encryption of the Service Master Key is a recent addition because service account changes (of the SQL Engine service) might have led to loss of data. Originally (and this is the behavior in CTP16 and previous) we were attempting to de-crypt and re-encrypt the SMK when the service account was changed using the SQL Configuration Manager. However, there were occasions when the user could be surprised:

If a user uses the Windows Services Control Manager (SCM) to change the service account (instead of SQL Configuration Manager), the decryption and re-encryption would not happen (Windows SCM does not have any hooks that would allow us to do this), and if the old account was deleted before realizing this, access to the Service Master Key could be lost forever (along with all the data it encrypted).
If a Local User is used as the service account, and the password for that account is reset, access to the Service Master Key is lost (again, along with all the data it encrypted).
The code we had that decrypted and re-encrypted the SMK during Service Account change (using SQL Configuration Manager) might fail, and because Service Account change is not an atomic action (there is no rollback functionality) we could leave the SQL instance in a state where it has a new service account, but the Service Master Key was still encrypted using the old service account credentials. Again, leading to loss of access to everything the SMK is protecting.

To avoid these situations we added the second encryption of the Service Master Key using the machine key. Specifically, the machine key encryption uses DPAPI and passes the flag CRYPTPROTECT_LOCAL_MACHINE. It also uses 256 bytes of 'Entropy' (randomly generated data), which is stored in the registry under:

HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Microsoft SQL Server\<Instance Name>\Security\Entropy

This registry key is ACL'ed so only Box Administrators and the SQLServer2005MSSQLUser$<ServerName>$<InstanceName> group have access. The Entropy, while not increasing the overall strength of the encryption, does increase the difficulty for another application, running under the same service account as the SQL Server service, to compromise the encryption key.

(NOTE: In SQL Express, when using a User Instance, the Entropy is stored in a file call Entropy.bin, in the user Local Settings\Application Data\... directory).

We have only just checked this feature in, it will be available in the RTM bits, however it did not make it into the RTM Books Online, (however, everything happens seamlessly in the background, so there are no configuration options to worry about).

We take protection of the Service Master Key very seriously, and while we have done our best to make sure you never get into a situation where you cannot decrypt it, we also strongly encourage you to backup the SMK using the 'BACKUP SERVICE MASTER KEY' DDL.

SQL Server 2005 password change without server restart, small issue with 32 bit installs on x64 OS

stuartpa — Fri, 16 Sep 2005 02:40:00 GMT

In an earlier post, I said it will be possible to change the Microsoft SQL Server 2005 service account passwords without requiring a service restart. Today we found a corner case. It turns out, when you install SQL Server 2005 32 bit editions on an x64 Operating System, running in WOW64, and you don't install a native 64 bit side-by-side SQL installation as well, there is a bug in the operating system (in the API LsaCallAuthenticationPackage) that will require a service restart.

In this specific case, we will catch the specific error code from Windows (ERROR_NOT_SUPPORTED) and pop a dialog with the following text:

The requested operation is successful. Changes will not be effective until the service is restarted.

As the message says, you will need to restart the service so the new password can be used. All other configurations are fine, and when the OS bug is fixed (I would hope the next service pack for Windows 2003/Windows XP), this dialog message will not be displayed. Again, this is only for SQL Server 2005 32 bit installations, running on an x64 operating system (in WOW64) with no side-by-side native SQL Server 2005 64 bit install.

Using SQL Server Management Studio (and Profiler) Tools with SQL Express 'User Instances'

stuartpa — Mon, 05 Sep 2005 19:53:00 GMT

In the SQL Server 2005 Express Edition, available for free here, there is new connection string option: 'user instance=true'. When this connection string option is specified, the SQL Express instance you are connecting to (we call this the parent instance), spins of another instance of the sqlservr.exe process, as a client process (not a server service), running in the security context of the currently logged on user. The connection object passed back to the client is to this new user instance (we also call this the 'child instance'), running as a client process in the user's security context.

Internally (in the SQL Engine Group), we call this feature RANU, 'Run As Normal User', it enables each user to have their own sqlservr.exe process, sandboxed to their own security context.

For troubleshootig purposes, it maybe useful to use your SQL Server Management Studio and/or SQL Server Profiler tools (obtained when purchasing a Workgroup/Standard/Enterprise edition) to connect to the 'user instance'.

To do this, in the directory:

C:\Documents and Settings\<username>\Local Settings\Application Data\Microsoft\Microsoft SQL Server Data\<instance name>

There will be a file called 'error.log'. Note: This file is called 'ERRORLOG' in all other editions of SQL Server.

Load up the 'error.log' file in notepad, and look for the line:

2005-09-04 20:55:00.42 Server Server local connection provider is ready to accept connection on [ \\.\pipe\37B74C8E-45AE-4B\tsql\query ].

You can use the string: \\.\pipe\37B74C8E-45AE-4B\tsql\query, (the hex numbers after the \pipe\ will be different on your machines), in the 'Connect' dialog boxes for SQL Server Management Studio and SQL Profiler.

I can't say we have fully tested it, but I have used these tools extensively with SQL Express 'User Instances' without problem. I have found it useful for troubleshooting issues specific to 'user instances'.

Change the SQL Server Service Account password without Server Restart

stuartpa — Sat, 03 Sep 2005 05:05:00 GMT

A couple of weeks ago, we checked-in a feature to Microsoft SQL Server 2005, that allows the various Service Account passwords to be changed, using the SQL Server Configuration Manager, without having to restart the respective services.

NOTE: It is going to be very important that you never use the plain old Windows Service Control Manager (SCM) to manipulate SQL Services. The SQL Server Configuration Manager does a lot more work in the background to keep security consistent across the installation.

Previous to this feature, when changing the service account password, if you did not restart the service, after a while (i.e. after the cached authentication tickets have expired), you would notice the following error message when the sqlservr.exe process tried to access remote resources:

Logon failure: unknown user name or bad password.

The above error message requires you to stop and restart the SQL service so the new Service Account password can be read in. With this recent check-in, you will no longer see the 'Logon failure' error message, the new password will be used, without having to stop and restart the service.

NOTE: SQL Server Configuration Manager does not change the password with the Windows Domain. That still has to be done using the 'Windows Security' dialog (i.e. Ctrl+Alt+Del).

This has been an often requested feature, that will increase service up time. It will be in CTP16 (which we release very shortly).

Setup Secure SQL Server Service Broker across databases in less than 10 seconds!

stuartpa — Tue, 26 Jul 2005 02:49:00 GMT

One of the most interesting new features in SQL Server 2005 is Service Broker. If you need more details on what it is and why you would want to use it, I will let the expert tell you. But, if you want to get it setup and configured on your machine now then look no further.

Below is a single T/SQL script, which you can cut and paste into a Query Window, that will have you sending messages from one database to the other in less time than it takes to say, “give me Secure-SQL-Server-Service-Broker now” (try and say that 5 times in quick succession).

It's free The reason why I think SQL Service Broker is going to be so interesting is because it enables cheap, easy, and secure deployment of hub-spoke applications. Why? Service Broker comes free with the SQL Server Express edition, and by using certificate based dialog- and transport-encryption, you can setup secure message based communications with many (potentially 1,000s, if you use the message forwarding capabilities to fan in/out) branch locations that report into a central hub.

By using the new public key/private key certificate based encryption and identity in SQL Server 2005, you don’t have to set up complicated cross branch/company/organization infrastructures (Active Directory, SSL, Kerberos, SQL Authentication etc.). NOTE: The hub does have to run a full version of SQL Server, e.g. Standard or Enterprise (The SQL Server Express Service Broker can only send through/to a full SQL Server edition). I have a feeling there are a lot of problems that can be easily solved with the platform we have built here, it will be interesting to see how people capitalize on this.

I couldn’t find a sample anywhere on the web of how to set up Service Broker with Certificates and CREATE USER WITHOUT LOGIN IN. So, in this quick sample, we use CERTIFICATEs, AUTHORIZATION on objects, The Database MASTER KEY, USERs WITHOUT LOGINs, and QUEUEs. Because of this reliance on other features, setting up Service Broker is not quite as simple as doing your first CREATE TABLE. In fact, it is about as easy as the first time you tried to transitively close a four-way join! Still, once you have done it a few times, it seems to make sense.

Service Broker borrows from the world of Service Orientated Architecture (SOA), the concepts of Initiators and Targets (for the Client/Server among us, you might say the Initiator is the Client and the Target is the Server, but that would make the SOA folks cringe, so we won’t say it).

5 stages to glory The 5 Stages to cross-database Secure Service Broker glory using certificate based dialog security:

1. TARGET - Set up the target database, export a certificate (with public key)

2. INITIATOR - Set up the initiator database, export a certificate (with public key) and import the target certificate

3. TARGET - Set up the target database, import the certificate of the initiator user

4. INITIATOR - Create a stored proc to send messages, and send your first message

5. TARGET - Check the message got to the Target database

If you cut and paste all 5 stages into a single Query window (running against an edition other than SQL Server Express), and then Execute them in one go, you will have set-up, sent, and received your first secure SSB message across databases.

-- STAGE 1: SETUP TARGET DATABASE

USE master;

/* Create Target Database */

IF EXISTS (SELECT * FROM sys.databases WHERE name = 'Target') DROP DATABASE Target;

CREATE DATABASE Target

-- Activate the Broker in this database (can only be done with ALTER DATABASE)

ALTER DATABASE Target SET ENABLE_BROKER

USE Target;

-- Create the database master key

CREATE MASTER KEY ENCRYPTION BY PASSWORD = '<Specify a password here>'

/* Set up Target User, and place certificate (without private key) in a file */

-- Create Target User without login

IF EXISTS (SELECT * FROM sys.sysusers WHERE name = 'TargetUser') DROP USER TargetUser;

CREATE USER TargetUser WITHOUT LOGIN

-- Create a Cert for the Initiator user

IF EXISTS (SELECT * FROM sys.certificates WHERE name = 'TargetCert') DROP CERTIFICATE TargetCert;

CREATE CERTIFICATE TargetCert

AUTHORIZATION TargetUser

WITH SUBJECT = 'Target Certificate', EXPIRY_DATE = '10/31/2009';

-- Backup the cert up so the Target database can import the cert (public key)

BACKUP CERTIFICATE TargetCert TO FILE = 'c:\TargetCert.cer';

/* Set up SSB */

-- First, we need to create a message type. This message type is very simple and allows any type of content

-- (Drop contract because it binds to message)

IF EXISTS (SELECT * FROM sys.service_contracts WHERE name = 'SubmissionContract') DROP CONTRACT SubmissionContract;

IF EXISTS (SELECT * FROM sys.service_message_types WHERE name = 'Message') DROP MESSAGE TYPE Message;

CREATE MESSAGE TYPE Message VALIDATION = NONE

-- Now create a contract that specifies what type types of messages

CREATE CONTRACT SubmissionContract (Message SENT BY INITIATOR)

-- Set up Target receive queue to hold messages

-- (Need to drop service before queue, due to binding)

IF EXISTS (SELECT * FROM sys.services WHERE name = 'TargetService') DROP SERVICE TargetService;

IF EXISTS (SELECT * FROM sys.service_queues WHERE name = 'TargetQueue') DROP QUEUE TargetQueue;

CREATE QUEUE TargetQueue

-- Create the required service and bind to be above created queue

CREATE SERVICE TargetService

AUTHORIZATION TargetUser

ON QUEUE TargetQueue (SubmissionContract)

-- Create a Local Route for the destination TargetService

IF EXISTS (SELECT * FROM sys.routes WHERE name = 'InitiatorRoute') DROP ROUTE InitiatorRoute;

CREATE ROUTE InitiatorRoute WITH SERVICE_NAME = 'InitiatorService', ADDRESS = 'LOCAL'

-- STAGE 2: SETUP INITIATOR DATABASE

USE master;

/* Create Initiator Database */

IF EXISTS (SELECT * FROM sys.databases WHERE name = 'Initiator') DROP DATABASE Initiator;

CREATE DATABASE Initiator;

-- Activate the Broker in this database

ALTER DATABASE Initiator SET ENABLE_BROKER;

USE Initiator;

-- Create the database master key

CREATE MASTER KEY ENCRYPTION BY PASSWORD = '<Specify a password here>';

/* Set up Initiator User */

-- Create Initiator User without Login

IF EXISTS (SELECT * FROM sys.sysusers WHERE name = 'InitiatorUser01') DROP USER InitiatorUser01;

CREATE USER InitiatorUser01 WITHOUT LOGIN;

-- Create a Cert with InitiatorUser AUTHORIZATION

-- This links the Cert to the User so SSB uses the CERT for Dialog security

IF EXISTS (SELECT * FROM sys.certificates WHERE name = 'InitiatorCert01') DROP CERTIFICATE InitiatorCert01;

CREATE CERTIFICATE InitiatorCert01

AUTHORIZATION InitiatorUser01

WITH SUBJECT = 'Initiator Certificate 01', EXPIRY_DATE = '10/31/2009';

-- Backup the cert up so the Target can import the cert (public key)

BACKUP CERTIFICATE InitiatorCert01 TO FILE = 'c:\InitiatorCert01.cer';

/* Set up SSB */

-- First, we need to create a message type. Note that our message type is

-- very simple and allows any type of content

-- (Need to drop contract before message, due to binding)

IF EXISTS (SELECT * FROM sys.service_contracts WHERE name = 'SubmissionContract') DROP CONTRACT SubmissionContract;

IF EXISTS (SELECT * FROM sys.service_message_types WHERE name = 'Message') DROP MESSAGE TYPE Message;

CREATE MESSAGE TYPE Message VALIDATION = NONE;

-- Once the message type has been created, we need to create a contract

-- that specifies who can send what types of messages

CREATE CONTRACT SubmissionContract (Message SENT BY INITIATOR);

-- Set up Initiator send queue to hold messages

-- (Need to drop service before queue, due to binding)

IF EXISTS (SELECT * FROM sys.services WHERE name = 'InitiatorService') DROP SERVICE InitiatorService;

IF EXISTS (SELECT * FROM sys.service_queues WHERE name = 'InitiatorQueue') DROP QUEUE InitiatorQueue;

CREATE QUEUE InitiatorQueue;

-- Create the Initiator service and bind to be above created Initiator queue

CREATE SERVICE InitiatorService

AUTHORIZATION InitiatorUser01 ON QUEUE InitiatorQueue;

-- Create a Local Route for the destination TargetService

IF EXISTS (SELECT * FROM sys.routes WHERE name = 'TargetRoute') DROP ROUTE TargetRoute;

CREATE ROUTE TargetRoute WITH SERVICE_NAME = 'TargetService', ADDRESS = 'LOCAL'

/* Set up the Target user using the Target EXPORTED CERT */

-- Create Target User without login

IF EXISTS (SELECT * FROM sys.sysusers WHERE name = 'TargetUser') DROP USER TargetUser;

CREATE USER TargetUser WITHOUT LOGIN;

-- Create a Cert from the external cert file

CREATE CERTIFICATE TargetCert

AUTHORIZATION TargetUser

FROM FILE = 'c:\TargetCert.cer'

-- Create a remote service binding (only needs to be done on the initator)

CREATE REMOTE SERVICE BINDING TargetBinding

TO SERVICE 'TargetService'

WITH USER = TargetUser

-- STAGE 3:

use Target;

/* Set up the Target user using the Target EXPORTED CERT */

-- Create Initiator User without Login

IF EXISTS (SELECT * FROM sys.sysusers WHERE name = 'InitiatorUser01') DROP USER InitiatorUser01;

CREATE USER InitiatorUser01 WITHOUT LOGIN

-- Create a Cert with InitiatorUser AUTHORIZATION

-- This links the Cert to the User so SSB uses the CERT for Dialog security

IF EXISTS (SELECT * FROM sys.certificates WHERE name = 'InitiatorCert01') DROP CERTIFICATE InitiatorCert01;

-- Create a Cert from the external cert file

CREATE CERTIFICATE InitiatorCert01

AUTHORIZATION InitiatorUser01

FROM FILE = 'c:\InitiatorCert01.cer'

GRANT SEND ON SERVICE::TargetService TO InitiatorUser01

-- STAGE 4: Create a proc send SSB Message from Initiator to Target

USE Initiator;

/* Send Proc To Send SSB Message from Initiator to Target */

IF EXISTS (SELECT * FROM sys.objects WHERE name = 'SendMessageProc') DROP PROCEDURE SendMessageProc;

CREATE PROCEDURE SendMessageProc AS

BEGIN

DECLARE @conversationHandle UNIQUEIDENTIFIER

DECLARE @message NVARCHAR(100)

BEGIN TRANSACTION;

BEGIN DIALOG @conversationHandle

FROM SERVICE InitiatorService

TO SERVICE 'TargetService'

ON CONTRACT SubmissionContract

-- Send a message on the conversation

SET @message = N'Your first cross database Secure SQL Service Broker message';

SEND ON CONVERSATION @conversationHandle

MESSAGE TYPE Message (@message)

COMMIT TRANSACTION

END

/* Send your first SSB message */

USE Initiator;

EXEC SendMessageProc;

-- STAGE 5: Make sure your message got there

USE Target;

SELECT convert( nvarchar(max), message_body ) from TargetQueue

You should see:

--------------------------------------

Your first cross database Secure SQL Service Broker message

(1 row(s) affected)

NOTE: If you run this script more than once, you will need to remove the files 'c:\InitiatorCert01.cer' and 'c:\TargetCert.cer'.

Tricky stuff Since it takes slightly over 100 lines of T/SQL to set this thing up, I won’t bore you with exactly what this all does except to say, I found the tricky parts are:

1. understanding why the ‘AUTHORIZATION’ clause matters so much on Service Broker objects when using certificate based dialog security

2. understanding why you need a database MASTER KEY

3. what is a REMOTE SERVICE BINDING anyway

4. why would you ever want to CREATE USER WITHOUT LOGIN

5. what is the magic that ensures messages actually get sent to the 'TargetService' using the credentials of the InitiatorUser01

If you are finding these points tricky yourself, don’t worry too much, I did as well. Although it all kind of makes sense to me now, the trickiness comes from securely wedging an asynchronous messaging infrastructure into what has always been a synchronous client/server system, with the emphasis on the ‘securely’ part. I can explain any of this points further is you need me to, just ask!

How about 2 machines? The purpose of this post was to get you going quickly; but, I can now hear you say, this is all fine and dandy, I am sending messages between two databases on the same (instance) machine. But what I really want is… send messages between databases on DIFFERENT machines, and I want to do it SECURELY. Okay, okay, if you ask nicely I will see what I can rustle up, in the mean time, you had better be installing SQL Server 2005 on that OTHER machine!

Table Valued Function magic in SQL Server 2005 using the new SQL/CLR integration

stuartpa — Thu, 21 Jul 2005 21:11:00 GMT

In my last post, I said we would look at how we could return a table that contains a directory listing of all the .xml files on my machine. Using the SQL/CLR LoadFile method we created in the last post, our goal is to come up with a single T/SQL statement that can dump all the .xml documents sitting as files in the file-system into our new SQL Server 2005 database.

Of course, to get a directory listing, I could use good old, rather unsafe xp_cmdshell:

xp_cmdshell 'dir c:\*.xml /s /b'

(NOTE: In SQL Server 2005 we have turned xp_cmdshell off by default, to among other things, encourage people not to use it!)

The xp_cmdshell route works, sort of. But what we also need is a ‘Last Write Time’ so we can do subsequent incremental loads. This makes the xp_cmdshell route trickier; I have to come up with some ‘dir command’ and then start parsing the xp_cmdshell output. Yuck!

Another xp_cmdshell problem The account that is used to get the directory listing is the service account of the sqlservr.exe process. What I want is the credentials of the user who is asking for the directory listing to be used to access the file system.

We are going to look at a new feature in SQL Server 2005 which will give us a safe, secure, fast and rich directory file listing, these are SQL CLR Table Valued Functions (also known as TVF) and a glimpse of the new security EXECUTE AS framework that will ensure the right, correctly privileged, security credentials ~~are used to access the filesystem~~. [UPDATE 7/21: I am wrong here. EXECUTE AS does NOT affect the security credentials we use when leaving the sqlserver.exe process. We always use the service account credentials to access resources external to the process.]

Show me the TVF magic TVFs are powerful, but there is a little bit of ‘magic source’ you just have to know, and that is the magic behind the ‘FillRow’ method. The ‘FillRow’ method is specified by adding the SqlFunction ‘function attribute’ to the header of your C# function, like this:

[SqlFunction(FillRowMethodName = "FillRow")]

public static IEnumerable DirectoryList( ...

The method name specified in the FillRowMethodName is implicitly called by the SQL/CLR framework each time the MoveNext() method on the returned IEnumerable object (or type that implements the IEnumerable interface) is called. The FillRow method must have a signature that looks like this:

private static void FillRow(Object obj, out <col1_type> <col1>, out <col2_type> <col2>, ... )

Where the 1st parameter (Object obj), once cracked as an object array, contains the values of one output row. The subsequent function parameters (out <col1_type> <col1> etc.)are ‘out’ parameters that contain the values that will appear in the columns for the row that is being constructed. If you don’t get this now, don’t worry, the code is usually just a simple pivot of a list of array elements from the cracked Object to the out parameter variables.

Stream me a table of files This how it is done:

using System;

using System.IO;

using System.Collections;

using Microsoft.SqlServer.Server;

public partial class UserDefinedFunctions

{

[SqlFunction(FillRowMethodName = "FillRow")]

public static IEnumerable DirectoryList(string rootDir, string wildCard, bool subDirectories)

{

ArrayList rowsArray = new ArrayList(); // Already implements IEnumerable, so we don't have to

DirectorySearch(rootDir, wildCard, subDirectories, rowsArray);

return rowsArray;

}

The entry function must be static and return a class that implements the IEnumerable interface. Notice the ‘magic source’, the SqlFunction attribute that specifies the FillRow method name, this FillRow method get called implicitly each time the SQL/CLR framework implicit calls the MoveNext() method on the rowsArray collection, and it is required for SQL/CLR TVFs.

Next, we go down to the filesystem and search the Directories, and then Files for the files that match the wildCard, adding each matching file as an element to the rowsArray collection.

      private static void DirectorySearch(string directory, string wildCard, bool subDirectories, ArrayList rowsArray)

      {

            GetFiles(directory, wildCard, rowsArray);

            if (subDirectories)

            {

                  foreach (string d in Directory.GetDirectories(directory))

                  {

                        DirectorySearch(d, wildCard, subDirectories, rowsArray);

                  }

            }

      }

      private static void GetFiles(string d, string wildCard, ArrayList rowsArray)

      {

            foreach (string f in Directory.GetFiles(d, wildCard))

            {

                  FileInfo fi = new FileInfo(f);

                  object[] column = new object[2];

                  column[0] = fi.FullName;

                  column[1] = fi.LastWriteTime;

                  rowsArray.Add(column);

            }

      }

Here is the magic FillRow method, it gets called once each time the framework calls .MoveNext() on the underlying ArrayList method (which implements the inherited IEnumerable interface)

private static void FillRow(Object obj, out string filename, out DateTime date)

{

object[] row = (object[])obj;

filename = (string)row[0];

date = (DateTime)row[1];

}

};

The FillRow method passes in an object as the 1st parameter, this object is then cracked as an array whose elements need to be assigned to the out parameters which match the columns for the row being constructed.

Save the source above as a single file ‘directorylist.cs’, and build it as an assembly library:

csc /target:library /out:"c:\DirectoryList.dll" "directorylist.cs"

(TIP: There maybe several csc.exe’s hanging out on your machine. You can find out which one you need by using the new sys.dm_clr_properties dynamic management view)

SELECT value FROM sys.dm_clr_properties WHERE name = 'directory'

value

-----------------------------------------------

C:\WINDOWS\Microsoft.NET\Framework\v2.0.50215\

To add the assembly and expose the C# function as a T/SQL function, run this:

IF EXISTS (SELECT * FROM sys.objects WHERE name = 'DirectoryList')

DROP FUNCTION DirectoryList;

IF EXISTS (SELECT * FROM sys.assemblies WHERE name = 'SqlClrAssembly')

DROP ASSEMBLY SqlClrAssembly;

CREATE ASSEMBLY SqlClrAssembly

FROM 'C:\DOCUME~1\stuartpa\LOCALS~1\Temp\SqlClrObjects.dll'

WITH PERMISSION_SET = EXTERNAL_ACCESS;

CREATE FUNCTION DirectoryList ( @root_directory nvarchar(max),

@wildcard nvarchar(max),

@subdirectories bit )

RETURNS TABLE (filename nvarchar(max), last_write_time datetime)

AS EXTERNAL NAME SqlClrAssembly.UserDefinedFunctions.DirectoryList;

By default SQL/CLR functionailty is what we call ‘Off By Default’ (or 'OBD'), this means that while we can do the above CREATE ASSEMBLY and CREATE FUNCTION, we cannot actually invoke the function. To turn the SQL/CLR functionality on, run the following command:

sp_configure "clr enabled", 1;

RECONFIGURE

Now we can test the function we have just created:

SELECT filename FROM dbo.DirectoryList( 'c:\windows', '*.xml', 1 )

filename last_write_time

-------------------------------------------------------- -----------------------

c:\windows\$NtServicePackUninstall$\dataspec.xml 2002-11-21 13:53:13.813

c:\windows\$NtServicePackUninstall$\filelist.xml 2002-08-29 05:00:00.000

c:\windows\$NtServicePackUninstall$\lclmm.xml 2002-08-29 05:00:00.000

c:\windows\$NtServicePackUninstall$\xsl-mappings.xml 2002-08-29 05:00:00.000

…

Putting it all together As I promised in my last post, we can now write a single T/SQL statement to insert the contents of all my (well-formed) .xml files as rows in the xml_documents table. Using the LoadFile SQL/CLR function and the xml_documents table created in my last post, with the DirectoryList SQL/CLR TVF we created in this post, the single T/SQL statement below dumps the contents (using dbo.LoadFile) of all the .xml files on my filesystem (as found with DirectoryList) into the XML data type column in the xml_documents table.

/* Insert the contents of all .xml files on c: into the xml_documents table */

INSERT INTO xml_documents

SELECT dbo.LoadFile( filename )

FROM dbo.DirectoryList( 'c:\windows', '*.xml', 0 )

WHERE last_write_time > 'the last time I ran it'

Getting those XML files into your brand spanking new relational database system

stuartpa — Wed, 20 Jul 2005 00:15:00 GMT

Up until now, XML documents have generally languished as mere files in the lowly file-system, our relational database systems haven’t seen them as important enough to warrant inclusion, up until SQL Server 2005 that is. In SQL Server 2005, XML documents are now first class (data type) citizens; no more painful shredding required to hammer those triangular trees into oblong tables.

With the new XML data type in SQL Server 2005, the rush is now on to get our XML documents out of the file-system and into the relational database, where they can, at last, feel validated (XML Schema validated that is!).

Let’s take a look at how we can get those XML documents sitting in the filesystem into SQL Server 2005.

A BULKed up OPENROWSET provider We have extended the OPENROWSET provider with the ‘BULK’ option; this allows the provider to read data directly from a file. We then added three optional arguments (SINGLE_CLOB, SINGLE_NCLOB, SINGLE_BLOB) for the BULK option, these arguments enable the contents of a file to be returned as a single-row, single-column row-set of, respectively, VARCHAR(max), NVARCHAR(max) or VARBINARY(max).

Example: The SINGLE_CLOB (non-Unicode) and SINGLE_NCLOB (Unicode) arguments enable us to read an XML file and insert it into an XML data type column.

[UPDATE 7/19: John Gallardo, rightly points out... "For importing XML data using OPENROWSET we recommend people use SINGLE_BLOB rather than SINGLE_CLOB or SINGLE_NCLOB.". I have updated the usage below. Thanks John!]

CREATE TABLE xml_documents( x XML );

INSERT INTO xml_documents ( x )

SELECT * FROM OPENROWSET(BULK N'myXmlFile.xml', SINGLE_BLOB) AS x;

A parameterization problem That was easy, but there is a problem. I have a lot of .xml files on my machine and they are all called different names. Unfortunately OPENROWSET does not allow me to parameterize the filename parameter (I have still yet to find out why), i.e. this fails:

DECLARE @filename NVARCHAR(max);

SET @filename = N'myOtherXmlFile.xml';

INSERT INTO xml_documents ( x )

SELECT * FROM OPENROWSET(BULK @filename, SINGLE_BLOB) AS x;

Msg 102, Level 15, State 1, Line 5

Incorrect syntax near '@filename'.

So unless all my file names are the same, which of course they are not, I am left with building statements at runtime using dynamic SQL -- slow, horrible, and often vulnerable to a sql injection attack -- which we want to avoid.

DECLARE @filename NVARCHAR(max);

SET @filename = N'myOtherXmlFile.xml';

DECLARE @SQLString NVARCHAR(max);

SET @SQLString = N'INSERT INTO xml_documents ( x ) ' +

'SELECT * FROM OPENROWSET(BULK ''' +

@filename + ''', SINGLE_BLOB) AS x';

EXEC sp_executesql @SQLString;

SQL/CLR integration to the rescue Here is a great example of how a quick, small and neat use of the CLR integration in SQL Server 2005 can save you from a T/SQL coding mess.

I create a function in C#, let’s say LoadFile( SqlString filename ), that does allow me to parameterize the filename, and then I allow access to that function from T/SQL. Here is how:

Save this file as ‘LoadFile.cs’

using System.IO;

public partial class UserDefinedFunctions

{

      [Microsoft.SqlServer.Server.SqlFunction]

      public static byte[] LoadFile( string filename )

      {

        using (BinaryReader binaryReader = new BinaryReader((Stream)File.OpenRead(filename)))

        {

            byte[] bytes = new byte[binaryReader.BaseStream.Length];

            binaryReader.Read(bytes, 0, bytes.Length);

            return bytes;

        }

      }

};

[Update 7/21: This function passes the contents of the file back as binary bytes (original post was returning a string). We return binary bytes instead of reading as text, so we do not lose the XML document encoding]

Compile the C# file (just using the standard C# compiler sitting in your Windows\Microsoft.Net directory):

C:\WINDOWS\Microsoft.NET\Framework\v2.0.50215\csc /target:library /out:"c:\temp\SqlClrObjects.dll" "LoadFile.cs"

Import the assembly library into SQL Server and expose the C# function as a TSQL function.

/* First enable SQL CLR functionality (which is off by default) */

sp_configure "clr enabled", 1

RECONFIGURE

/* Create the assembly. Do a defensive drop of assembly and function (that maybe bound to the assembly) first */

IF EXISTS (SELECT * FROM sys.objects WHERE name = N'LoadFile')

DROP FUNCTION LoadFile;

IF EXISTS (SELECT * FROM sys.assemblies WHERE name = N'SqlClrAssembly')

DROP ASSEMBLY SqlClrAssembly;

CREATE ASSEMBLY SqlClrAssembly

FROM N'c:\temp\SqlClrObjects.dll'

WITH PERMISSION_SET = EXTERNAL_ACCESS;

// Note the PERMISSION_SET = EXTERNAL_ACCESS is required because the C# function we have written accesses the file-system

/* Expose the C# function in the Assembly as a T/SQL function */

CREATE FUNCTION LoadFile ( @filename NVARCHAR(max) )

RETURNS VARBINARY(max)

AS EXTERNAL NAME SqlClrAssembly.UserDefinedFunctions.LoadFile;

/* Test the function */

SELECT CONVERT( XML, dbo.LoadFile( 'c:\myOtherXmlFile.xml' ) ) AS xml

xml

-----------------------------------------

<please>Don't forget me</please>

(1 row(s) affected)

So Easy Now, we can use our new parameterized LoadFile function to insert an XML file (where the filename is specified as a variable) into the XML data type column

DECLARE @filename NVARCHAR(max);

SET @filename = N'myOtherXmlFile.xml';

INSERT INTO xml_documents ( x )

SELECT dbo.LoadFile ( @filename );

This is far neater, safer (and I bet faster) than the horrible dynamic SQL (sp_executesql) above.

If Only If I had a table containing the names of all the .XML files on my file-system, i.e. like this:

CREATE TABLE files ( filename NVARCHAR(max) );

INSERT INTO files VALUES( N'c:\myXmlFile.xml' );

INSERT INTO files VALUES( N'c:\myOtherXmlFile.xml' );

I could write a single T/SQL statement to insert the contents of all my .xml files as rows in the xml_documents table

INSERT INTO xml_documents ( x )

SELECT dbo.LoadFile ( filename ) FROM files;

(2 row(s) affected)

SELECT * FROM xml_documents;

<please>Put me in the RDBMS</please>

<please>Don't forget me</please>

But that is for another post.

In this post we have seen some of the enhancements to the OPENROWSET provider for loading files, we have seen a glimpse of the XML data type, and we have seen what may be the shortest SQL/CLR function you will ever see. If these help get you started, let me know. If I have left you with more questions than answers, then fire the questions my way and I will get back to you.

In my next post we will look at how we can create a SQL/CLR Table Valued Function (TVF) that returns a table containing a directory listing of all the .xml files on my machine.

Wrapping up Microsoft SQL Server 2005 - looking at the new features

stuartpa — Mon, 18 Jul 2005 23:16:00 GMT

As we work on wrapping up the Microsoft SQL Server 2005 release (formerly known as 'Yukon'), I am taking some time to play with some of the newer features, looking for those last, hard to find, bugs that maybe still hiding out. As I come across topics of interest, particularly those I can't find referenced anywhere else on the web, I will post working samples here.

I work in the 'SQL Engine' group where I manage teams responsible for the quality of the Security & Metadata features, the new SQL CLR integration, and the Execution Engine (Cursors, Trace, Events, Notifications, Performance Montior etc.). During this release I also led teams in the Engine that worked on the XML integration (the new XML Datatype and XML Query) and the Engine Protocol stack, particularly the Native HTTP SOAP stack. It is these areas I find most interesting.

If you have any questions about any areas of the Database Engine in SQL Server, fire them my way and I will do my best to get them answered.