Simon Ince's Blog : SQL

Flattening XML Data in SQL Server

simonince — Fri, 24 Apr 2009 19:11:00 GMT

If you’ve got some XML data in a SQL Server column, how can you flatten it out and query it or present it as though it was relational data? It turns out this is quite easy...

Setup

Let’s create a simple table to hold our data;

CREATE TABLE XmlSourceTable

(

RecordId INT IDENTITY(1,1) NOT NULL PRIMARY KEY,

XmlData XML NOT NULL

)

And we’ll define some straightforward XML to import;

<?xml version="1.0" ?>

<Root>

<Name>Simon</Name>

<Skill>Cooking</Skill>

<Skill>Cleaning</Skill>

</Skills>

</Person>

<Name>Peter</Name>

<Skill>Ironing</Skill>

</Skills>

</Person>

</Root>

Ages may have been changed to protect the innocent J

Next, we’ll import it into my table using one of the mechanisms SQL Server provides – each XML file will be imported into a single row in the target table.

INSERT INTO XmlSourceTable(XmlData)

SELECT *

FROM OPENROWSET(

BULK 'C:\XmlSource.xml', SINGLE_BLOB)

AS ImportSource

After this, if we do a quick query...

SELECT * FROM XmlSourceTable

... we can see that we get a single row back containing an ID and some XML;

RecordId XmlData

----------- -------

1 <Root><Person><Name>Simon</Name><Age>20</Age... (snip)

(1 row(s) affected)

Queries

The simplest way to extract this data is to use the CROSS APPLY keyword, as this executes a function against each row and then adds the returned data to the result set. Combining this with a method that can be called on the XML data type called nodes, we get some great results. A quick query like this;

SELECT

pref.value('(Name/text())[1]', 'varchar(50)') as PersonName,

pref.value('(Age/text())[1]', 'int') as PersonAge,

pref.query('Skills') as PersonSkills

FROM

XmlSourceTable CROSS APPLY

XmlData.nodes('/Root/Person') AS People(pref)

... yields a completely different result set to our last query;

PersonName PersonAge PersonSkills

---------- --------- ----------------------

Simon 20 <Skills><Skill>Cooking</Ski... (snip)

Peter 21 <Skills><Skill>Ironing</Ski... (snip)

(2 row(s) affected)

We can see this query has flattened my single row of relational data with embedded hierarchical XML into two rows and columns of relational data. I’ve also included a subset of the XML as a column, just to show I can! Of course, if I wanted to I could modify this to get a list of people and their skills;

SELECT

pref.value('(Name/text())[1]', 'varchar(50)') as PersonName,

sref.value('(text())[1]', 'varchar(50)') as PersonSkill

FROM

XmlSourceTable CROSS APPLY

XmlData.nodes('//Person') AS People(pref) CROSS APPLY

pref.nodes('Skills/Skill') AS Skill(sref)

What is it Doing?

This SQL can be difficult to understand when you first look at it, but it isn’t really that complex. Breaking it down there are three key concepts that we’ll cover below – using my first query above that fetches PersonName, PersonAge, and PersonSkills as an example.

CROSS APPLY

The first concept is the use of CROSS APPLY, which many people haven’t used before. What this is doing is roughly equivalent to the following steps (note: this is my idiots guide to how I think about it, not a description of how the query optimiser does it);

1. Fetch the rows from the XmlSourceTable table.

2. For each row, call the “nodes” function on the XmlData column. This could be some other function too – it needn’t be on XML data. See the docs on APPLY for more info.

3. Duplicate the XmlSourceTable row once for every row returned by the table valued function “nodes”.

4. Add the columns returned by the “nodes” function to the columns in the result set.

5. Continue doing filtering, joining, and column selection as for any other SQL query.

I hope that makes it a little clearer.

nodes() function

The XML data type in SQL Server defines a number of methods that can be called on it. One of these is “nodes” – and what this does is basically select a list of XML nodes that match an XQuery expression. Knowing this, look at the statement in my SQL;

XmlData.nodes('/Root/Person') AS People(pref)

This is using the path “/Root/Person” to ensure that all Person nodes that exist under the Root are selected. The result is aliased as a table named People, and each resulting XML node will be output as a separate row, in the “pref” column.

Plenty of alternative syntaxes are available for selecting this list of nodes, and this really is the core of how to flatten out the XML. I’ve also used “//Skill” syntax in my second query, for example, which selects every Skill node found in the XML it is used on.

Other XML functions

The last concept is the use of the selector XML functions – in my example I’ve used “value()” and “query()”. Both of these use XQuery expressions to select XML data.

Firstly, I’ve used the value() function to extract a specific value from the XML, and convert it to a SQL Server data type;

pref.value('(Name/text())[1]', 'varchar(50)') as PersonName

The “text()” function here retrieves the inner text from within the XML “Name” node. The “[1]” suffix acts as an indexer, and fetches the first result matched. I know there’s only one name node per person, but I need to make sure SQL Server knows this too. Finally, the second argument is the data type it should be converted to. For an Age, I’ve used ‘int’, but here we’re converting to a varchar type.

The “query()” function allows me to return an XML fragment;

pref.query('Skills') as PersonSkills

This returns the XML that matches the “Skills” node, and is found underneath the current Person element held in “pref” (i.e. the search is relative to the contents of pref). This means it returns fragments such as;

<Skills><Skill>Ironing</Skill></Skills>

Conclusion

None of this is all that difficult once you know how – so I hope this has given you a quick start to using XML data in SQL! Don’t forget to read up more generally on the XML data type and XML indexes.

Procedural versus Set-Based SQL

simonince — Mon, 30 Mar 2009 12:22:00 GMT

During my day job I come across customers that are struggling to make a system perform to the expectations of their users, and they often see this as a SQL Server problem. This post is designed to demonstrate how much of a difference your approach to SQL tasks can make to the performance and scalability of your system. Of course, the real trick is to make sure you performance test thoroughly; a system can behave totally differently in a live environment to how it was seen to behave during development and system testing.

The Scenario

For this post I’ve tried to illustrate a common scenario with the attached SQL script. Very simply, we have Customers, Orders, OrderItems, and Products tables following a very familiar relationship pattern.

I’ve then invented a set of Stored Procedures to retrieve a list of “Big Orders”. The idea is that a “Big Order” is defined according to some rules, based on the date, size of the order, a substring in the order description, and so on.

There are three types of Stored Procedure, and they are all based on approaches that I’ve regularly seen in real implementations, and therefore hopefully illustrate my points well;

1. GetBigOrdersReselecting. This procedure also uses a function named IsBigOrder. Basically it retrieves a list of Order Ids to work with, and then loops through them calling IsBigOrder with each. IsBigOrder then selects the row it needs to work with using that ID, and determines if the order is “big”. If it is, GetBigOrdersReselecting adds this ID to the result set. This means the row is “reselected” multiple times – a very common approach.

2. GetBigOrdersCursor. This procedure opens a cursor and steps through all the orders determining if the order is “big”. If it is, it adds the ID to the result set using a temporary table. This approach is used in many, many systems.

3. GetBigOrdersSetBased. This procedure uses a single SELECT statement to fetch all the orders that are considered “big”... and is my preferred approach and the reason for this post.

The Tests

What I wanted to show was how each of these approaches behaves differently as the volume of data increases. Therefore I have used a Visual Studio 2008 Data Generation Plan to fill my tables with test data (with ratios Customer:Order:OrderItem of 1:5:15). I have then run each of the Stored Procedures and captured some statistics to look at.

I did this for two test data sets – those that included less than 15% of “big” orders (“Low Hit Ratio”), and those that included about 45% of “big” orders (“High Hit Ratio”). This is intended to reflect two different types of SQL – those that retrieve and process a large percentage of the data as a batch job might, and those that need a smaller percentage of the rows, like an OLTP system might.

It is at this point you must realise that this post is indicative only. It is actually pretty unscientific due to my time constraints. I’m running SQL 2008 on my laptop, with various programs open (e.g. Outlook), using randomly generated data that is different each time, without any sensible indexing strategy, without recompiling stored procs between runs, and so on. In other words, I believe the figures show a valid pattern, but don’t go on the individual millisecond results. Just take the core message on board.

The Results

Let’s get to the point then, and look at how long each of the three approaches took to run (in seconds) for increasing volumes of records in the Order table.

	50	250	500	1,000	2,500	5,000	10,000	25,000	50,000	100,000
Set based High Hit %	0.000	0.000	0.017	0.017	0.046	0.047	0.126	0.237	0.453	0.686
Cursor High Hit %	0.000	0.013	0.030	0.046	0.140	0.313	0.437	1.007	2.074	3.687
Reselect High Hit %	0.017	0.127	0.406	1.507	8.727	5.193	20.073
Set based Low Hit %	0.000	0.000	0.013	0.017	0.033	0.064	0.046	0.186	0.343	0.513
Cursor Low Hit %	0.000	0.013	0.017	0.030	0.077	0.173	0.280	0.734	1.413	2.530
Reselect Low Hit %	0.017	0.127	0.406	1.490	8.917	5.157	19.747

Note I didn’t bother running the 25,000+ runs for the “reselect” approach. There is also an interesting anomaly in there (I wish I had time to investigate!), in that “reselect” seems to temporarily improve around the 5,000 record mark. That could be for all sorts of reasons but at least you know I’m not faking my results J

Anyway, I think two graphs illustrate the results best; the first includes the “reselect” procedure results, the second excludes it so that we can see more detail of the other two.

Including Reselect

Excluding Reselect

You can clearly see a few things from this;

1. The Set-based approach is faster than any other approach for anything but the smallest amounts of data.

2. The Cursor approach gets a lot slower quickly (fairly proportionally, at least).

3. The Reselect approach is unpredictable, and much, much slower than either of the other two approaches. It almost spirals out of control quickly - with very few records it is probably in the unacceptable performance category for an OLTP system.

It is important to understand that as these queries run slower and slower so they will also have a disproportionately bad effect on any multi-user system. There is more chance that multiple connections will be executing queries against the same data at the same time, more memory will probably be in use, the processor will be busier, and your disks might well be more active. This increases the chances of deadlocks and more. Therefore the wrong approach to SQL statements not only compromises the performance of your queries, but also the scalability of your concurrent queries, and ultimately the stability of your system.

Remember that my example just uses SELECTs, but we could easily be looking at UPDATE statements too.

Implementation

So how is this set-based approach implemented for complex logic you have traditionally used procedural code for? Quite easily actually, after a bit of practice - complex rules can often be expressed using a combination of AND/OR, CASE, and more. Have a look at the HAVING clause of my set-based query;

SELECT

... SQL removed...

HAVING

(

o.OrderDate < GETDATE() AND

(

SUM(oi.NumberOfItems * p.Price) > 100 AND

CASE SUBSTRING(o.OrderDescription, 0, 4)

WHEN '011' THEN 1

WHEN '012' THEN 1

ELSE 0

END = 1

) OR

(

SUM(oi.NumberOfItems * p.Price) <= 100 AND

SUM(oi.NumberOfItems) > 3

)

) OR

(

o.OrderDate >= GETDATE() AND

SUM(oi.NumberOfItems * p.Price) > 150

)

This is by no means perfect. I could replace SUBSTRING with LIKE, I haven’t tuned it by looking at the query plan at all, and I should consider adding some indices; I might even choose to cache some data using an indexed view, and more. But without any of this work it still runs much faster than the other options, and I think you can see from my HAVING clause how easy it is to express the logic.

I’m trying to keep this post as brief as possible, so I won’t go into approaches to writing this set-based logic – but one key point is to ensure you use mutually exclusive OR clauses (so you might duplicate some lines – such as my use of o.OrderDate), and don’t forget to test, test, test. In fact, if you’re replacing procedural code you might consider automating running both queries against the same data and comparing the results.

Summary

This is one of the easiest summaries I’ve ever written...

Whenever you’re writing SQL always start out with writing set-based statements. It may look harder at first, but it gets easier once you’ve done it a few times, and I firmly believe the effort up front pays dividends when you come to scaling your application. Set-based SQL is also far easier to tune than procedural logic using the usual SQL tricks of changing indexes, partitioning, and so on.

There are always exceptions to rules of thumb, and I’m sure you’ll have a scenario where something can’t or shouldn’t be done in a set-based way... but just be absolutely sure that’s the case. Usually it comes down to data that must be dealt with in an ordered manner, perhaps with a running cumulative total or similar.

If you try reworking some SQL into a set-based approach I’d love to hear what you find; whether it is huge performance improvements, or otherwise – feel free to comment here.

Mapping two Tables to one Entity in the Entity Framework

simonince — Mon, 23 Mar 2009 21:45:00 GMT

Whilst at Q-Con a few weeks ago someone asked me about how to map two database tables to one entity in the Entity Framework... something a lot of people want to do at some point or another. At this point I must thank Chris Barker for pinging me some links to get this working. It seems a common thing to need to do so I thought I’d post briefly here.

Options

There are actually two main options;

1. Do the mapping in the Entity Framework, which requires that both tables must share the same primary key. Therefore if you have Customer and CustomerAddress tables (with a 1:1 relationship) the CustomerId must be the primary key on the CustomerAddress table.

2. Do the mapping in the database, using an updatable View that you manually map into the Entity Framework’s model. Rick has a post about how you might do this here.

If you want to use Option 1, which I personally prefer (assuming you have control of the database schema at this point), there is a walkthrough in the documentation here. The walkthrough creates a new entity type – I actually cheated and made it even simpler.

Walkthrough

I have two simple tables – Customer and CustomerAddress. I want them to be encapsulated in a single Customer entity within my Entity Framework model.

When I generate my entity model this gives me two entities in both the Conceptual and Storage schemas, appropriately mapped. It looks a bit like this;

The problem is, I want my address fields on the Customer entity. There are two ways to achieve this... the “designer” way is to right click “Line1” and choose Cut, then Paste it into the Scalar Properties in Customer, then repeat for all the remaining properties (except of course CustomerId – that’s already there).

The XML fan way is to dive into the Conceptual Schema in the dbml file and add these properties manually. I find this quicker as I can copy/paste and tweak the names very easily and all in one operation (for example, if I want to call a field AddressLine1 instead). I end up with the following XML;

This gives us a slightly different looking interim model, as CustomerAddress starts to become a bit empty;

So now we have a nice complete Entity in our Conceptual space; next we need to map these new properties to the table in the Store. I could do this in the mapping section of the XML, or we can use the designer. The tool makes some assumptions here that actually automate mapping all the properties for you if you haven’t changed their names, so I’ll use that for simplicity.

Right-click on the Customer entity and choose “Table Mapping”. You’ll see something like the following. We can now choose to add a Table to the mapping;

As soon as I pick CustomerAddress, it pre-populates all the fields for me with a guess at their mapping. If you’ve renamed any properties in your entity you need to fix them up manually. Also, map the CustomerId field in the database to the Id property on the entity. The result is a mapping that looks like this;

Finally, delete the CustomerAddress entity from the designer (which just deletes it from the Conceptual schema, not the Storage schema – which is good because we need to table still!). In deleting this entity, the navigation property named CustomerAddress that was on Customer disappears, completing the tidy-up of our model.

... and we’re finished!

A Quick Query

Just to prove it, a nice simple query like this now works;

var customersAndAddresses = from c in db.Customer

where c.Name.StartsWith("Simon")

select c;

var result = customersAndAddresses.First();

string address = String.Format("{0}\n{1}\n{2} {3}",

result.Name,

result.Line1,

result.Line2,

result.PostCode);

Using SQL Profiler to see what’s going on shows that the Entity Framework is doing the join for us, just as we would expect;

Good huh?

Hope that helps!

Hierarchies with HierarchyID in SQL 2008

simonince — Fri, 17 Oct 2008 20:08:00 GMT

If you’ve been following my blog you should remember a couple of posts about hierarchical data in SQL Server, and how Common Table Expressions can be a great way to extract that data. If you’ve not read them, you can find them here and here.

In SQL Server 2008 we have a brand new shiny data type called “HierarchyID”. This data type is designed precisely for storing hierarchical data in a compact, efficient manner that makes it easy to perform common operations, and hence massively simplifies matters when retrieving data.

So I thought we could briefly revisit the two scenarios I’ve mentioned before and see how SQL Server 2008 makes our lives easier. Remember in my previous posts we had a table that looked something like this;

CREATE TABLE dbo.Employee

(

[Id] INT IDENTITY(1,1) NOT NULL PRIMARY KEY,

[Name] NVARCHAR(50) NOT NULL,

[ManagerId] INT NULL

)

ALTER TABLE [dbo].[Employee]

WITH CHECK ADD CONSTRAINT [FK_Employee_Employee]

FOREIGN KEY([ManagerId])

REFERENCES [dbo].[Employee] ([Id])

This is really simple – we just have a record with an ID, and a foreign key to the “parent” record (the employee’s manager in this case).

Our New Table Format

With HierarchyID, the data type stores the whole path to the current record in the “Id” column. This means we don’t have to do crazy recursion to work out who my boss’ boss is – it’s all there in the path all the time. This also means I don’t need a foreign key back to another record in the same table! Confused? Let’s see it in action.

CREATE TABLE EmployeeWithHierarchyID

(

[Id] hierarchyid not null primary key,

[Name] nvarchar(50) not null

)

... and that’s it. To insert some records, a few lines such as the following can be used;

INSERT INTO EmployeeWithHierarchyID ([Id], [Name]) VALUES

(hierarchyid::GetRoot(), 'Simon Ince'),

('/1/', 'Someone Else'),

('/1/1/', 'Another Person')

* I’ve used another nice feature of SQL 2008 here – separating multiple lines of data to insert with commas.

This “path” syntax of ‘/1/1/’ and so on means that I will be at the root of the tree, Someone Else will report to me, and Another Person will in turn report to them. Once I’ve filled my table with data, I can select it out as follows;

SELECT

Id,

Id.ToString() AS [Path],

Id.GetLevel() AS [Level],

Name

FROM EmployeeWithHierarchyID

I’ve used a couple of functions here on the HierarchyID data type to demonstrate what you can do;

Id	Path	Level	Name
0x	/	0	Adams, Ellen
0x68	/2/	1	Adams, Terry
0x78	/3/	1	Birkby, Dana
0x7C20	/3/4/	2	Bishop, Scott
0x8C	/5/	1	Caron, Rob
0x8E50	/5/6/	2	Dunker, Andrea
0x8E59C0	/5/6/7/	3	Francis, Cat
0x8E5A20	/5/6/8/	3	Gilmore, Eric
0x8E5A60	/5/6/9/	3	Hamilton, David
0x8E5A7540	/5/6/9/10/	4	Johnson, Barry

What you should be able to see here is a few things;

1. The HierarchyID field is stored in a binary format, that isn’t human readable.

2. The ToString function immediately shows the path that the binary format represents. So I know for example that Andrea Dunker reports to Rob Caron, who in turn reports to Ellen Adams... just by reading the “path” output by the ToString function.

3. The GetLevel function calculates how many steps down the hierarchy the current record is. It can do all this based on just the binary content of the HierarchyID field again.

Good huh? OK, so let’s get to replicating the queries I created in my other posts.

Scenario 1: The Easy Way Round

In the first scenario all I wanted to do was retrieve all records for employees that reported (directly or indirectly) to a specific employee. For example, every employee reports at least indirectly to Ellen Adams in the data above, but only Terry Adams and, Dana Birkby, and Rob Caron report directly to her.

It turns out this query is unbelievably simple;

SELECT

Id,

Id.ToString() AS [Path],

Id.GetLevel() AS [Level],

Name

FROM EmployeeWithHierarchyID

WHERE Id.IsDescendantOf('/5/') = 1

In this case, I select everyone that reports to Rob Caron. And that’s it! The IsDescendantOf function handles the rest for you – and of course it is very efficient because all it needs is the path information encoded in the Id field.

Of course, here I’ve used a hard coded path value as the parameter to IsDescendantOf, but I could equally have selected a value from the database based on a name, or passed one from somewhere else.

Scenario 2: Reversed!

The next scenario was how to select the entire management chain of an employee. So given a junior staff member, how do I select his boss, his boss’ boss, his boss, and so on? This is a little more complex, as we need to walk the stack going back up the chain.

At first thought, I assumed I’d need another Common Table Expression; perhaps something like this?

WITH Ancestors(Id, [Name], AncestorId)

(

SELECT

Id,

[Name],

Id.GetAncestor(1)

FROM

EmployeeWithHierarchyId

WHERE

Id = '/5/6/9/10/'

UNION ALL

SELECT

e.Id,

e.[Name],

e.Id.GetAncestor(1)

FROM

EmployeeWithHierarchyID e

INNER JOIN Ancestors a

ON e.Id = a.AncestorId

)

SELECT *, Id.ToString() FROM Ancestors

All you need to do is replace/parameterise the ‘/5/6/9/10/’ path that I’ve used to select the employee we’re interested in. This works fine, but is it the optimum way of achieving it?

Nope. Let’s try again!

DECLARE @TheEmployee hierarchyid

SELECT @TheEmployee = Id

FROM EmployeeWithHierarchyID

WHERE [Name] = 'Johnson, Barry';

SELECT

Id,

Id.ToString() AS [Path],

Id.GetLevel() AS [Level],

Id.GetAncestor(1),

Name

FROM EmployeeWithHierarchyID

WHERE @TheEmployee.IsDescendantOf(Id) = 1

This time I’ve selected the employee by name, just so I can get hold of an instance of a HierarchyID data type that relates to him. This means I can perform operations on it just like I have done on columns in the table before. So next, I just reverse the order of my IsDescendantOf call in the WHERE clause – now I say “Is Barry Johnson a descendant of the current record?”. If he is, we include it in the result set.

Simple huh?

Summary

Well, this has been a very quick post just to get you started with HierarchyID, but I hope it has made it obvious just how much easier and more efficient it can be for most hierarchies. Enjoy!

Row Versions: What’s that all about?

simonince — Tue, 20 May 2008 17:47:00 GMT

Intro

I did some work with a customer recently to understand what the different Isolation Levels they could use in SQL Server 2005 meant to their system. Of course, we’ve all seen the standard isolation levels that have been around for ages, but SQL 2005 introduces two new options;

· Read Committed Isolation using Row Versioning

· Snapshot Isolation

Both of these new isolation levels use Row Versioning to function, and both sound very similar initially, but there are subtle differences.

Background

Basically what these isolation levels do is keep track of changes to data rows, and store the history of changes in tempdb for as long as they are needed. When another client comes along that needs to read the same piece of data, the database engine may choose to serve up a previous version from tempdb.

Why is this useful? Because even if the row that I want to read is locked by a client that is currently writing to it, I can still read the previous row version.

For more info on how all this works check out the documentation online. Pay particular attention to “Choosing Row Versioning Based Isolation Levels”.

Enabling READ_COMMITTED_SNAPSHOT

To enable this type of Row Versioning, you must apply a change to the database itself;

ALTER DATABASE MyDatabase SET READ_COMMITTED_SNAPSHOT ON;

This now means that any connection to the database that asks to use the old Read Committed isolation level will actually use the new Row Versioning based implementation. For example, if the following command is issued against a database with this setting turned on, it will use Row Versioning automatically (emphasis is mine, not SQL servers J);

SET TRANSACTION ISOLATION LEVEL READ COMMITTED

SELECT * FROM MyTable

Therefore, exercise some caution when enabling this setting on existing systems.

Enabling Snapshot Isolation

Snapshot Isolation is a brand new isolation level. To use it you must enable it at the database level;

ALTER DATABASE MyDatabase SET ALLOW_SNAPSHOT_ISOLATION ON;

And then explicitly use the new isolation level in SQL (emphasis is mine again!);

SET TRANSACTION ISOLATION LEVEL SNAPSHOT

SELECT * FROM MyTable

So what’s the difference?

There are quite a few little differences, which are detailed in the documentation I linked to earlier, but there is one really key one that isn’t immediately obvious and people often skim read without taking in the implications. To quote the docs;

· “Read committed isolation using row versioning provides statement-level read consistency”

· “Snapshot isolation provides transaction-level read consistency”

What this means, is that when you are using Read Committed isolation with Row Versioning you will receive the latest version of each row available when your statement started to execute. However, when using Snapshot Isolation, you will receive the latest version of each row available when the current transaction started to execute.

Consider the following diagram;

This shows a “control” transaction running concurrently with one of five other experimental transactions, with time running vertically. These should demonstrate the differences. What we are interested in is whether the select statement in each of our experimental transactions returns Row 1 including the changes applied in our control transaction, or Row 1 as it was before the control transaction updated it.

These results are illustrated in the following table;

	Read Committed with Row Versioning	Snapshot Isolation
Experiment 1	New	Old
Experiment 2	Old	Old
Experiment 3	Old	Old
Experiment 4	New	Old
Experment 5	New	New

· “Old” implies we receive a row version that predates the control transaction’s update.

· “New” implies we receive a row version that includes the control transaction’s update.

I hope that illustrates the point. The only experiment that retrieved the latest version of the row when running under snapshot isolation was the one when the transaction started after the control transaction had committed. Read committed, however, fetches the latest committed row version irrespective of when the current transaction started.

Summary

To summarise, I’ve re-phrased the statements from the docs that we saw earlier to try and make them even more explicit;

Read Committed with Row Versioning will return the latest committed row version when the current statement started to execute.

Snapshot Isolation will return the latest committed row version when the current transaction started to execute.

Common Table Expressions, Reversed

simonince — Wed, 31 Oct 2007 12:57:00 GMT

There was an interesting question left on my last post entitled Hierarchies WITH Common Table Expressions. Basically in this post I showed a couple of ways that you can use SQL Server 2005’s new “Common Table Expressions” (the “WITH” SQL statement) to retrieve a subset of a hierarchy of data. The question concerned inverting how this was done – I had shown how to retrieve all employees that report (directly or indirectly) to one manager – but what if I wanted to retrieve the whole management chain above a specified individual?

Consider the organisation chart below;

In my previous post, I enabled you to specify the IT Director, and get the following results;

Reporting to IT Director

Head of Operations

Head of Development

Team Leader 1

Team Leader 2

Developer

That is, a whole branch of the organisation. But if instead I wanted to retrieve the management chain for the employee labelled “Developer”, how would I approach this? I have highlighted the individuals I would expect to see in the result set on the organisation chart below;

The key here is to invert the way that you think about the query. We have previously been thinking about retrieving the CEO first, and then joining it to those who report to him. Flipping that concept on its head, lets instead first retrieve the individual who’s management chain we want;

DECLARE @TheEmployee nvarchar(20)

SET @TheEmployee = “Developer”;

SELECT

Id, [Name], ManagerId

FROM

dbo.Employee

WHERE

[Name] = @TheEmployee

This is pretty easy. So what do we want next? We want this developer’s boss, but in the SQL we’ve just written we already know this boss’ Identifier from the ManagerId column! So let’s make a leap to using a CTE;

DECLARE @TheEmployee nvarchar(20)

SET @TheEmployee = “Developer”;

WITH OrganisationChart (Id, [Name], ManagerId)

AS(

SELECT

Id, [Name], ManagerId

FROM

dbo.Employee

WHERE

[Name] = @TheEmployee

UNION ALL

SELECT

emp.Id, emp.[Name], emp.ManagerId

FROM

dbo.Employee emp

INNER JOIN OrganisationChart ON

OrganisationChart.ManagerId = emp.Id

)

SELECT * FROM OrganisationChart

This looks like a lot more SQL, but it isn’t really. The first clause of the CTE is the SQL we wrote to retrieve our “Developer”. Then we have our recursive clause that joins this result to any employee that has an “Id” field that matches the “ManagerId” field for our Developer. Of course, this is recursive, so we then get the manager of our manager in the next result... and so on.

And that’s it – we can easily get the whole management chain above any individual using a single SQL statement with a single parameter.

I hope you find that useful - and can I also thank jholovacs for the question, as I think this is a great real world example of the potential of CTEs!

Hierarchies WITH Common Table Expressions

simonince — Wed, 17 Oct 2007 15:47:00 GMT

Storing hierarchical data in a database is a very common task, and SQL Server 2005 brings us Common Table Expressions (CTEs) – a very useful tool for retrieving such data both elegantly and efficiently. I haven’t actually used these before, and they came to mind during a session with a customer, so I did a bit of research to see if they would meet the customer’s needs, and thought I’d publish the path I took.

For the purposes of this post, I’ll be using the simple database schema shown below;

The key point to note is that an employee has a manager, and this is represented as a foreign key field (named Manager) on the Employee table that points to another record in the same Employee table, yielding our hierarchy.

Adapting the SQL from the Common Table Expression documentation in SQL Server Books Online results in the query below;

WITH OrganisationChart (Id, [Name], JobTitle, Manager) AS
(
      SELECT
            Id, [Name], JobTitle, Manager
      FROM
            dbo.Employee
      WHERE 
            Manager IS NULL
      UNION ALL 
      SELECT
            emp.Id, emp.[Name], emp.JobTitle, emp.Manager
      FROM
            dbo.Employee emp
      INNER JOIN OrganisationChart ON 
            emp.Manager = OrganisationChart.Id
)
SELECT * FROM OrganisationChart

Breaking down the WITH statement, our CTE, we see two distinct parts. The first SELECT fetches the top level employees in our organisation, those which have no manager. The second SELECT is the really clever bit – it selects all employees that have a manager in the results of a recursive call to the OrganisationChart CTE. This recursion builds a tree of employees and their manager when it is invoked by SELECT-ing from the CTE – the final line in the script. The documentation contains more explanation about how these clauses work together.

Running this query gives some results that might look like those below;

Now when I first ran this query, I have to be honest, my first reaction was “so what?”! As an application developer, what does this give me that a “SELECT * FROM Employee” statement wouldn’t? Surely with either statement I’ve got to load all the data into memory and build my object hierarchy before I can filter it for a specific management branch of the organisation that I’m after?

Well, yes.

The documentation adds something that makes the query slightly more useful though – namely a “level” field, which indicates what level of the hierarchy this employee is in. Our new query looks very similar to the first, then, but with one more field;

WITH OrganisationChart (Id, [Name], JobTitle, [Level], Manager) AS

(

SELECT

            Id, [Name], JobTitle, 0, Manager
      FROM
            dbo.Employee
      WHERE 
            Manager IS NULL
      UNION ALL 
      SELECT
            emp.Id, emp.[Name], emp.JobTitle, [Level] + 1, emp.Manager
      FROM
            dbo.Employee emp
      INNER JOIN OrganisationChart ON 
            emp.Manager = OrganisationChart.Id
)
SELECT * FROM OrganisationChart
ORDER BY [Level]

On the top level clause, the “Level” is fixed at zero for obvious reasons. And relying on recursion, Level is then defined in the second clause as the previous level plus one. This yields results as follows;

Now the results seem much more useful – the Level field not only gives us an order that helps us build our in-memory object hierarchy by organising the data logically, but it also may meet your full requirements – if you only want a list of people that are 3 steps removed from a Director, a simple WHERE clause on the script’s final SELECT statement would suffice.

But let’s be honest – this isn’t what I usually want. What I want to be able to do is say “show me everyone that reports to X, directly or indirectly”. That is, to retrieve one whole branch of an organisation with a simple WHERE clause. This enables me to load only the data I want into memory, minimising memory consumption, processing, network bandwidth, and so on. But at the moment I can’t do this – if I filter on the Manager column I will only get direct reports.

Well, it turns out the answer is surprisingly simple. Consider the following modified query;

WITH OrganisationChart (Id, [Name], JobTitle, [Level], Manager, [Root]) AS
(
      SELECT
            Id, [Name], JobTitle, 0, Manager, Id
      FROM
            dbo.Employee
      WHERE 
            Manager IS NULL
      UNION ALL 
      SELECT
            emp.Id, 
emp.[Name], 
emp.JobTitle, 
[Level] + 1, 
emp.Manager, 
[Root]
      FROM
            dbo.Employee emp
      INNER JOIN OrganisationChart ON 
            emp.Manager = OrganisationChart.Id
)
SELECT * FROM OrganisationChart
WHERE [Root] = 11

I have named the new field “Root”, and it holds the top-most manager for each employee. The key to making this work is that in the first SELECT clause of the WITH statement, Root is hard-coded to the Identifier of the current record. In the second SELECT clause, and therefore for all other levels, Root is selected directly out of the recursive call to the OrganisationChart CTE – so it is never modified throughout the hierarchy.

This immediately enables us to add a WHERE clause to limit the results to employees that report to a single manager, directly or indirectly, such as follows;

In this example, I have retrieved all employees that report to the Finance Director – and no-one else.

Suppose you don’t want the whole organisation branch – perhaps you want to retrieve the hierarchy that report to a manager in charge of a group of teams, further down the organisation? No problem, just change the WHERE clause in the WITH statement’s first SELECT statement; for example, instead of WHERE Manager IS NULL you could try filtering on another known field, such as WHERE [Name] = ‘A Ince’. You could of course create a Stored Procedure, and parameterise this WHERE clause to enable you to retrieve a hierarchy below any individual you specify.

So to conclude, Common Table Expressions are indeed a powerful tool to ensure you can retrieve all the data you need, and only the data you need, in a flexible and straightforward way. This post has shown just one simple way that you can optimise your data retrieval with them. Enjoy!