Anthony Bloesch's Web Log

Paging SQL Server result sets

AnthonyBloesch — Mon, 09 Nov 2009 20:21:00 GMT

Paging through result sets is an approach for reducing the network and client resources used to display large result sets. Essentially, the approach is to load only a page (e.g. 100 rows) of data at a time. It is likely that a user will only want to see a page of data. If they want to see more data then the next page is retrieved etc. Because the entire result set is probably not retrieved and does not need to be stored on the client the network and client resource demands will be lower. Ideally, latency for showing the first rows of data should also be lower since only a subset of data is needed.

The most commonly advocated SQL Server paging approach is based on the row_number ranking function.

with NumberedResult as
(
select row_number() over (order by M.[Id]) as [Row], M.[Id], M.[Name]
from [System_Runtime].[Methods] as M
)
select NR.[Id], NR.[Name]
from NumberedResult as NR
where NR.[Row] between @startRow and @stopRow;

This works by associating a row number with each row and fetching blocks of row numbers. For this to work correctly there must be a fixed row order defined by the over expression and the result set must be unchanging. Otherwise each call to the query may return results in a different order or even different results destroying the illusion of paging through a point in time result set.

A generally better approach is to use top to retrieve a page of data.

select top(@pageSize) M.[Id], M.[Name]
from [System_Runtime].[Methods] as M
where M.[Id] > @lastId
order by M.[Id]

As before, for this to work correctly there must be a fixed row order defined by the order by clause and the result set must be unchanging. The reason this will usually work better is that if the order by clause is covered by an index then the data can be found without sorting the entire data set. In theory, row_number could work this way also but in practice does not. The difference can be seen by looking at query plans—typically a row_number based query will form the entire result set while a top based approach will seek to the first index entry and then return the next page of rows in the index.

If there is no fixed row order or the result set can change then data can be read into a temporary table and processed by either of the two previous approaches.

ADO.Net has good support for paging data http://msdn.microsoft.com/en-us/library/tx1c9c2f.aspx. The Fill method has a specific overload to support paging in of data.

Below I compare the approaches for two interesting performance questions for a table of .Net Framework with 207 966 rows:

· What is the cost of retrieving the first page?

· What is the cost of retrieving all the pages?

Below are the results in seconds to perform the tests 100 times. As always it is wise to not make assumptions in performance or concurrency critical code and to test various approaches yourself.

Approach	First page (sec)	All pages (sec)
Row number	2.78	7 310.47
Top	0.26	51.17
Temporary table	111.13	139.94

When applicable, the top approach has clear performance advantages. Also it will typically use less server resources than the other approaches.

Here are the test scripts (run with the query option to discard the result set after execution but to collect client statistics).

-- row_number approach: First page
dbcc freesystemcache ('All')

declare @repetitions int = 100;

declare @i int = 0;
while @i < @repetitions
begin
declare @pageSize bigint = 100;
declare @lastrow bigint = (select count(*) from [System_Runtime].[Methods]);

declare @startRow bigint = 1;

declare @stopRow bigint = @startRow + @pageSize - 1;

with NumberedResult as
(
    select row_number() over (order by M.[Id]) as [Row], M.[Id], M.[Name]
    from [System_Runtime].[Methods] as M
)
select NR.[Id], NR.[Name]
  from NumberedResult as NR
where NR.[Row] between @startRow and @stopRow;

set @i += 1;
end;

-- row_number approach: All pages
dbcc freesystemcache ('All')

declare @repetitions int = 100;

declare @i int = 0;
while @i < @repetitions
begin
declare @pageSize bigint = 100;
declare @lastrow bigint = (select count(*) from [System_Runtime].[Methods]);

declare @startRow bigint = 1;

while @startRow < @lastRow
begin
    declare @stopRow bigint = @startRow + @pageSize - 1;

    with NumberedResult as
    (
      select row_number() over (order by M.[Id]) as [Row], M.[Id], M.[Name]
      from [System_Runtime].[Methods] as M
    )
    select NR.[Id], NR.[Name]
    from NumberedResult as NR
    where NR.[Row] between @startRow and @stopRow;

    set @startRow += @pageSize;
end;

set @i += 1;
end;

-- top approach: First page
dbcc freesystemcache ('All')

declare @repetitions int = 100;

declare @i int = 0;
while @i < @repetitions
begin
declare @pageSize bigint = 100;
declare @maxId bigint = (select max(M.Id) from [System_Runtime].[Methods] as M);

declare @lastId bigint = 0;

select top(@pageSize) M.[Id], M.[Name]
from [System_Runtime].[Methods] as M
where M.[Id] > @lastId
order by M.[Id]

set @i += 1;
end;

-- top approach: All pages
dbcc freesystemcache ('All')

declare @repetitions int = 100;

declare @i int = 0;
while @i < @repetitions
begin
declare @pageSize bigint = 100;
declare @maxId bigint = (select max(M.Id) from [System_Runtime].[Methods] as M);

declare @lastId bigint = 0;

while @lastId < @maxId
begin
    select top(@pageSize) M.[Id], M.[Name]
    from [System_Runtime].[Methods] as M
    where M.[Id] > @lastId
    order by M.[Id]

    set @lastId += @pageSize;
end;

set @i += 1;
end;

-- temporary table approach: First page
dbcc freesystemcache ('All')

declare @repetitions int = 100;

declare @i int = 0;
while @i < @repetitions
begin
declare @pageSize bigint = 100;

create table #results
(
    [Row] bigint not null primary key,
    [Id]   bigint not null,
    [Name] nvarchar(max) not null
);

insert into #results ([Row], [Id], [Name])
    select row_number() over (order by M.[Id]) as [Row], M.[Id], M.[Name]
    from [System_Runtime].[Methods] as M

declare @startRow bigint = 1;

declare @stopRow bigint = @startRow + @pageSize - 1;

select R.[Id], R.[Name]
from #results as R
where R.[Row] between @startRow and @stopRow;

drop table #results;

set @i += 1;
end;

-- temporary table approach: All pages
dbcc freesystemcache ('All')

declare @repetitions int = 100;

declare @i int = 0;
while @i < @repetitions
begin
declare @pageSize bigint = 100;

create table #results
(
    [Row] bigint not null primary key,
    [Id]   bigint not null,
    [Name] nvarchar(max) not null
);

insert into #results ([Row], [Id], [Name])
    select row_number() over (order by M.[Id]) as [Row], M.[Id], M.[Name]
    from [System_Runtime].[Methods] as M

declare @lastrow bigint = (select count(*) from #results);
declare @startRow bigint = 1;

while @startRow < @lastRow
begin
    declare @stopRow bigint = @startRow + @pageSize - 1;

    select R.[Id], R.[Name]
    from #results as R
    where R.[Row] between @startRow and @stopRow;

    set @startRow += @pageSize;
end;

drop table #results;

set @i += 1;
end;

"Oslo" repository lifecycle/versioning whitepaper

AnthonyBloesch — Tue, 07 Apr 2009 19:44:00 GMT

Repositories face issues to do with versioning schema and data as well as integrating with an organization's lifecycle processes. Here is a whitepaper I wrote on how the "Oslo" team thinks about handling various lifecycle/versioning issues http://msdn.microsoft.com/en-us/library/dd638048.aspx. The whitepaper deals with the following issues:

Application lifecycle management (ALM): Facilitating team development of software in a continuous cycle of steps (that is, definition, design, development, testing, deployment, and management). Traditionally, Source Code Control Systems (SCCS) and build systems, like Visual Studio Team Foundation Server, automate ALM.

Schema evolution: Changing schemas over time in such a way that applications continue to run unchanged or with minimal changes.

"Oslo" product versioning: Upgrading a customer's "Oslo" repository, from version n to version n+1, in such a way that the data is preserved.

Application versioning: The ability to upgrade applications, stored in the "Oslo" repository, to new versions. The upgrade process has the option to retain the data from older versions of the application.

Operational versioning: The ability to understand operational/management changes to a deployed application.

How to make a copy of the repository

AnthonyBloesch — Tue, 07 Apr 2009 02:22:00 GMT

Here is a nice article on how to make a copy of the “Oslo” repository so you can later restore it to its previous state http://blogs.msdn.com/modelcitizen/archive/2009/04/06/how-to-clean-up-the-repository.aspx. This is great for experimenting with the repository and for edit debug cycles with domains. The steps are essentially what we do internally at Microsoft.

Favorite keyboard shortcuts

AnthonyBloesch — Tue, 24 Mar 2009 18:36:00 GMT

One of the duties of working at Microsoft is the day-to-day use of pre-beta and pre-release products—we call it eating our own dog food. The idea is to find and fix real-life product issues before customers see them. In honor of the wonderful experience I have had with Windows 7 dog-food, attached are a list of my favorite shortcuts in US Letter docx, A4 docx, US Letter XPS and A4 XPS formats. I have included Windows 7, Internet Explorer 8 and SQL Server Management Studio 2008 shortcuts.

A thorough list of shortcuts for major Microsoft products can be found at http://www.microsoft.com/enable/products/keyboard.aspx.

SQL Server 2008 error handling best practice

AnthonyBloesch — Tue, 10 Mar 2009 22:45:00 GMT

Error handling in SQL Server 2008 needs careful implementation. The Microsoft “Oslo” Repository’s API has the further problem that we cannot mandate the error handling logic in our callers. Thus a stored procedure call could be in a transaction or not and in a try-catch block or not. Below is the pattern we have chosen based on experimentation and best practice guidance from the SQL Server team and other SQL Server experts. A test script for all the interesting cases can be found at http://blogs.msdn.com/anthonybloesch/attachment/9469577.ashx.

For a good overview of SQL Server error handling see http://www.sommarskog.se/error-handling-I.html and http://www.sommarskog.se/error-handling-II.html. In our design, we had the following main issues and mitigations:

· A transaction rollback will rollback to the outermost transaction but if there is an outer transaction we would like to only rollback the changes to the module’s inner transaction. The solution is to use SQL Server save points if there is an outer transaction.

· A sufficiently severe raiserror will terminate a module if there is an outer try-catch block but if there is none then execution will continue in the module. The solution is to always explicitly return after raising an error.

· Some developers like to use stored procedure return values to encode error states. The solution is to return appropriate error codes.

· SQL Server will raise warnings if the transaction depth entering and leaving a transaction do not match. The solution is to be careful.

· Triggers have an implicit transaction. The solution is to use a simplified pattern for triggers where a transaction is never started.

· Save points need unique names if modules can nest otherwise you can rollback to the wrong save point. The solution is to use a GUID to name the save points.

Here is the pattern for stored procedures (eliding our special error reporting routines):

if parameter error
begin
raiserror(N'…', 16, 0);

return -1;
end
else
begin
begin try
    declare @hasOuterTransaction bit = case when @@trancount > 0 then 1 else 0 end;
    declare @rollbackPoint nchar(32) = replace(convert(nchar(36), newid()), N'-', N'');

    if @hasOuterTransaction = 1
    begin
      save transaction @rollbackPoint;
    end
    else
    begin
      begin transaction @rollbackPoint;
    end;

    Do work;

    if @hasOuterTransaction = 0
    begin
      commit transaction;
    end;
end try
begin catch
    if xact_state() = 1
    begin
      rollback transaction @rollbackPoint;
    end;

    execute Standard module error handler;

    return -error_number();
end catch;
end;

Here is the pattern for triggers (eliding our special error reporting routines):

if parameter error
begin

rollback transaction;

raiserror(N'…', 16, 0);

return;
end
else
begin
begin try

    Do work;

end try
begin catch
    rollback transaction;

    execute Standard module error handler;

    return;
end catch;
end;

Testing strings for equality counting trailing spaces

AnthonyBloesch — Sat, 21 Feb 2009 00:13:00 GMT

The SQL standard requires that string comparisons, effectively, pad the shorter string with space characters. This leads to the surprising result that N'' ≠ N' ' (the empty string equals a string of one or more space characters) and more generally any string equals another string if they differ only by trailing spaces. This can be a problem in some contexts. Unicode combining marks further complicate the issue because for many collations two distinct byte strings should compare as equal. For example, \u212b (Å) should be equal to \u0041 (A) + \u030a (combining ring above).

There are a number of approaches to the problem that people have suggested but they all have flaws:

1. Convert the strings to varbinary values and then test for equality. This does not handle combining marks correctly.

2. Replace all the space characters with an unlikely character like the unit separator (\u001f) and then test for equality. This can cause two strings to test equal if they match except for space characters corresponding to the unlikely character.

3. Append a character that is not treated oddly by any collation like ‘X’ to each string and compare them. This could cause an error if the string is already of maximum size.

4. Use the like operator to do the equality check. This may cause two strings to test equal if they match because of wildcard characters in the second string.

5. Use XQuery’s string comparison. This does not handle combining marks correctly.

6. Test for equality normally and check that the data lengths match. This does not handle combining marks correctly.

Overall I like 3 the best as it has lowest risk.

Here is a simple test script for testing various approaches:

declare @test table

(

string1 nvarchar(100) collate Latin1_General_100_CI_AS_KS_WS,

string2 nvarchar(100) collate Latin1_General_100_CI_AS_KS_WS,

equal bit

);

insert into @test (string1, string2, equal)

values (N' ', N' ', 1),

(N' ', N' ', 0),

(nchar(0x212b), nchar(0x0041) + nchar(0x030a), 1),

(N'a', N'%', 0),

(N' ', nchar(0x0001f), 0);

select convert(varbinary(10), T.string1) as [String1], convert(varbinary(10), T.string2) as [String2],

case when case when T.string1 = T.string2 then 1 else 0 end = T.equal then N'OK'

else N'wrong'

end as [=],

case when case when convert(varbinary(max), T.string1) = convert(varbinary(max), T.string2) then 1 else 0 end = T.equal then N'OK'

else N'wrong'

end as [varbinary],

case when case when replace(T.string1, N' ', nchar(0x001f)) = replace(T.string2, N' ', nchar(0x0001f)) then 1 else 0 end = T.equal then N'OK'

else N'wrong'

end as [replace],

case when case when T.string1 + N'X' = T.string2 + N'X' then 1 else 0 end = T.equal then N'OK'

else N'wrong'

end as [append],

case when case when T.string1 like T.string2 then 1 else 0 end = T.equal then N'OK'

else N'wrong'

end as [like],

case when convert(xml, N'').exist(N'/.[sql:column("T.string1") eq sql:column("T.string2")]') = T.equal then N'OK'

else N'wrong'

end as [XQuery],

case when case when T.string1 = T.string2 and datalength(T.string1) = datalength(T.string2) then 1 else 0 end = T.equal then N'OK'

else N'wrong'

end as [datalength]

from @test as T;

This results in the table:

String₁	String₂	=	varbinary	replace	append	like	XQuery	datalength
0x2000	0x2000	OK	OK	OK	OK	OK	OK	OK
0x2000	0x20002000	wrong	OK	OK	OK	OK	OK	OK
0x2B21	0x41000A03	OK	wrong	OK	OK	OK	wrong	wrong
0x6100	0x2500	OK	OK	OK	OK	wrong	OK	OK
0x2000	0x1F00	OK	OK	wrong	OK	OK	OK	OK

Instead of triggers over views (part 1)

AnthonyBloesch — Tue, 17 Feb 2009 02:58:00 GMT

Views are useful for creating a business entity based view data while allowing for an efficient logical schema. However, views are normally not updatable--limiting their utility. However, SQL Server’s instead of triggers allow many of these views to be updatable. SQL Server view-based instead of triggers can be a tricky to use. Below are some experiments that show how to write them and some of their properties. A complete script is attached.

In instead of triggers, two pseudo-tables inserted and deleted contain the before and after values of the operation. The update and columns_updated functions indicate if a column is updated by the operation causing the trigger to fire.

Imagine the following table:

create table [Test].[TestsTable]

(

[Id] int not null identity constraint [PK_TestsTable] primary key,

[Value1] nvarchar(100) not null,

[Value2] nvarchar(100),

[Value1and2] as [Value1] + [Value2]

);

with the following view:

create view [Test].[Tests]

select T.[Id], T.[Value1], T.[Value2]

from [Test].[TestsTable] as T;

The following instead of triggers make the view updatable. The highlighted portions are to show the behavior of the inserted and deleted pseudo-tables and the update and columns_updated functions:

create trigger [Test].[InsertTestTrigger] on [Test].[Tests]

instead of insert as

begin

select case when update([Id]) then N'yes' else N'no' end as [Id Updated],

case when update([Value1]) then N'yes' else N'no' end as [Value1 Updated],

case when update([Value2]) then N'yes' else N'no' end as [Value2 Updated],

[Test].[BinaryToBinaryNumber](columns_updated()) as [Column Update Mask];

select *

from inserted

order by [Id];

select *

from deleted

order by [Id];

insert into [Test].[TestsTable] ([Value1], [Value2])

select I.[Value1], I.[Value2]

from inserted as I;

end;

create trigger [Test]. [UpdateTestTrigger] on [Test].[Tests]

instead of update as

begin

select case when update([Id]) then N'yes' else N'no' end as [Id Updated],

case when update([Value1]) then N'yes' else N'no' end as [Value1 Updated],

case when update([Value2]) then N'yes' else N'no' end as [Value2 Updated],

[Test].[BinaryToBinaryNumber](columns_updated()) as [Column Update Mask];

select *

from inserted

order by [Id];

select *

from deleted

order by [Id];

if update([Id])

begin

raiserror (N'Error: May not updatethe column [Test].[TestsTable].[Id]', 16, 0);

end

else

begin

update [Test].[TestsTable]

set [Value1] = I.[Value1],

[Value2] = I.[Value2]

from inserted as I

where [Test].[TestsTable].[Id] = I.[Id];

end;

create trigger [Test].[DeleteTestTrigger] on [Test].[Tests]

instead of delete as

begin

select case when update([Id]) then N'yes' else N'no' end as [Id Updated],

case when update([Value1]) then N'yes' else N'no' end as [Value1 Updated],

case when update([Value2]) then N'yes' else N'no' end as [Value2 Updated],

[Test].[BinaryToBinaryNumber](columns_updated()) as [Column Update Mask];

select *

from inserted

order by [Id];

select *

from deleted

order by [Id];

delete from [Test].[TestsTable]

from deleted as D

where [Test].[TestsTable].[Id] = D.[Id];

end;

Suppose the following insert statement is executed.

insert into [Test].[Tests] ([Value1], [Value2])

values (N'1-1', N'1-2'),

(N'2-1', null);

Then the following update pattern is the result. Because this is an insert operation all columns are marked as updated. Notice that the bit corresponding to the first column is the low bit not the high bit. Also, the bit set is based on the column ID not the column ordinal.

Id Updated	Value1 Updated	Value2 Updated	Column Update Mask
Yes	Yes	Yes	0b00000111

The inserted pseudo-table is:

Id	Value1	Value2
0	1-1	1-2
0	2-1	NULL

The deleted pseudo-table is empty:

Id	Value1	Value2

The resulting view is:

Id	Value1	Value2
1	1-1	1-2
2	2-1	NULL

Suppose the following insert statement is executed.

insert into [Test].[Tests] ([Value1])

values (N'3-1'),

(N'4-1');

Then the following update pattern is the result. Because this is an insert operation all columns are marked as updated even unspecified columns.

Id Updated	Value1 Updated	Value2 Updated	Column Update Mask
Yes	Yes	Yes	0b00000111

The inserted pseudo-table is:

Id	Value1	Value2
0	3-1	NULL
0	4-1	NULL

The deleted pseudo-table is:

Id	Value1	Value2

The resulting view is:

Id	Value1	Value2
1	1-1	1-2
2	2-1	NULL
3	3-1	NULL
4	4-1	NULL

Suppose the following update statement is executed.

update [Test].[Tests]

set [Value1] += N' updated';

The following update pattern is the result. Only the Value1 column is marked as updated.

Id Updated	Value1 Updated	Value2 Updated	Column Update Mask
No	Yes	No	0b00000010

The inserted pseudo-table is:

Id	Value1	Value2
1	1-1 updated	1-2
2	2-1 updated	NULL

The deleted pseudo-table is:

Id	Value1	Value2
1	1-1	1-2
2	2-1	NULL

The resulting view is:

Id	Value1	Value2
1	1-1 updated	1-2
2	2-1 updated	NULL
3	3-1	NULL
4	4-1	NULL

Suppose the following delete statement is executed.

delete from [Test].[Tests]

where [Id] > 2;

The following update pattern is the result. Since this is a delete operation no column is marked as updated.

Id Updated	Value1 Updated	Value2 Updated	Column Update Mask
No	No	No	0b

The inserted pseudo-table is:

Id	Value1	Value2

The deleted pseudo-table is:

Id	Value1	Value2
3	3-1	NULL
4	4-1	NULL

The resulting view is:

Id	Value1	Value2
1	1-1 updated	1-2
2	2-1 updated	NULL

In the next part I will look at some issues you need to be aware of when working with instead of triggers.

Large tuple uniqueness constraints in SQL Server

AnthonyBloesch — Wed, 04 Feb 2009 20:13:00 GMT

SQL Server 2008 limits unique constraints to 900 bytes of data per tuple. Here is a technique for enforcing uniqueness over larger tuple sizes. In many cases, the technique is more efficient than SQL Server’s unique constraints and can be used for performance critical cases.

Hash index based implementation

Essentially, the approach stores a hash of the columns in that need to be unique and uses this to minimize the number of rows that need to be checked when a row is inserted or updated. A trigger is used to enforce the constraint. A script with test code can be found attached.

As an example, suppose you want to add a unique constraint over Column1 and Column2 to the table:

create table [Schema].[Table]

(

[Id] int not null constraint [PK_Table] primary key,

[Column1] nvarchar(max) not null,

[Column2] nvarchar(max) null

);

Firstly add a column to store the hash and an index over the hash column:

create table [Schema].[Table]

(

[Id] int not null constraint [PK_Table] primary key,

[Column1] nvarchar(max) not null,

[Column2] nvarchar(max) null,

[Internal_Hash_Column1_Column2] as checksum([Column1], [Column2]) persisted not null

);

create index [Unique_Hash_Table_Column1_Column2] on [Schema].[Table] ([Internal_Hash_Column1_Column2]) include ([Column1], [Column2]);

If the columns are typically large then you may get better overall performance by removing the included columns from the index. Secondly, add a trigger to enforce the constraint:

create trigger [ConstraintInsertTrigger_Table] on [Schema].[Table]

after insert, update as

begin

if (update([Column1]) or update([Column2])) and

exists (select *

from [Schema].[Table] as T

inner join inserted as I on I.[Internal_Hash_Column1_Column2] = T. [Internal_Hash_Column1_Column2] and

I.[Column1] = T.[Column1] and

(I.[Column2] = T.[Column2] or (I.[Column2] is null and T.[Column2] is null))

group by I.[Internal_Hash_Column1_Column2], I.[Column1], I.[Column2]

having count(*) > 1)

begin

if @@trancount > 0

begin

rollback transaction;

end;

raiserror(N'Error: Values in the columns [Schema].[Table] ([Column1], [Column1]) must be unique.', 16, 0);

end;

Care is needed in handling nullable columns like Column2.

Below are some simple test cases. The insert statement: