SQL Programmability & API Development Team Blog : Native Types and Methods

How are GUIDs compared in SQL Server 2005?

mathh — Tue, 07 Nov 2006 01:20:00 GMT

In general, equality comparisons make a lot of sense with uniqueidentifier values. However, if you find yourself needing general ordering, then you might be looking at the wrong data type and should consider various integer types instead.

If, after careful thought, you decide to order on a uniqueidentifier column, you might be surprised by what you get back.

Given these two uniqueidentifier values:

@g1= '55666BEE-B3A0-4BF5-81A7-86FF976E763F'
@g2 = '8DD5BCA5-6ABE-4F73-B4B7-393AE6BBB849'

Many people think that @g1 is less than @g2, since '55666BEE' is certainly smaller than '8DD5BCA5'. However, this is not how SQL Server 2005 compares uniqueidentifier values.

The comparison is made by looking at byte "groups" right-to-left, and left-to-right within a byte "group". A byte group is what is delimited by the '-' character. More technically, we look at bytes {10 to 15} first, then {8-9}, then {6-7}, then {4-5}, and lastly {0 to 3}.

In this specific example, we would start by comparing '86FF976E763F' with '393AE6BBB849'. Immediately we see that @g2 is indeed greater than @g1.

Note that in .NET languages, Guid values have a different default sort order than in SQL Server. If you find the need to order an array or list of Guid using SQL Server comparison semantics, you can use an array or list of SqlGuid instead, which implements IComparable in a way which is consistent with SQL Server semantics.

Sorting undefined characters in Unicode and/or Windows collation

Jun Fang — Fri, 07 Apr 2006 02:21:00 GMT

When comparing two Unicode strings in non-binary collations, SQL Server uses a library that is essentially same as the Windows API CompareStringW. It defines a weight for each recognized character, and then use the weights to compare characters. However, not all code points are defined in the sorting library. They may be undefined because:

1) The code point is not defined in Unicode standard.
2) The code point is defined in Unicode standard, but not defined by Windows yet. It takes time and effort to define linguistic sorting semantics for new characters. Windows team typically needs to work with local standard body and/or regional PMs to define sorting rules for new characters. They add new character support in every release, and try to catch up. Some characters may have font defined, therefore could be correctly displayed, but still not defined in terms of comparison. For example, NCHAR(13144) - NCHAR(13174).
3) The code point is defined in Windows, but not defined in SQL Server yet.

Windows NLS team has decided that undefined characters are ignored during string comparison, partly because there is no real good way to compare them against other defined characters. SQL Server inherited this semantics. This does cause some confusing behavior. See below examples.

declare @undefined_char1 nvarchar(10), @undefined_char2 nvarchar(10)
set @undefined_char1 = nchar(0x0000)
set @undefined_char2 = nchar(13144)
select 'Undefine characters compare equal to empty string'
where @undefined_char1 = ''
select 'All undefine characters compare equal'
where @undefined_char1 = @undefined_char2
go

create table t (c nvarchar(10))
go
create unique index it on t(c)
go
-- first insert succeeds, but second insert fails with duplicate key error.
insert t values (nchar(0x0000))
insert t values (nchar(13144))
go

As you can see, since all undefined characters compare equal, they could cause duplicate key errors. Similarly, if you create one table with name of an undefined character, and then try to create another table with another undefined character, the second table creation would fail due to duplicate names, even though the code points of the two undefined characters are different. This could also cause confusing results in string matching builtins such as CHARINDEX, PATINDEX, or LIKE.

While these results seem confusing, the basic rule is actually very simple, i.e., undefined characters are ignored during comparison and string matching. Once you understand and remember this rule, the behavior should be easy to understand.

There have been arguments whether undefined characters should be ignored. Since this is the behavior on Windows platform, and there is not a definitely better way to sort them, and for backwards compatibility, we are going to maintain this behavior.

If your app needs to work with these undefined characters and expect to treat them as regular characters, you can use binary collation. In binary collation, comparison is done purely based on code points, not linguistic rules, so there is no notion about defined vs. undefined.

Multiplication and Division with Numerics

mathh — Wed, 29 Mar 2006 20:21:00 GMT

It can be surprising to see certain results when doing numeric arithmetic:

declare @num1 numeric(38,10)
declare @num2 numeric(38,10)
set @num1 = .0000006
set @num2 = 1.0
select cast( @num1 * @num2 as numeric(38,10))

Yields: .0000010000

Instead of: .0000006000

Why?

Well, Books Online (see Precision, Scale, and Length) dictates the following rule for numeric arithmetics:

Operation	Result precision	Result scale *
e1 + e2	max(s1, s2) + max(p1-s1, p2-s2) + 1	max(s1, s2)
e1 - e2	max(s1, s2) + max(p1-s1, p2-s2) + 1	max(s1, s2)
e1 * e2	p1 + p2 + 1	s1 + s2
e1 / e2	p1 - s1 + s2 + max(6, s1 + p2 + 1)	max(6, s1 + p2 + 1)
e1 { UNION \| EXCEPT \| INTERSECT } e2	max(s1, s2) + max(p1-s1, p2-s2)	max(s1, s2)

In this case, the multiplication result precision and scale are computed as follows:

Precision = P1 + P2 + 1 = 38 + 38 + 1 = 77

Scale = S1 + S2 = 10 + 10 = 20

So conceptually, the result should be numeric(77, 20), which isn't allowed. This is where the fine prints come in:

* The result precision and scale have an absolute maximum of 38. When a result precision is greater than 38, the corresponding scale is reduced to prevent the integral part of a result from being truncated.

Books Online currently doesn't go into the specifics on how such truncation is performed.

Since the precision is higher than 38, we try to avoid truncating the integral part of the value by reducing the scale (thus truncating the decimal part of the value instead). How much scale should be sacrificed? There is no right answer. If we preserve too much, and the result of the multiplication of large numbers will be way off. If we preserve too little, multiplication of small numbers becomes an issue.

In SQL Server 2005 RTM (and previous versions), we decided preserve a minimum scale of 6 in both multiplication and division. So our numeric(77,20) is truncated as numeric(38,6), and then is then casted as numeric(38,10). However at this point it is too late, and some data has been lost. This explains the result you would see in the previous expression.

Because of this, it is important to try to minimally quantify the precision and scale of the operands involved in multiplication and division. In this case:

declare @num1 numeric(18,10)
declare @num2 numeric(18,10)
set @num1 = .0000006
set @num2 = 1.0
select cast( @num1 * @num2 as numeric(38,10))

The resulting type would be numeric(37,20). Since this type's precision and scale does not exceed our current limits, no implicit truncation is performed. We then cast the result as numeric(38,10), which doesn't cause data loss in our case.

If you cannot accurately type the values involved in multiplication and division, for example if they are procedure parameters that are called with wildly different values, you may want to look into approximate numeric types (float, real), or perhaps define your own very high capacity user defined exact numeric data type using the CLR.

I hope this was useful. Let us know if there are particular topics you'd like to hear about!

-Mat

Newsequentialid (Histrory/Benefits and Implementation)

dragant — Thu, 23 Mar 2006 19:41:00 GMT

In general, we made significant improvements in SQL Server scalability during Yukon. One of the areas of improvement is replication scalability. While doing merge replication testing we found out that scaling was severely affected by high number of I/O operations. The cause of the problem was that new rows were inserted in random disk pages. Guid generating function (newid) was returning non-sequential guids which resulted in random B-tree lookups. After some investigation we figured out that we could use the new OS function UuidCreateSequential with some byte scrambling to convince the rest of SQL engine that guids are produced in sequential order.

Therefore, we implemented the new intrinsic function, Newsequentialid which is nothing more than a wrapper over the OS function UuidCreateSequential. Therefore all of the semantics of UuidCreateSequential are present in newsequentialid intrinsic. The official documentation for UuidCreateSequential is here.

The conclusion: If you are generating guids as your row identifiers, you may be interested in newsequentialid function. For example, after replication folks started using this function they were able to completely fill their data and index pages. Also, the row inserts did not require searches through the B-trees because the last modified page was already in memory.

You should be aware of all of the limitations of UuidCreateSequential before you start using newsequentialid. For example, computers without network card will return only locally unique values and the IDs returned can be used to discover some network properties of your server.

Additional notes:

1) While testing newsequentialid in highly concurrent environments, we found that on AMD64 machines the OS function may return same values. Therefore, we put the global mutex so that only one user can be going through newsequentialid at the time.

2) The old guid generating function NewId is also a wrapper over an OS function CoCreateGuid, therefore all of the semantics of CoCreateGuid exist in NewId.

3) The only additional work that we do in newsequentialid is that we take guids and arrange bytes such that values work well with existing engine guid compare algorithm. Since input bits are unique and we only move the bytes returned by the OS, the output is also guaranteed to be unique.

4) Because two consecutive calls to this function are guaranteed to return different results, the function is marked as non-deterministic. Therefore, we opted to allow the use of this function only as default value. The reason for this decision is because algebrizer and optimizer are limited in what they can do with non-deterministic functions.