SQL Programmability & API Development Team Blog : SQL Server 2000

Predicate ordering is not guaranteed

Jun Fang — Fri, 12 May 2006 22:40:00 GMT

A typical programmer may expect that the predicates are always evaluated in the order that they are specified, but this is not true in database systems.

For example, for the following clause,

where col11 = 5 and convert(int, col2) = 100

programmers may think col11 = 5 is always evaluated first, but this is not guaranteed. Query Optimizer may choose to first evaluate the second predicate, convert(int, col2) = 100. In fact, predicates may be pushed down the query tree, and not evaluated at the same time.

Such reordering is generally benign, as the row has to satisfy both conditions to qualify. But it may make a difference if the second predicate would cause an error if the first condition is not met. The following example demonstrates this.

Here we have a table with category and value both as string columns. From business logic, we know that when category is 'ID', value always contains a string with an integer ID. Therefore, we create a view, ID_View, that only shows the rows with category 'ID', and in the view, we will convert the value column to integer, and project as ID column.

-- Create table
create table dbo.test(
id int not null primary key,
category varchar(30) not null,
value varchar(50) not null,
name varchar(30))
go

-- Populate this table with data
-- The logic is that when category is 'ID', value is a string of integer ID
insert into dbo.test
values (1, 'Text', 'Hello world', 'foo')
insert into dbo.test
values (2, 'ID', '123', 'bar')
go

-- Create a view that only shows the rows that has category 'ID',
-- and in that case, convert the value to an integer ID
create view dbo.ID_view as
select convert(int, value) ID, name from dbo.test
where category = 'ID'
go

-- this will show the rows where category is 'ID'
select * from dbo.ID_view
go

-- this will cause an error
select * from dbo.ID_view
where ID = 123
go

Selecting all rows from the view works perfectly fine. However, when we try to select from the view where ID is a certain value, 123, we get an runtime error:

Server: Msg 245, Level 16, State 1, Line 1
Syntax error converting the varchar value 'Hello world' to a column of data type int.

The reason is that ID = 123 is expanded into convert(int, value) = 123, and it gets evaluated before the predicate category = 'ID'. Therefore, for the first row, with category as 'Text', and value as 'Hellow world', the conversion from value to int caused the runtime error.

In this particular example, there is no advantage of evaluating which predicate first, so Query Optimizer could choose either order. In other cases, it may be more efficient to evaluate the second predicate first, and Query Optimizer has no idea if the predicate evaluation would cause a runtime error, since it doesn't know the correlation between the two columns.

Note that the error happens on SQL 2000, but the query works fine on SQL 2005 because Query Optimizer happens to choose the other order. However, there is no guarantee that it won't change again in the future.

The correct way to implement such correlation is to use a CASE expression.

create view dbo.ID_view_new as
select case when category = 'ID' then convert(int, value) else NULL end ID, name from dbo.test
where category = 'ID'
go
-- this will works fine
select * from dbo.ID_view_new
where ID = 123
go

In the view definition, even though the filter indicates all rows will have category 'ID', in the select list, we still use the CASE expression to check again and return NULL otherwise. This can help prevent the error even with predicate reordering.

In a word, the order of evaluation for predicates is never guaranteed, so application logic should not depend on such order. If a predicate depends on other predicates and may cause runtime error if other conditions are not met, application should use CASE expression to make it work in any usage.

An article on FOR XML in Microsoft SQL Server 2000 and 2005

ekogan — Fri, 14 Apr 2006 01:04:00 GMT

I just posted an article What does server side FOR XML return? (http://blogs.msdn.com/sqlprogrammability/articles/576095.aspx) which gives some details of the design and performance characteristics of various FOR XML flavors.

I also added links to XML whitepapers and publications on the design of XML features in Microsoft SQL Server 2005 to the links section of the main SQL Programmability & API Development Team Blog page (http://blogs.msdn.com/sqlprogrammability/default.aspx).

Best regards,

Eugene Kogan

Technical Lead

Microsoft SQL Server Relational Engine

Sorting undefined characters in Unicode and/or Windows collation

Jun Fang — Fri, 07 Apr 2006 02:21:00 GMT

When comparing two Unicode strings in non-binary collations, SQL Server uses a library that is essentially same as the Windows API CompareStringW. It defines a weight for each recognized character, and then use the weights to compare characters. However, not all code points are defined in the sorting library. They may be undefined because:

1) The code point is not defined in Unicode standard.
2) The code point is defined in Unicode standard, but not defined by Windows yet. It takes time and effort to define linguistic sorting semantics for new characters. Windows team typically needs to work with local standard body and/or regional PMs to define sorting rules for new characters. They add new character support in every release, and try to catch up. Some characters may have font defined, therefore could be correctly displayed, but still not defined in terms of comparison. For example, NCHAR(13144) - NCHAR(13174).
3) The code point is defined in Windows, but not defined in SQL Server yet.

Windows NLS team has decided that undefined characters are ignored during string comparison, partly because there is no real good way to compare them against other defined characters. SQL Server inherited this semantics. This does cause some confusing behavior. See below examples.

declare @undefined_char1 nvarchar(10), @undefined_char2 nvarchar(10)
set @undefined_char1 = nchar(0x0000)
set @undefined_char2 = nchar(13144)
select 'Undefine characters compare equal to empty string'
where @undefined_char1 = ''
select 'All undefine characters compare equal'
where @undefined_char1 = @undefined_char2
go

create table t (c nvarchar(10))
go
create unique index it on t(c)
go
-- first insert succeeds, but second insert fails with duplicate key error.
insert t values (nchar(0x0000))
insert t values (nchar(13144))
go

As you can see, since all undefined characters compare equal, they could cause duplicate key errors. Similarly, if you create one table with name of an undefined character, and then try to create another table with another undefined character, the second table creation would fail due to duplicate names, even though the code points of the two undefined characters are different. This could also cause confusing results in string matching builtins such as CHARINDEX, PATINDEX, or LIKE.

While these results seem confusing, the basic rule is actually very simple, i.e., undefined characters are ignored during comparison and string matching. Once you understand and remember this rule, the behavior should be easy to understand.

There have been arguments whether undefined characters should be ignored. Since this is the behavior on Windows platform, and there is not a definitely better way to sort them, and for backwards compatibility, we are going to maintain this behavior.

If your app needs to work with these undefined characters and expect to treat them as regular characters, you can use binary collation. In binary collation, comparison is done purely based on code points, not linguistic rules, so there is no notion about defined vs. undefined.

Multiplication and Division with Numerics

mathh — Wed, 29 Mar 2006 20:21:00 GMT

It can be surprising to see certain results when doing numeric arithmetic:

declare @num1 numeric(38,10)
declare @num2 numeric(38,10)
set @num1 = .0000006
set @num2 = 1.0
select cast( @num1 * @num2 as numeric(38,10))

Yields: .0000010000

Instead of: .0000006000

Why?

Well, Books Online (see Precision, Scale, and Length) dictates the following rule for numeric arithmetics:

Operation	Result precision	Result scale *
e1 + e2	max(s1, s2) + max(p1-s1, p2-s2) + 1	max(s1, s2)
e1 - e2	max(s1, s2) + max(p1-s1, p2-s2) + 1	max(s1, s2)
e1 * e2	p1 + p2 + 1	s1 + s2
e1 / e2	p1 - s1 + s2 + max(6, s1 + p2 + 1)	max(6, s1 + p2 + 1)
e1 { UNION \| EXCEPT \| INTERSECT } e2	max(s1, s2) + max(p1-s1, p2-s2)	max(s1, s2)

In this case, the multiplication result precision and scale are computed as follows:

Precision = P1 + P2 + 1 = 38 + 38 + 1 = 77

Scale = S1 + S2 = 10 + 10 = 20

So conceptually, the result should be numeric(77, 20), which isn't allowed. This is where the fine prints come in:

* The result precision and scale have an absolute maximum of 38. When a result precision is greater than 38, the corresponding scale is reduced to prevent the integral part of a result from being truncated.

Books Online currently doesn't go into the specifics on how such truncation is performed.

Since the precision is higher than 38, we try to avoid truncating the integral part of the value by reducing the scale (thus truncating the decimal part of the value instead). How much scale should be sacrificed? There is no right answer. If we preserve too much, and the result of the multiplication of large numbers will be way off. If we preserve too little, multiplication of small numbers becomes an issue.

In SQL Server 2005 RTM (and previous versions), we decided preserve a minimum scale of 6 in both multiplication and division. So our numeric(77,20) is truncated as numeric(38,6), and then is then casted as numeric(38,10). However at this point it is too late, and some data has been lost. This explains the result you would see in the previous expression.

Because of this, it is important to try to minimally quantify the precision and scale of the operands involved in multiplication and division. In this case:

declare @num1 numeric(18,10)
declare @num2 numeric(18,10)
set @num1 = .0000006
set @num2 = 1.0
select cast( @num1 * @num2 as numeric(38,10))

The resulting type would be numeric(37,20). Since this type's precision and scale does not exceed our current limits, no implicit truncation is performed. We then cast the result as numeric(38,10), which doesn't cause data loss in our case.

If you cannot accurately type the values involved in multiplication and division, for example if they are procedure parameters that are called with wildly different values, you may want to look into approximate numeric types (float, real), or perhaps define your own very high capacity user defined exact numeric data type using the CLR.

I hope this was useful. Let us know if there are particular topics you'd like to hear about!

-Mat