Isaac @ MSDN : sql server

Where are my Spatial Columns?

isaac — Tue, 15 Apr 2008 23:48:00 GMT

Hi Folks,

I've been asked a few times how to find out what spatial columns are defined in a database. We don't have any special table for this, but you can easily find out by looking at the usual system views:

SELECT ta.name as table_name, co.name as column_name
FROM sys.tables ta JOIN sys.columns co
      ON ta.object_id = co.object_id
JOIN sys.types ty
      ON co.user_type_id = ty.user_type_id
WHERE ty.name = 'geography' OR ty.name = 'geometry'

There's nothing special about spatial here: you can replace the type names in the WHERE clause of the query with any other type you'd like to find as well. For example, a simple change finds all integer columns:

SELECT ta.name as table_name, co.name as column_name
FROM sys.tables ta JOIN sys.columns co
      ON ta.object_id = co.object_id
JOIN sys.types ty
      ON co.user_type_id = ty.user_type_id
WHERE ty.name = 'int'

Cheers,
-Isaac

[16 April 2008]: Updated to correct a typo in the first query.

Connect and Implicit Casts

isaac — Thu, 10 Apr 2008 19:57:00 GMT

Hi Folks,

Perhaps it is surprising, based on the content here, that I'm not a full-time spatial head here in SQL Server land. (We leave that job to Ed.) I have, of course, been spending a lot of time on spatial, but it turns out I have other responsibilities as well, particularly around the broader type system. One of my jobs is to take a look at issues that come in through Connect that have to do with the type system and make sure they get handled appropriately.

(If you don't use Connect, please do! Issues filed there funnel directly into our bug-tracking system, giving you a direct line to the engineering team. It is the place to go with that pesky bug or feature request.)

These Connect issues are an interesting mix. some good, solid bugs, and some not bugs at all. It's these latter ones I find most interesting, since the majority of them point out quirks in our behavior that, while correct (and often even documented) are still somewhat confusing. I'm going to try to start writing a bit more about common issues here in the hopes of clearing at least a little of that confusion.

Which brings us to implicit casts. An implicit cast is a cast that happens without any explicit direction from the user. If you use CAST or CONVERT, you're using an explicit cast, and that's not what we're talking about here. We're talking about code like this:

SELECT '1'
UNION
SELECT 37

Since the UNION operator has to produce a consistent result, the types of each of its inputs must be the same. Of course, that's not the case here: the first SELECT yields a varchar, the second an int. Faced with this, the system inserts an implicit conversion, attempting to coerce the varchar to an int.

But why convert the varchar to an int and not the other way around? After all, it could convert 37 to '37' instead. First, we have to understand that while this case is simple---the values are literals---this need not be the case. and so it may be very hard for the system to determine up-front what the actual values are. E.g.:

SELECT T.a -- a varchar column
UNION
SELECT U.b -- an int column

Or worse:

SELECT M.a FROM -- some horribly complex query returning a varchar
UNION
SELECT N.b FROM -- another equally horrible query returning an int

In order to base this decision on the actual data, SQL Server would have to accumulate each side of the UNION, inspect all of the values, and hope to find a type that each of the values could be coerced to. Perhaps worse, if the type depended on the actual data retrieved, then it could be very hard to determine what the result type of the query would be; it could even change if the data were updated.

So SQL Server doesn't do this. Instead of basing the conversion on the data itself, it bases it on the type of the data. So, when the server looks at any of these examples, all it really sees is:

SELECT <varchar>
UNION
SELECT <int>

The server then decides which one to convert, in this case it converts the varchar to an int. How does it make this choice? It simply has a precedence list of types, which we can see in the Data Type Precedence topic of Books Online. The abbreviated version is:

1. user-defined data types (highest)
2. sql_variant
3. xml
...
16. int
...
27. varchar
...

Int lands higher than varchar, and so the varchar is converted to an int. The actual data returned by each select is not used at all in making this decision, only the types. The system can make this decision with relatively simple logic, and the type of the result is consistent as well.

This brings us to our pop-quiz. What will this do?

SELECT 'a'
UNION
SELECT 37

I'll give you a minute...

Got it?

Okay, here's the result:

Msg 245, Level 16, State 1, Line 1
Conversion failed when converting the varchar value 'a' to data type int.

Make sense?

Cheers,
-Isaac

More on the Multi-Level Grid

isaac — Tue, 08 Apr 2008 19:32:00 GMT

Hi Folks,

In my last indexing post, I filled in most of the details about our multi-level grid index. Let me clean up a few lingering questions about our planar grid. We'll do this Q&A style:

What happens if I set the maximum number of cells-per-object so small that I cannot obtain a covering of my object?

To make this concrete, let's take a look at one of the pictures from last time:

In this schematic, there are four top-level cells, and we can see that we need at least three cells to tile the object. In this picture, we are using 13 cells, so we've set the maximum number of cells-per-object to at least that. But what if we set it to two?

The answer is that in the case where the number of first-level cells needed to tile the object exceeds the max cells-per-object, we violate cells-per-object constraint. This is really the only situation in which we'll do that, but it's necessary in order to ensure that we tile the object, and this is required in order to guarantee correct query results.

Keep in mind that in a real index, the number of top level cells is either 16, 64, or 256, and this will only happen if the max cells-per-object exceeds that value.

What if I have an object that falls outside, or partially outside the bounding box I have defined? Will I miss results?

There is an implicit extra cell in the tessellation, what we call the root cell. The root cell covers everything outside of the bounding box you've defined, and will be used to cover any object that touches it. We can use this to guarantee correct results even if your object isn't well contained by your bounding box. What will suffer is performance, and so it is still important to get the bounding box correct.

Are there times where you can answer the query using solely the index?

Yes. Consider a cell that is touched by one object, and completely covered by another. We can tell that these two objects must intersect based solely on this fact: there is no need to actually run the intersection predicate. We call this "intermediate filtering" (not the best name, but we needed something to call it) and we do implement it. To do this, we store whether the object touches or covers the cell in the index row.

Which predicates are supported on an index?

It depends on the type. For geometry, we support STIntersects, STEquals, STOverlaps, STTouches, STWithin, STContains, STDistance, and STFilter. On geography we only support STIntersects, STEquals, STDistance, and Filter. We don't support the others simply because they don't exist on the type.

What is the difference between STIntersects and Filter?

If there isn't an index, they do exactly the same thing. This can happen on the client, or on the server if you haven't defined an index (or if the index isn't chosen for some reason). When there is an index involved, Filter will not do any secondary filtering. I.e., it is only guaranteed to return a superset of the results an STIntersects predicate will return: it may return extras.

This is useful for applications that are simply displaying results, are not sensitive to some extras, and want to avoid the cost of the secondary filter. If you need precise results, use STIntersects.

If there are more questions, please ask, and I'll answer them here. So far we've only talked about geometry, so next time I'll either fill in the gaps around geography or answer more questions.

Cheers,
-Isaac

The Unexpected Too-Large Polygon

isaac — Sun, 06 Apr 2008 03:25:00 GMT

Hi Folks,

I recently got contacted via email with the following problem:

...

I have 2 complex polygons, representing district boundaries. The polygons look correct, but I'm getting exceptions when I try to create the type.

I've attached the wkt polygons to the email.

These polygons are stored in a table, but I've tried it without the table and get the same result.

Here's the sql I'm trying to execute:

declare @g geography;
set @g = geography::STPolyFromText(@wkt, 4268)
select @g.ToString()

where @wkt is replaced by the correct polygon string.

...

Now, I should point out that this is a very nice error report. They've told me what their data is, they included their data, and they told me very clearly what they're trying to do. (Perhaps knowing what version they're using would good, but I'm getting picky.)

But here's the kicker that saved me any debugging:

...
Here is the error message I receive:

A .NET Framework error occurred during execution of user defined routine or aggregate 'geography':
Microsoft.SqlServer.Types.GLArgumentException: 24205: The specified input does not represent a valid geography instance because it exceeds a single hemisphere. Each geography instance must fit inside a single hemisphere. A common reason for this error is that a polygon has the wrong ring orientation.
Microsoft.SqlServer.Types.GLArgumentException:
   at Microsoft.SqlServer.Types.GLNativeMethods.ThrowExceptionForHr(GL_HResult errorCode)
   at Microsoft.SqlServer.Types.GLNativeMethods.GeodeticIsValid(GeometryData g)
   at Microsoft.SqlServer.Types.SqlGeography.IsValidExpensive()
   at Microsoft.SqlServer.Types.SqlGeography.ConstructGeographyFromUserInput(GeometryData g, Int32 srid)
   at Microsoft.SqlServer.Types.SqlGeography.STPolyFromText(SqlChars polygonTaggedText, Int32 srid)

I looked at all the nodes, and they are all in the same hemisphere, so I'm a little confused with what might be going on.
...

So, what's going on? Ring ordering.

I've posted on this before, but let's recap. On a round Earth, defining a polygon by a ring is ambiguous. Consider the polygon defined by a ring around the equator: does this describe the northern or southern hemisphere? A small ring could describe either, say, a small island in a large ocean, or the large ocean itself.

To get out of this pickle, we disambiguate with a standard trick: we take ring orientation into account. We simply define the left side of the ring as it is drawn to be inside the polygon.

If we put this together with our current (unfortunate) limitation that we do not deal with objects that exceed a hemisphere, and it becomes clear what's going on. While the polygon looks like it's nice and small, the rings are inverted, and so it's actually quite large. We can't handle it: cue the exception.

I describe this hemisphere limitation as unfortunate, and it remains one of the biggest items the spatial team would like to fix up, but in this case it has actually helped catch an error.

I'll talk a bit more about our hemisphere limitation in a future post. It's not quite as simple as we let on.

Cheers,
-Isaac

February CTP Optimizer Issue

isaac — Tue, 11 Mar 2008 06:42:00 GMT

Hi Folks,

We don't call them betas, but that's basically what CTPs are: they let us suss out problems before we drop a final product on everyone. Well, we've found a regression in the February CTP spatial support that we'd like to let you know about.

Essentially, a costing problem was introduced that leads us to choose poor plans for spatial queries. This seems to usually manifest itself by choosing merge joins instead of loop joins between the spatial index and the base table, and that generally doesn't work too well.

Why is a merge join such a bad plan in this case? If you've been following my indexing posts, you'll understand that we generally end up with a small number of rows that pass our primary filter (our index). We have to then join those back to the base table to pull out the spatial object to which the index row refers.

This is usually best done through a loop join that runs over each index row and seeks into the base table for each of the objects. However, if a merge join is chosen, then we risk scanning the entire base table to feed into our merge. This is usually not a wise choice.

What can you do about this if you're running the February CTP? A good general fix is to force loop joins in your query. Perhaps the simplest way to do this is by adding an "OPTION (LOOP JOIN)" to the end of your query.

On the positive side, this has already been fixed, so you should see better plans in our next public release.

Cheers,

-Isaac

The Upcoming Geography Coordinate Order Swap---A FAQ

isaac — Wed, 05 Mar 2008 20:24:00 GMT

Hi Folks,

I just thought I'd take a few moments clarify the upcoming coordinate order swap for the geography type. Here's a quick FAQ on the issue:

What exactly is the change?

We are swapping the coordinate order for well-known text (WKT) and well-known binary (WKB) formats from latitude-longitude to longitude-latitude. E.g., a point in the Seattle area might be represented in WKT as "POINT(44 -122)" in the current ordering; in the new order it will be "POINT(-122 44)".
Why are we making this change?

Because of customer feedback. While common practice almost universally uses latitude-longitude ordering, the de facto standard for WKT and WKB is to use a longitude-latitude order. Making this change will make it much easier for customers to load WKT and WKB data into the geography type.
What about GML?

The standard paractice for GML is to use latitude-longitude ordering, and so no change will be made for GML.
Is the on-disk format changing?

Nope. Ths is just a matter of changing our routines that import and export data. Nothing beyond those routines will change.
What about geometry?

There is no change to the geometry type. Geometry deals with objects on the plane, and is pretty agnostic about the coordinate order outsidfe of the STX and STY properties.
When is the swap happening?

It is not the current Februrary CTP, but the team is working on it now. Expect to see it in our next public release.

If there are more questions, I'll be happy to answer them here.

Cheers,
-Isaac

Minor Update: I originally got the WKT examples at the top wrong, placing commas between coordinates. The examples have been updated to their correct, comma-free state. (Thanks, Steven!)

Basic Multi-Level Grids

isaac — Sat, 01 Mar 2008 05:17:00 GMT

Hi Folks,

Last time, we highlighted several problems with a simple grid index. If you don't recall---and since it's been a while, that wouldn't be a surprise---you may want to review them. In this post I'll start to describe how we get around them.

In SQL Server, we don't use a simple grid like the one described. Instead, we use a multi-level grid, which ends up looking very much like a quad tree. Basically, instead of having one grid level, we have four, with each lower-level grid nested in the one above.

Let's look at an example. If we have a 2x2 grid at each level, then we can tile the object below with three tiles at the top-level grid:

These tiles cover the object, which is a property we want to maintain for our indexing scheme, but there's a lot of extra coverage. This extra means we're likely to have a good number of false positives from our primary filter. We can go to a second-level gridding instead:

At this level, we've trimmed some of the extra slop from the edges, but our description of the object has become more complex as well. With our four levels, we can push this even further---here's what a fourth-level tiling of the object would look like:

This description is nice and tight---there's very extra covering of the object---but it's also very complicated. What's key to keeping the complexity down is that we don't require that all object use the same level tiling, nor do we require that all tiles for a particular object all be at the same level. We can mix as we wish.

The ability to mix levels in our tiling leads us to two optimizations. The first, which is always used, is that if we find that all subtiles of a particular tile are touched by the object, then we won't further subdivide the cell. Subdividing gives us nothing but more cells. Perhaps this is clearer visually; below is a picture of the same tiling above with this optimization applied:

As you can see, this greatly reduces the number of tiles we have, and does so by mixing levels. A second way to reduce tiles is through an explicit limit. We default this limit to 16, but the user can tweak it to their liking. When we decompose an object, we do a breath-first walk down the tree decomposing tiles, and we stop when we hit the pre-defined limit. E.g., we might end up with:

There's much more to say about this, but before I leave it for now, let me point out that while these examples use 2x2 decompositions at each level, we use more. The exact amount is, in fact, adjustable: the user can set each level to low (4x4) medium (8x8) or high (16x16).

How should the user tweak these various parameters? It's hard to give general advice, as it's a very data- and query-dependent calculation, but we feel that we have picked some defaults that will work well for most people. We may have better advice as more people start using the index and we get reports on what worked for them. (Yes, we're begging you to send us your findings!)

Next time we'll continue with some more details about the mult-level grid.

Cheers,
-Isaac

One Type, Two Types...

isaac — Tue, 29 May 2007 17:15:00 GMT

This is a little bit less of an introductory post than the last one, but there was quite a bit of discussion about our decision to split our spatial types in two---one for our planar ("flat-Earth") model and one for our ellipsoidal ("round-Earth") model---so I thought I'd address it. To be honest, this is something that we struggled with, but in the end we favored the two-type model. Here's a quick look at of some of our reasoning.

Let me begin with a roughly analogous situation: SQL Server's floating- and fixed-point types, float and decimal. These types really have the same interface, so we could have conceivably had one type with a floating-point/fixed-point option. Why choose two types over one?

One reason is that mingling the two would be confusing, since their semantics differ. If we didn't separate the types, then we could write a method that took that single numeric type as input. If we performed any operations that depended on the difference in semantics, then we would have to be careful to check which one we were dealing with and proceed accordingly. We could skip these checks if we always expected, say, fixed-point, but we could be almost certain that if our code survived long enough, some schmuck would eventually hand us a floating-point value.

By separating the types, there is a clear division, and given an instance of a particular type we know exactly what we're dealing with. Additionally, we know that nobody could hand us the wrong type later on.

So by separating the types we've made things more explicit and clear, but one can argue that since the semantics are often the same---or at least close enough that we don't care---we could save code by unifying the types. This is undoubtedly true: there's a tradeoff to be made.

We face a similar problem with spatial types. The two types often behave quite similarly, but there are some key differences. The tradeoff is the same: you can save code by unifying the types at the expense of clarity and robustness.

So, what's different between the planar and ellipsoidal types? Well, beyond the obvious---one is flat, one is round---here are some examples:

In the planar system, distances and areas are given in the same unit of measure as coordinates. E.g., the distance between (2, 2) and (5, 6) is 5 units, regardless of what units are. In a geodetic system, coordinates are given in degrees, but it hardly makes sense to give lengths and areas in degrees or square degrees---we'd much prefer something like meters.

In the planar system, we don't care about the orientation of a polygon. E.g., a polygon described by ((0, 0), (10, 0), (0, 20), (0, 0)) is the same as one described by ((0, 0), (0, 20), (10, 0), (0, 0)). The OGC Simple Features for SQL specification doesn't dictate a ring ordering, so we don't enforce one. In a geodetic system, a polygon is ambiguous without an orientation. E.g., which hemisphere would a ring around the equator describe?

In planar coordinates, it makes sense to use a bounding box as a cheap substitute for an object. In geodetic coordinates, a bounding circle is more natural.

OGC talks about outer rings and inner rings, but this distinction makes little sense for a geodetic type: any ring of a polygon can be taken to be the outer one.

Of course, this is only a start. In addition, we are aiming to simplify things a little bit in our round-Earth type. We thought that given all of this, merging the two types would be confusing, especially for the non-experts out there, so we decided to separate things into two types.

Is that the right decision? We think so, but we recognize that not everyone may agree with us. We look forward to hearing from you.

Cheers,

-Isaac

SQL Server Spatial Support: An Introduction

isaac — Thu, 17 May 2007 01:28:00 GMT

I’ve been pretty quiet around here, and at least part of the reason is that we’ve been pretty tight-lipped about what we’ve been up to. We’ve now gone public—we’re providing support for geospatial data in our next version of SQL Server, codenamed Katmai. The Virtual Earth folks mentioned this in their blog (others, too) but now I can make a little more noise.

What I’d like to do first is try to explain to the folks out there who aren’t experts in the field what this all means and give a very high-level tour of what we’re providing.

First, what is geospatial data? For starters, we mean locations on the Earth. It’s more than location, though: we mean the location and shape of objects on the Earth. Think of the description of roads, states, lakes, etc. For example, think of something like Live Maps:

In addition to the normal objects, I’ve added a polygon that shows Microsoft’s main campus. We want to be able to store all of the data on this map—the roads, the parks, and user-generated polygon—in SQL Server. Being a database, we want to be able to ask questions about the data. For example, “What are the roads that intersect Microsoft’s main campus?” A more complex example would be “What is the area of all parks within 1 kilometer of Microsoft’s main campus?”

How do we do this? First, we need new data types to be able to store this information. Toward this end, we are introducing two new types in Katmai:

· The first type, “geography”, will store points, lines, polygons, and collections of these in latitude/longitude coordinates using a round-Earth model. Most commonly-available data is given in latitude/longitude coordinates, so we expect that most people will want to use this type. Furthermore, this type will give correct computations on a true ellipsoidal model of the planet. What is the area of Indonesia? Will my flight from Seattle to Beijing take me over North Korea? Where can I store my GPS readings? This is the type for you.

· A “geometry” type to support flat-earth data. For those of you who are familiar with such things, this type is our OGC-compliant offering. In some ways, this is a more specialty offering for people who need to work in projected map coordinates either for legacy or legal reasons, but this type can be used for things like interior spaces as well, e.g., “Where in this warehouse is my book?”

If we take our roads data above—perhaps all of the roads for the United States—we could store them in a table Roads:

Roads(name varchar(30), location geography)

I.e., geography is a column type just like any other. We expose a pretty comprehensive set of operations on these type through a method-based interface. For example, if we have a geometry variable @microsoft that represents Microsoft’s main campus, we can find out which roads intersect it with the query:

SELECT name
FROM Roads
WHERE location.STIntersects(@microsoft) = 1

Given a similar Parks table containing all US parks, we can ask our parks question from above:

SELECT SUM(location.STArea())
FROM Parks
WHERE location.STDistance(@microsoft) < 1.0

Of course, beyond answering this query, we need to be able to answer the query quickly. Since my data may be very large, speed is going to mean having a good spatial index. Perhaps that will be the subject of my next post. Watch this space for more.

Cheers,
-Isaac