Isaac @ MSDN : spatial

Eye Candy of the Highest Order

isaac — Mon, 02 Jun 2008 12:01:00 GMT

Hi Folks,

One of the most common questions about the spatial support in SQL Server is how to visualize results. To illustrate the problem, consider a query over zip code data:

select id, shapegeog
from zipcodes
where id like '981%'

The results, while an accurate description of the data, are profoundly unhelpful:

We can improve the human readability of this by fetching WKT instead of the original geography data:

While we can now read it, it still doesn't mean much. Unlike social security numbers or names, visualization is almost required if you want to understand spatial data. And if you really want to visualize the data, our only avenue has been to make use of an external program, such as Virtual Earth. It's not super hard to make a mashup with VE, but it is another step.

Thanks to some very hard work of a number of people, a very late feature has been slipped into the server: a map control. With the map control, if you run the query, you get the following:

And the magic reveals itself when you click that mysterious "Spatial result" tab:

These maps are interactive: you can zoom; you can pan; you can even label the results. It isn't aimed at reporting, and this isn't a reusable component that can be used in other applications, but it can be invaluable in giving you a feel for what your results look like. It goes far beyond eye candy: it actually allows you to understand your data.

Unfortunately, it's a bit of a tease at the moment: the code just went in, which means none of this will be available in our upcoming RC0. It will be in RTM, however, so you shouldn't have to wait too long.

I can claim precious little credit for this, but if you're at TechEd this week, I'd be happy to demo it for you. So would Goldie Chaudhuri, who actually led the effort.

Cheers,
-Isaac

Our Upcoming Builder API

isaac — Fri, 30 May 2008 18:42:00 GMT

Hi Folks,

We have one more upcoming pre-release before we're done with SQL Server 2008, and while I've posted a few articles about the coordinate order swap, there's another exciting piece we're releasing: our builder API.

Let's say you wanted to write a very simple method that shifted a geometry instance in space. You could write this this today, but it would require writing code to digest one of the formats we support---WKT, WKB, or GML---perform the operation, and then recreate one of these data formats. When the operation is so simple, the bulk of the effort needed will be in moving between formats.

This seems pretty silly, given the fact that we certainly have code that already digests these instances programmatically. Why don't we give you access?

Now we do.

If you scroll down, you'll see a lot of C# code. The solution we have can only be accessed directly through the CLR, but since the CLR is hosted in SQL Server, functionality developed can be deployed to the server and used through T-SQL. This new functionality, which will be released in RC0, has three key parts:

IGeometrySink / IGeographySink interfaces. These interfaces expose six methods---SetSRID, BeginGeometry, BeginFigure, AddLine, EndFigure, and EndGeometry---that can be called to describe a spatial instance with full fidelity. Anyone can build a class that implements these methods, and pass an instance of their class to the...
...Populate() method on geography and geometry instances. This method takes an IGeometrySink or IGeographySink instance and makes the appropriate calls to exactly describe the instance. If, e.g., you wanted to build code to create KML from geography instances, you'd build a KML translator that implemented IGeographySink, and call Populate() with it. If you wanted to write the shift example, you'd still have the problem of creating the result instance, and this is handled by...
...Builder classes for geometry and geography. SqlGeometryBuilder and SqlGeographyBuilder implement their respective sink interfaces, and expose an additional property: ConstructedGeometry. After a series of calls to the interface that create a complete instance, a geometry or geography instance will be available in the property. Because the interfaces are the same for both digesting and creating spatial instances, it is very easy to pipeline input and output operations.

So, let's take a look at a working example that solves our shifting problem. The main part is to write our own implementation of IGeometrySink that will perform the shifting. Our constructor will take the amount of shift we want in the x and y directions, as well as a target IGeometrySink instance. The implementations of each of the IGeometrySink methods will simply pass the calls through to the target, except that every point passed through will be shifted first.

using Microsoft.SqlServer.Types;

namespace Tools
{
    class GeometryShifter : IGeometrySink
    {
        IGeometrySink _target;
        double _xShift;
        double _yShift;

        public GeometryShifter(double xShift, double yShift, IGeometrySink target)
        {
            _target = target;
            _xShift = xShift;
            _yShift = yShift;
        }

        // Each AddLine call will just move the endpoint by the required amount.
        public void AddLine(double x, double y, double? z, double? m)
        {
            _target.AddLine(x + _xShift, y + _yShift, z, m);
        }

        // Each BeginFigure call will just move the start point by the required amount.
        public void BeginFigure(double x, double y, double? z, double? m)
        {
            _target.BeginFigure(x + _xShift, y + _yShift, z, m);
        }

        // Just pass through without change.
        public void BeginGeometry(OpenGisGeometryType type)
        {
            _target.BeginGeometry(type);
        }

        // Just pass through without change.
        public void EndFigure()
        {
            _target.EndFigure();
        }

        // Just pass through without change.
        public void EndGeometry()
        {
            _target.EndGeometry();
        }

        // Just pass through without change.
        public void SetSrid(int srid)
        {
            _target.SetSrid(srid);
        }
    }
}

To use this in SQL Server, we need to wrap it in a function that can be registered and used in the server. To do this, we create a function that puts the pipeline together and runs it.

using Microsoft.SqlServer.Types;

namespace Tools
{
    public partial class Functions
    {
        public static SqlGeometry ShiftGeometry(SqlGeometry g, double xShift, double yShift)
        {
            SqlGeometryBuilder constructed = new SqlGeometryBuilder();
            GeometryShifter shifter = new GeometryShifter(xShift, yShift, constructed);
            g.Populate(shifter);
            return constructed.ConstructedGeometry;
        }
    }
}

Because of SQL Server's CLR support, we can build this to Tools.dll and load it into the server. After that, we can register the function and use it directly from T-SQL:

create assembly Tools from 'c:\tmp\Tools.dll'
go

create function ShiftGeometry(@g geometry, @x float, @y float)
returns geometry
as external name Tools.[Tools.Functions].ShiftGeometry
go

declare @g geometry = 'LINESTRING (0 0, 1 1, 12 3, 4 0)'

select @g.ToString()
union all
select dbo.ShiftGeometry(@g, 10, 10).ToString()
go

This yields:

LINESTRING (0 0, 1 1, 12 3, 4 0)
LINESTRING (10 10, 11 11, 22 13, 14 10)

This is a pretty simple example, but it illustrates the power of the approach. Now you can build functions like this without all of the parsing glue.

Our spatial implementation makes use of this kind of API internally, and it was this internal use that inspired us to release it to users of the system as well. We see many cases in which new spatial operations can be added to the server through this API, and we hope that you can find some as well. (Looking at the Spatial forum, we can find many examples of problems that can be tackled with this approach, and this list is nowhere near exhaustive.)

Cheers,
-Isaac

Where are my Spatial Columns?

isaac — Tue, 15 Apr 2008 23:48:00 GMT

Hi Folks,

I've been asked a few times how to find out what spatial columns are defined in a database. We don't have any special table for this, but you can easily find out by looking at the usual system views:

SELECT ta.name as table_name, co.name as column_name
FROM sys.tables ta JOIN sys.columns co
      ON ta.object_id = co.object_id
JOIN sys.types ty
      ON co.user_type_id = ty.user_type_id
WHERE ty.name = 'geography' OR ty.name = 'geometry'

There's nothing special about spatial here: you can replace the type names in the WHERE clause of the query with any other type you'd like to find as well. For example, a simple change finds all integer columns:

SELECT ta.name as table_name, co.name as column_name
FROM sys.tables ta JOIN sys.columns co
      ON ta.object_id = co.object_id
JOIN sys.types ty
      ON co.user_type_id = ty.user_type_id
WHERE ty.name = 'int'

Cheers,
-Isaac

[16 April 2008]: Updated to correct a typo in the first query.

More on the Multi-Level Grid

isaac — Tue, 08 Apr 2008 19:32:00 GMT

Hi Folks,

In my last indexing post, I filled in most of the details about our multi-level grid index. Let me clean up a few lingering questions about our planar grid. We'll do this Q&A style:

What happens if I set the maximum number of cells-per-object so small that I cannot obtain a covering of my object?

To make this concrete, let's take a look at one of the pictures from last time:

In this schematic, there are four top-level cells, and we can see that we need at least three cells to tile the object. In this picture, we are using 13 cells, so we've set the maximum number of cells-per-object to at least that. But what if we set it to two?

The answer is that in the case where the number of first-level cells needed to tile the object exceeds the max cells-per-object, we violate cells-per-object constraint. This is really the only situation in which we'll do that, but it's necessary in order to ensure that we tile the object, and this is required in order to guarantee correct query results.

Keep in mind that in a real index, the number of top level cells is either 16, 64, or 256, and this will only happen if the max cells-per-object exceeds that value.

What if I have an object that falls outside, or partially outside the bounding box I have defined? Will I miss results?

There is an implicit extra cell in the tessellation, what we call the root cell. The root cell covers everything outside of the bounding box you've defined, and will be used to cover any object that touches it. We can use this to guarantee correct results even if your object isn't well contained by your bounding box. What will suffer is performance, and so it is still important to get the bounding box correct.

Are there times where you can answer the query using solely the index?

Yes. Consider a cell that is touched by one object, and completely covered by another. We can tell that these two objects must intersect based solely on this fact: there is no need to actually run the intersection predicate. We call this "intermediate filtering" (not the best name, but we needed something to call it) and we do implement it. To do this, we store whether the object touches or covers the cell in the index row.

Which predicates are supported on an index?

It depends on the type. For geometry, we support STIntersects, STEquals, STOverlaps, STTouches, STWithin, STContains, STDistance, and STFilter. On geography we only support STIntersects, STEquals, STDistance, and Filter. We don't support the others simply because they don't exist on the type.

What is the difference between STIntersects and Filter?

If there isn't an index, they do exactly the same thing. This can happen on the client, or on the server if you haven't defined an index (or if the index isn't chosen for some reason). When there is an index involved, Filter will not do any secondary filtering. I.e., it is only guaranteed to return a superset of the results an STIntersects predicate will return: it may return extras.

This is useful for applications that are simply displaying results, are not sensitive to some extras, and want to avoid the cost of the secondary filter. If you need precise results, use STIntersects.

If there are more questions, please ask, and I'll answer them here. So far we've only talked about geometry, so next time I'll either fill in the gaps around geography or answer more questions.

Cheers,
-Isaac

The Unexpected Too-Large Polygon

isaac — Sun, 06 Apr 2008 03:25:00 GMT

Hi Folks,

I recently got contacted via email with the following problem:

...

I have 2 complex polygons, representing district boundaries. The polygons look correct, but I'm getting exceptions when I try to create the type.

I've attached the wkt polygons to the email.

These polygons are stored in a table, but I've tried it without the table and get the same result.

Here's the sql I'm trying to execute:

declare @g geography;
set @g = geography::STPolyFromText(@wkt, 4268)
select @g.ToString()

where @wkt is replaced by the correct polygon string.

...

Now, I should point out that this is a very nice error report. They've told me what their data is, they included their data, and they told me very clearly what they're trying to do. (Perhaps knowing what version they're using would good, but I'm getting picky.)

But here's the kicker that saved me any debugging:

...
Here is the error message I receive:

A .NET Framework error occurred during execution of user defined routine or aggregate 'geography':
Microsoft.SqlServer.Types.GLArgumentException: 24205: The specified input does not represent a valid geography instance because it exceeds a single hemisphere. Each geography instance must fit inside a single hemisphere. A common reason for this error is that a polygon has the wrong ring orientation.
Microsoft.SqlServer.Types.GLArgumentException:
   at Microsoft.SqlServer.Types.GLNativeMethods.ThrowExceptionForHr(GL_HResult errorCode)
   at Microsoft.SqlServer.Types.GLNativeMethods.GeodeticIsValid(GeometryData g)
   at Microsoft.SqlServer.Types.SqlGeography.IsValidExpensive()
   at Microsoft.SqlServer.Types.SqlGeography.ConstructGeographyFromUserInput(GeometryData g, Int32 srid)
   at Microsoft.SqlServer.Types.SqlGeography.STPolyFromText(SqlChars polygonTaggedText, Int32 srid)

I looked at all the nodes, and they are all in the same hemisphere, so I'm a little confused with what might be going on.
...

So, what's going on? Ring ordering.

I've posted on this before, but let's recap. On a round Earth, defining a polygon by a ring is ambiguous. Consider the polygon defined by a ring around the equator: does this describe the northern or southern hemisphere? A small ring could describe either, say, a small island in a large ocean, or the large ocean itself.

To get out of this pickle, we disambiguate with a standard trick: we take ring orientation into account. We simply define the left side of the ring as it is drawn to be inside the polygon.

If we put this together with our current (unfortunate) limitation that we do not deal with objects that exceed a hemisphere, and it becomes clear what's going on. While the polygon looks like it's nice and small, the rings are inverted, and so it's actually quite large. We can't handle it: cue the exception.

I describe this hemisphere limitation as unfortunate, and it remains one of the biggest items the spatial team would like to fix up, but in this case it has actually helped catch an error.

I'll talk a bit more about our hemisphere limitation in a future post. It's not quite as simple as we let on.

Cheers,
-Isaac

February CTP Optimizer Issue

isaac — Tue, 11 Mar 2008 06:42:00 GMT

Hi Folks,

We don't call them betas, but that's basically what CTPs are: they let us suss out problems before we drop a final product on everyone. Well, we've found a regression in the February CTP spatial support that we'd like to let you know about.

Essentially, a costing problem was introduced that leads us to choose poor plans for spatial queries. This seems to usually manifest itself by choosing merge joins instead of loop joins between the spatial index and the base table, and that generally doesn't work too well.

Why is a merge join such a bad plan in this case? If you've been following my indexing posts, you'll understand that we generally end up with a small number of rows that pass our primary filter (our index). We have to then join those back to the base table to pull out the spatial object to which the index row refers.

This is usually best done through a loop join that runs over each index row and seeks into the base table for each of the objects. However, if a merge join is chosen, then we risk scanning the entire base table to feed into our merge. This is usually not a wise choice.

What can you do about this if you're running the February CTP? A good general fix is to force loop joins in your query. Perhaps the simplest way to do this is by adding an "OPTION (LOOP JOIN)" to the end of your query.

On the positive side, this has already been fixed, so you should see better plans in our next public release.

Cheers,

-Isaac

The Upcoming Geography Coordinate Order Swap---A FAQ

isaac — Wed, 05 Mar 2008 20:24:00 GMT

Hi Folks,

I just thought I'd take a few moments clarify the upcoming coordinate order swap for the geography type. Here's a quick FAQ on the issue:

What exactly is the change?

We are swapping the coordinate order for well-known text (WKT) and well-known binary (WKB) formats from latitude-longitude to longitude-latitude. E.g., a point in the Seattle area might be represented in WKT as "POINT(44 -122)" in the current ordering; in the new order it will be "POINT(-122 44)".
Why are we making this change?

Because of customer feedback. While common practice almost universally uses latitude-longitude ordering, the de facto standard for WKT and WKB is to use a longitude-latitude order. Making this change will make it much easier for customers to load WKT and WKB data into the geography type.
What about GML?

The standard paractice for GML is to use latitude-longitude ordering, and so no change will be made for GML.
Is the on-disk format changing?

Nope. Ths is just a matter of changing our routines that import and export data. Nothing beyond those routines will change.
What about geometry?

There is no change to the geometry type. Geometry deals with objects on the plane, and is pretty agnostic about the coordinate order outsidfe of the STX and STY properties.
When is the swap happening?

It is not the current Februrary CTP, but the team is working on it now. Expect to see it in our next public release.

If there are more questions, I'll be happy to answer them here.

Cheers,
-Isaac

Minor Update: I originally got the WKT examples at the top wrong, placing commas between coordinates. The examples have been updated to their correct, comma-free state. (Thanks, Steven!)

Basic Multi-Level Grids

isaac — Sat, 01 Mar 2008 05:17:00 GMT

Hi Folks,

Last time, we highlighted several problems with a simple grid index. If you don't recall---and since it's been a while, that wouldn't be a surprise---you may want to review them. In this post I'll start to describe how we get around them.

In SQL Server, we don't use a simple grid like the one described. Instead, we use a multi-level grid, which ends up looking very much like a quad tree. Basically, instead of having one grid level, we have four, with each lower-level grid nested in the one above.

Let's look at an example. If we have a 2x2 grid at each level, then we can tile the object below with three tiles at the top-level grid:

These tiles cover the object, which is a property we want to maintain for our indexing scheme, but there's a lot of extra coverage. This extra means we're likely to have a good number of false positives from our primary filter. We can go to a second-level gridding instead:

At this level, we've trimmed some of the extra slop from the edges, but our description of the object has become more complex as well. With our four levels, we can push this even further---here's what a fourth-level tiling of the object would look like:

This description is nice and tight---there's very extra covering of the object---but it's also very complicated. What's key to keeping the complexity down is that we don't require that all object use the same level tiling, nor do we require that all tiles for a particular object all be at the same level. We can mix as we wish.

The ability to mix levels in our tiling leads us to two optimizations. The first, which is always used, is that if we find that all subtiles of a particular tile are touched by the object, then we won't further subdivide the cell. Subdividing gives us nothing but more cells. Perhaps this is clearer visually; below is a picture of the same tiling above with this optimization applied:

As you can see, this greatly reduces the number of tiles we have, and does so by mixing levels. A second way to reduce tiles is through an explicit limit. We default this limit to 16, but the user can tweak it to their liking. When we decompose an object, we do a breath-first walk down the tree decomposing tiles, and we stop when we hit the pre-defined limit. E.g., we might end up with:

There's much more to say about this, but before I leave it for now, let me point out that while these examples use 2x2 decompositions at each level, we use more. The exact amount is, in fact, adjustable: the user can set each level to low (4x4) medium (8x8) or high (16x16).

How should the user tweak these various parameters? It's hard to give general advice, as it's a very data- and query-dependent calculation, but we feel that we have picked some defaults that will work well for most people. We may have better advice as more people start using the index and we get reports on what worked for them. (Yes, we're begging you to send us your findings!)

Next time we'll continue with some more details about the mult-level grid.

Cheers,
-Isaac

One Type, Two Types...

isaac — Tue, 29 May 2007 17:15:00 GMT

This is a little bit less of an introductory post than the last one, but there was quite a bit of discussion about our decision to split our spatial types in two---one for our planar ("flat-Earth") model and one for our ellipsoidal ("round-Earth") model---so I thought I'd address it. To be honest, this is something that we struggled with, but in the end we favored the two-type model. Here's a quick look at of some of our reasoning.

Let me begin with a roughly analogous situation: SQL Server's floating- and fixed-point types, float and decimal. These types really have the same interface, so we could have conceivably had one type with a floating-point/fixed-point option. Why choose two types over one?

One reason is that mingling the two would be confusing, since their semantics differ. If we didn't separate the types, then we could write a method that took that single numeric type as input. If we performed any operations that depended on the difference in semantics, then we would have to be careful to check which one we were dealing with and proceed accordingly. We could skip these checks if we always expected, say, fixed-point, but we could be almost certain that if our code survived long enough, some schmuck would eventually hand us a floating-point value.

By separating the types, there is a clear division, and given an instance of a particular type we know exactly what we're dealing with. Additionally, we know that nobody could hand us the wrong type later on.

So by separating the types we've made things more explicit and clear, but one can argue that since the semantics are often the same---or at least close enough that we don't care---we could save code by unifying the types. This is undoubtedly true: there's a tradeoff to be made.

We face a similar problem with spatial types. The two types often behave quite similarly, but there are some key differences. The tradeoff is the same: you can save code by unifying the types at the expense of clarity and robustness.

So, what's different between the planar and ellipsoidal types? Well, beyond the obvious---one is flat, one is round---here are some examples:

In the planar system, distances and areas are given in the same unit of measure as coordinates. E.g., the distance between (2, 2) and (5, 6) is 5 units, regardless of what units are. In a geodetic system, coordinates are given in degrees, but it hardly makes sense to give lengths and areas in degrees or square degrees---we'd much prefer something like meters.

In the planar system, we don't care about the orientation of a polygon. E.g., a polygon described by ((0, 0), (10, 0), (0, 20), (0, 0)) is the same as one described by ((0, 0), (0, 20), (10, 0), (0, 0)). The OGC Simple Features for SQL specification doesn't dictate a ring ordering, so we don't enforce one. In a geodetic system, a polygon is ambiguous without an orientation. E.g., which hemisphere would a ring around the equator describe?

In planar coordinates, it makes sense to use a bounding box as a cheap substitute for an object. In geodetic coordinates, a bounding circle is more natural.

OGC talks about outer rings and inner rings, but this distinction makes little sense for a geodetic type: any ring of a polygon can be taken to be the outer one.

Of course, this is only a start. In addition, we are aiming to simplify things a little bit in our round-Earth type. We thought that given all of this, merging the two types would be confusing, especially for the non-experts out there, so we decided to separate things into two types.

Is that the right decision? We think so, but we recognize that not everyone may agree with us. We look forward to hearing from you.

Cheers,

-Isaac

SQL Server Spatial Support: An Introduction

isaac — Thu, 17 May 2007 01:28:00 GMT

I’ve been pretty quiet around here, and at least part of the reason is that we’ve been pretty tight-lipped about what we’ve been up to. We’ve now gone public—we’re providing support for geospatial data in our next version of SQL Server, codenamed Katmai. The Virtual Earth folks mentioned this in their blog (others, too) but now I can make a little more noise.

What I’d like to do first is try to explain to the folks out there who aren’t experts in the field what this all means and give a very high-level tour of what we’re providing.

First, what is geospatial data? For starters, we mean locations on the Earth. It’s more than location, though: we mean the location and shape of objects on the Earth. Think of the description of roads, states, lakes, etc. For example, think of something like Live Maps:

In addition to the normal objects, I’ve added a polygon that shows Microsoft’s main campus. We want to be able to store all of the data on this map—the roads, the parks, and user-generated polygon—in SQL Server. Being a database, we want to be able to ask questions about the data. For example, “What are the roads that intersect Microsoft’s main campus?” A more complex example would be “What is the area of all parks within 1 kilometer of Microsoft’s main campus?”

How do we do this? First, we need new data types to be able to store this information. Toward this end, we are introducing two new types in Katmai:

· The first type, “geography”, will store points, lines, polygons, and collections of these in latitude/longitude coordinates using a round-Earth model. Most commonly-available data is given in latitude/longitude coordinates, so we expect that most people will want to use this type. Furthermore, this type will give correct computations on a true ellipsoidal model of the planet. What is the area of Indonesia? Will my flight from Seattle to Beijing take me over North Korea? Where can I store my GPS readings? This is the type for you.

· A “geometry” type to support flat-earth data. For those of you who are familiar with such things, this type is our OGC-compliant offering. In some ways, this is a more specialty offering for people who need to work in projected map coordinates either for legacy or legal reasons, but this type can be used for things like interior spaces as well, e.g., “Where in this warehouse is my book?”

If we take our roads data above—perhaps all of the roads for the United States—we could store them in a table Roads:

Roads(name varchar(30), location geography)

I.e., geography is a column type just like any other. We expose a pretty comprehensive set of operations on these type through a method-based interface. For example, if we have a geometry variable @microsoft that represents Microsoft’s main campus, we can find out which roads intersect it with the query:

SELECT name
FROM Roads
WHERE location.STIntersects(@microsoft) = 1

Given a similar Parks table containing all US parks, we can ask our parks question from above:

SELECT SUM(location.STArea())
FROM Parks
WHERE location.STDistance(@microsoft) < 1.0

Of course, beyond answering this query, we need to be able to answer the query quickly. Since my data may be very large, speed is going to mean having a good spatial index. Perhaps that will be the subject of my next post. Watch this space for more.

Cheers,
-Isaac