Virtual Earth 3D team blog : DataSource

X File bulk display

NikolaiF — Mon, 21 Sep 2009 00:42:00 GMT

I was asked the other day about how to load multiple x file -based models into VE3D without having to place each one manually. I've written up a sample here. The idea is more or less a fusion of the existing ActorDataSource and XFile samples, using the actor from the XFile sample rather than the bunnies. I've also mocked up a very simple data file that contains placement information, rather than generating data on the fly as for the bunnies.

The way I've done this, the parsing code for the x files is done in the actor. In this way, swapping out file formats should be straightforward so long as you have parsing code for it (sorry, only x files are directly implemented in the engine, but the results of any parse should still be easily convertible into VE3D graphics objects).

For the sake of simplicitly I did do one thing wrong. The flow of the code is like this: multiple background threads all execute QueryPrimitivesInternal at the same time, and when they are finished, they move through a lock into ActorBuilder. They are still on background threads, but they now execute serially. So, really the data retrieval (in this case, loading from resources, but probably a network request) and the parsing, as appropriate, should happen in QueryPrimitivesInternal, and the action in ActorBuilder should be kept to a minimum. I violated this rule in the sample code for the sake of illustration, but you should be aware of it.

Finally, an important consideration is level of detail. Dense and numerous models can slow down loading and rendering. DataSources take advantage of the ActorBounds and scale concepts, such that it is possible to use simplified models at certain LODs, and switch to more detailed as the user zooms in. If your needs are less extreme, then you can simply use scale to cause models to stop rendering when not in view, and at a certain distance, which is what I have done in the sample.

Have fun!

Actor data source questions

NikolaiF — Sun, 26 Jul 2009 21:06:00 GMT

I just fielded a few good questions via email about actor data sources, and thought it would make a good post.

Actor data sources are a way of getting little bits of your code (actors) into the world, handing spatial indexing and cache management for you. When each actor is in view, they are given the opportunity to execute code in the Render and Update functions. When the cache is full and an actor is evicted, they again may execute code in the OnRemove function.

You create them by implementing a DataSource with usage Actor, and an ActorBuilder which translates the Primitives returned into Actor objects, along with information like what areas of the Earth the actors should be used in, as an ActorBounds object. All of this can be seen in the ActorDataSource sample.

In the sample, however, only one Primitive is returned per request, and only one Actor. What if you want more? The important thing to know is that there has to be a 1-1 correspondence between Primitives and Actors *. You must create one Actor for every Primitive used. If you create more than one Actor for a single Primitive, only one will appear.

The reason for this is the Actor update story. It is possible to update an Actor already in the system by two methods. When an Actor is updated, the PrimitiveId from the passed-in primitive is used to find and update the actor in question. If you add multiple actors per primitive, an ambiguity arises as to what you actually want to do.

It is possible to resolve this ambiguity by creating "nested" actors. This is simple. Just create the actors you want, and then a "container"

*: actually you can create a single Actor with multiple Primitives, but I don't want to confuse the issue for now.

Updating an Actor:

Expiration. On the Entity attached to your primitive, add a property "ExpiresInSeconds":

e.EntityTypes["bunny"].Properties.Create("ExpiresInSeconds", typeof(double));

...

entitySpec.Properties.SetValue("ExpiresInSeconds", 10.0);

Your QueryPrimitivesInternal function will be called again for every area the affected Actors live, and OnRemove will be called for the expiring Actors.

Manual. You can specify a specific PrimitiveId to update using OnDataChanged in your source DataSource.

PrimitiveIdWriteableCollection wc = new PrimitiveIdWriteableCollection();
wc.Add(new BunnyPrimitiveId(this, tile));
OnDataChanged(new SpatialExtent(null, null, wc.GetReadOnlyCopy()));

You must also implement the QueryPrimitivesByIdsInternal function. A collection of ids is provided, and you must return an array of Primitives that matches the input (so, the first item in the id collection should result in a Primitive at array index 0, etc). If an element of the array is left null, that means to delete the Actor corresponding to that PrimitiveId. Your ActorBuilder is then called to allow you to create the new replacement Actors.

You can also specify a region of the Earth to update in the SpatialExtent, but update by entity is not supported. Updating very large areas can be expensive, so use spatial regions with care. Update by PrimitiveId is the most efficient.

WMS Data

NikolaiF — Tue, 09 Jun 2009 20:21:00 GMT

I was having a discussion with Kurt Guenther from Infusion yesterday on the topic of WMS servers and VE3D. There is a large amount of very interesting spatial data out there served by Web Map Services, or WMS servers. VE3D is able to process this data using ConnectionParameter-based DataSources. Setting it up is pretty easy:

ConnectionParameters cp = new ConnectionParameters(http://sampleserver1.arcgisonline.com/ArcGIS/rest/services/Portland/Portland_ESRI_Neighborhoods_AGO/MapServer/export?bbox={16},{17},{18},{19}&bboxSR=4326&layers=&layersDefs=&size=256,256&imageSR=102113&format=png&transparent=true&dpi=&f=image);

string connectionParameters = cp.ToString();

DataSourceLayerData layerData = new DataSourceLayerData(LayerName, Name, connectionParameters, DataSourceUsage.TextureMap);

Host.DataSources.Add(layerData);

Drop this in to your Activate (plug-in) or Initialized handler (winforms etc) app and fly to Portland, OR to see the data (thanks to ArcGIS for the sample data and Kurt for the pointer).

I won't go into all the details of the WMS spec here, but I will call out a few important parts. First, in order for this to work you must specify the bbox to be in Geographic coordinates (4326). This is usually the default. Second, you must pay attention to the returned projection. In the above case, we have instructed the server to return 102113, which is Web Mercator, VE3D's native image projection. This data can be consumed directly with very little overhead. However, different servers support only certain formats and projections, and you must be careful for any particular server (it is possible to query this using WMS's GetCapabilities). Nearly all support returning images in Geographic coordinates, 4326. It is possible to instruct VE3D to reproject this to Web Mercator (this is the only reprojection VE3D will do for you, fortunately it will work for the majority of data).

Now, to look at ConnectionParameters and what it can do for you. First, there's the big url. Notice that the bbox has string.Format replacement variables in it: {16}, {17}, etc. Requests to the server are made on a per-tile basis (the T key is useful here). For each request, these variables are replaced with data specific to the tile in question. Here is a list of available parameters:

0 Reserved

1 Map Style

2 Round-robin integer

3 Reserved

4 Quadkey

5 Extension

6 Generation

7 Stripe (0-3)

8 Tile Host

9 App Host

10 Language code

11 Region code

12 Tile LOD

13 Tile X

14 Tile Y

15 Low order stripe (0-1)

16 MinLong

17 MinLat

18 MaxLong

19 MaxLat

20 RequestToken

21 Culture name

The most interesting ones are 4 (the main quad-key, very useful for already-tiled data), 12, 13, 14, and and 16-19. To see some of this in action, check out the TerrainImages sample (also available in the sample code of course). Programatically, you can investigate these values using the TileId class, using GetRequestCode, GetPosition, and GetLatLonBoundingBox.

As for other values on ConnectionParameters, you can control reprojection (if your data is in Geographic, specify CoordinateReferenceSystem = WGS84CoordinateReferenceSystem.Instance, as noted before this is the only reprojection natively supported, so otherwise leave this field uninitialized), the bounds of where the data is valid is terms of Lat/Long and LOD (useful to reducing unnecessary network requests), and caching behavior.

Caching behavior deserves special discussion, because WMS servers are often quite slow. By default, data loaded using ConnectionParameters is not cached locally. Once it moves out of memory, it must be re-queried. If you set CacheRetention however, the data will be kept locally for the period of time specified (unless it is evicted due to space restrictions in the interim). If you set CacheRetention = new TimeSpan(24, 0, 0), for example, for 24 hours from the time of query a given tile will not be re-queried. Note that some servers are dynamic, such as weather data, so it's important to use an appropriate value here.

And just to give Kurt a poke in the ribs, please remember to only add DataSources after/during the Initialized event!

Update:

John Fletcher from Latitude Geographics pointed out that the example I give above isn't actually WMS compliant in its query parameters. He suggests an alternate sample, of the British Columbia, CA area:

ConnectionParameters p = new ConnectionParameters("http://openmaps.gov.bc.ca/mapserver/libcwms2?SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&LAYERS=DBM_7H_MIL_BATHYMETRIC_POLY,BC_LABEL,DBM_7H_MIL_DRAINAGE_LINE,DBM_7H_MIL_DRAINAGE_POLY,DBM_7H_MIL_ROADS_LINE,DBM_7H_MIL_POLITICAL_POLY_PS,DBM_7H_MIL_POPULATION_POINT&STYLES=,,,,,,&SRS=EPSG:4326&BBOX={16},{17},{18},{19}&WIDTH=256&HEIGHT=256&FORMAT=image/png&TRANSPARENT=TRUE&EXCEPTIONS=application/vnd.ogc.se_inimage&");

p.CoordinateReferenceSystem = Microsoft.MapPoint.CoordinateSystems.Wgs84CoordinateReferenceSystem.Instance;

Host.DataSources.Add(new DataSourceLayerData("foo", "bar", p.ToString(), DataSourceUsage.TextureMap));

John also provided a link to the WMS spec repository (http://www.opengeospatial.org/standards/wms). Note that the sample above is for WMS 1.1.1. The most current version is 1.3.0, but many servers use the older, or support both. No matter which you use (or even for something a little off spec like my original example) the interface with VE3D is the same.

Data Format Revision

NikolaiF — Mon, 06 Apr 2009 20:17:00 GMT

As part of our version update, we are also updating the file format of our 3D buildings. The upside is that the new data format is significantly smaller, for faster download. The downside is that old versions of the code cannot understand the new data. However, in the short term this should not affect you. There will be a period where old data will still be served to clients that are using the Javascript API and the Managed API. This represents a bit of a compromise, because even clients that have updated to the new version will get the old data for this period, but we felt that this was better than releasing new code and breaking old (and existing) code on the same day.

To see the new data after the version update, simply navigate to maps.live.com and click 3D. The new data will be live on the site.

Once this period is over, clients that have updated will automatically begin receiving the new data without any required changes or interruptions. You will probably hardly notice, except that buildings of course will load faster. Clients that have not updated will break! You've been warned! The only issue really will be that buildings will not appear, and only if your code is directly adding model data sources.

The exact length of this period is TBD, but probably about a month. Our hope is that most everyone will update by then anyway, because the new version really is better. I will blog a reminder when the date is decided and approaching. As always, let me know if you will need any help during the transition.

In addition, in several samples I directly linked to the manifest files for some resources, like imagery, dem, and models. These are the urls used when creating DataSourceLayerData objects. I encourage you to instead use fwlinks, which are just redirects to the same resources. This will help us update data while minimizing impact to clients.

Road:  http://go.microsoft.com/fwlink/?LinkID=98770
Aerial:  http://go.microsoft.com/fwlink/?LinkID=98771
Hybrid:  http://go.microsoft.com/fwlink/?LinkID=98772
DEM:  http://go.microsoft.com/fwlink/?LinkID=98774
Models:  http://go.microsoft.com/fwlink/?LinkID=98775

Edit: took down the traffic link. If you are interested in traffic data, please see here: http://msdn.microsoft.com/en-us/library/bb877877.aspx

Actors and DataSources

NikolaiF — Mon, 27 Oct 2008 23:51:00 GMT

In this sample, we create a DataSource, which is a class that can be used for large-scale data sets. It provides hooks into the system's spatial index, memory cache, and background data threads. Data from the DataSource will only be requested when it is in view, and will be automatically cached according to how often it is used and how much memory it consumes. The DataSource we create is a purely generated one consisting of a large number of animated bunnies spread evenly across the globe, and would be impossibly large to try to deal with all at once.

See it in action!

When the PlugIn is Activated, the ActorDataSource is created and added to the system. When we create the DataSource we set up its Ontology, which is a system for defining the type of data the DataSource contains.

The next time the Camera Query runs, a set of areas in the Spatial Index that are in view are determined, called tiles. Each tile is sent as a lat/long bounding box to the Data Threads, which call all added DataSources, including ours. The entry point here is QueryPrimitivesInternal. In this function we would normally do some sort of query to a backend, such as a SQL database or the network, but for simplicity we simply create a BunnyPrimitive in the center of the query region. A primitive is a piece of data that represents a thing in the world, in the case a Bunny, but is not itself an Actor. When we return the Primitive, it is passed to our ActorBuilder, called BunnyBuilder. This creates actual Actors which can render or react to the data represented by each Primitive. Each Actor also needs an ActorBounds, which represents a spatial area that, when in view, should cause it to render. When this happens each Actor's Update and Render functions are called on the Render Thread just like normal Actors.

Get the code!

Terrain Imagery Overlays

NikolaiF — Sun, 12 Oct 2008 20:15:00 GMT

There are four ways to add custom imagery to VE3D. The first three involve using built-in classes, and you just have to provide information on where to get the data. The fourth involves implementing your own DataSource.

See it in action!

This sample demonstrates how to add common terrain imagery data to the world, and adds a simple tour to display the example data sets. We then use two events, one a built in data load event, and the other a custom event we created using the Step class, to tell the control to FlyTo a series of locations.

When the PlugIn is Activated, we create three datasources using built-in settings. The first uses a SingleImageDataSource, which reads data from a Bitmap object in memory and draws it on the terrain. The second uses a "content manifest", which is an xml file that describes the data, how to display it, and where to get it. An example is here. The last uses a parameter string, and is similar to VETileSourceSpecification. The % replacement variables in tilesource have the same numbers as those listed at the top of the xml manifest. The parameter string is also a good method for adding WMS server data.

It is also possible to create your own datasources.

See it in action!

This sample demonstrates how to add terrain imagery via a custom datasource. This one generates a colored Mandelbrot set.

When the PlugIn is Activated, we create our DataSource and add it to the system. We also save off any existing datasources so we can add them back again later. Once it is added, the data threads query it when we move into a new area, entering at QueryPrimitivesInternal. We use the bounding area of the query to generate an appropriate section of the Mandelbrot set, and return it as a rasterized image. This image is then displayed by the system.

This method is good if you have special request logic, or want to do some of the work client-side (in this case, all the work is client-side). This saves you from having to have a large server dataset of pre-rendered tiles, or execute expensive code on your servers. There are notes in the sample code for how to use this code to get information about the area you are looking at and generate appropriate request URLs, etc.

Get the code!