shoshe

Eye on Earth Network Launched at RIO+20

2012-06-25T19:15:00Z

I’ve been selfishly waiting to complete the launch of Eye on Earth Network before sharing more information about the status of Eye on Earth. Last September my team at Microsoft began working with Esri and the European Environment Agency to expand the capability of Eye on Earth – transforming it from a bespoke solution into an open, self-service platform for exchanging environmental information. I am proud to inform you that the new Eye on Earth Network went live at RIO+20 to major fanfare. Not only has the Eye on Earth Network has been chosen for the Sustainia100, the Eye on Earth Network is now part of The Future We Want: Outcome document adopted at Rio+20, paragraph 274:

“We recognize the importance of space-technology-based data, in situ monitoring, and reliable geospatial information for sustainable development policy-making, programming and project operations. In this context, we note the relevance of global mapping and recognize the efforts in developing global environmental observing systems, including by the Eye on Earth network and through the Global Earth Observation System of Systems. We recognize the need to support developing countries in their efforts to collect environmental data.”

The re-launch turns the system into a global public information network for creating and sharing environmental information. Scientists, government, universities and citizen groups can now contribute data and map-based visualizations into this network. The portfolio of content is still small, but growing rapidly, and includes daily ocean temperature forecasts, agricultural areas in Europe, USA land cover, and more. More than anything this information helps set the baseline for the planet, in a very public, accessible way.

I’m honored to be part of the team that is able to make a difference. I would like to appeal to you to make a difference, by sharing your knowledge of your local environment with the world. There are three ways you can contribute:

Explore interactive maps made by others and discover new ways of seeing the environment.
Contribute your environmental observations or data. By further sharing these contributions you are helping to expand the knowledge base.
Create your own new custom maps and share them within closed groups or with everyone. See how this cutting edge technology can become an effective tool for you.

Sharing is everything…

Namaste,

Shoshanna

Intelligently Managing the Earth's Ecosystem

2011-10-28T23:38:00Z

Many of you who follow me on Twitter may have figured out that I, and my product team in Microsoft, have been investing in environmental sustainability solutions. One specific custom solution, called Eye on Earth, was developed by Microsoft and the European Environment Agency and was released for the first time over three years ago. Eye on Earth is a hub for experts, governments, and the public to exchange information about their environment, transforming millions of citizens into active participants in their local environments. www.eyeonearth.eu

My team was asked to turn this custom Azure solution into a repeatable pattern and service that governments and NGOs around the world could use to engage in discussions with their citizens. That’s where my story begins. With the help of Rob Bernard and the Environmental Sustainability team at Microsoft, I began my journey into the world of environmental agencies, NGOs, ecology groups, environmental scientists, commercial solution providers, global initiatives, and environmental advocates. With the eye of an engineer (I have very little tolerance for chaos, even if it’s for the better good) I became both frightened and hopeful, and I developed my own perspective for building a sustainable environment for our shared planet. I admit I’m naïve. I’m not a specialist. But, I’ll still take the brave step of sharing my thoughts with you.

Sustainability

The Earth is a closed system. Effectively managing the Earth’s ecosystem and precious, limited resources can be thought of as a business intelligence problem. Business intelligence problems are best solved with centralized engineering processes that benefit from unlimited access to authoritative data, unlimited storage and computational capacity, and the best available analytics tools and technology. In today’s economic environment, sophisticated industries use BI to respond to changing conditions in days, hours, and even minutes.

And yet, modern civilization is built on the strength of diverse, decentralized, complex social and economic institutions. Independent governments and NGOs throughout the world often create conflicting policies for managing the Earth’s assets based on country conditions, whether political, social, natural, or economic. These policies direct industry, government, and citizen action. Unlike industry, the results of those policies on the Earth’s ecosystem are measured in years, decades, and even millennium –far longer than reign of most modern governments.

Recognizing the decentralized nature of this BI problem, there are straight forward, concrete actions that government and industry can take to ensure that the most creative, innovative solutions emerge:

Publish timely, authoritative datasets. Environmental datasets include air quality, carbon emissions, water quality and water reserves, soil, weather, biodiversity, and more.
Publish scientific baselines for environmental policy decisions.
Make it easy to collect and correlate environmental data with industry datasets, with scientific datasets, and even with citizen data contributions. While all data may not be authoritative, enough data will become statistically significant.
Measure the results and reward positive change.

Governments are already creating incentives that encourage scientists, students and citizens worldwide to create the best solutions and take action based on those solutions. Simple solutions include interactive bicycle maps, automated home conservation systems, crop watering plans, and wastewater recycling systems. The overwhelming support and response to Eye on Earth is yet another positive indicator that NGOs and governments are willing to engage in difficult environmental dialogs with citizens.

We are close to being able to equip our citizens, governments, and scientists with the data and systems we need to make informed decisions about environmental policy. We have the technology. Our scientists have the brainpower. But the solution needs us, too. We must have the will and patience to advocate for government initiatives that require all environmental data is published and publically accessible in days or months of its collection. It must include data collected by governments, by government-funded research, and it must include environmental use and impact data from industries. We must demand access to the information we need to help us create local solutions with global impact. It’s our responsibility to sustain this most precious resource to which we all owe our lives.

Thanks for your attention,

Shoshanna Budzianowski

(Note, these thoughts are my own and do not represent the position or polices of Microsoft Corporation.)

Querying Azure DataMarket Xignite BATSLastSale with OData

2011-04-13T23:40:00Z

I've been waiting for BATSLastSale data for a while. The BATSLastSale service gets real-time stock quotes including open, close, high, low and last sale prices and trading volume. Here's the URL to the data service on DataMarket. https://api.datamarket.azure.com/Data.ashx/Xignite/BatsLastSale/ You won't be able to use it until you subscribe to the service.

If you're not familiar with BATS, you can learn more at www.batstrading.com. In six short years since its founding the BATS Exchange has become the 3rd largest US securities exchange, and just days ago they announced a new listing service for U.S. Stocks, enabling companies to go public on exchanges other than NYSE and Nasdaq. (http://finance.yahoo.com/news/BATS-submits-plans-to-list-apf-2831227869.html?x=0&.v=2) The BATS data is brought to Azure DataMarket via Xignite. Xignite is the market leader in on-demand financial market data distribution. Xignite is listing their data services on DataMarket to give developers, like me, broad access to financial data. I'm thrilled, because I don't need to pay lots of money, and I can do simple and powerful things.

The first step for me anytime a new dataset gets published into DataMarket is to play with the data. Power Pivot and Excel are all good choices, but for this dataset I want something more real time, and I'm not a big UI person. I prefer my data raw. Yes, I could buy into a trading platform and get this same data, but I'm a recreational trader. I'm not even a day trader. Before I go any further, a simple console app that lets me input a symbol and get back a quote should do nicely.

Xignite BATSLastSale is a fixed query data service, so I followed the MSDN instructions almost without fail: http://msdn.microsoft.com/en-us/library/gg312154.aspx#Step3 There's one small bug in the MSDN instructions which you may have discovered by now. The instructions don't tell you to add a reference to System.Data.Services.Client into your project. Without that, the proxy class that you download from the data service page in DataMarket won't work.

My very simple project has three files:

XigniteBatsLastSaleContainer.cs This is the proxy class that I downloaded and included in my project

BATSData.cs I created this file. It has my helper classes for reading the service

Program.cs Not a very good name - main.

You can read the code below. Here's what the output looks like in the console window:

Enter stock symbol or q to quit: MSFT

BATS Data for MSFT

Symbol: MSFT

Company Name: Microsoft Corporation

Last: 25.59

Delay: 0.004

Enter stock symbol or q to quit: GOOG

BATS Data for GOOG

Symbol: GOOG

Company Name: Google Inc.

Last: 576.11

Delay: 0.004

Enter stock symbol or q to quit:

Notice there’s a Delay value in the results set. 4 ms is pretty darn amazing, but beware, this is the delay from BATs to Xignite. Xignite has the ability to service trade desks. That’s not the sweet spot for DataMarket. The Delay value doesn’t include DataMarket latency, and unless you’re on a dedicated low latency network, you won’t experience anything close to that.

There are a lot more values in the results set than included in my code example, and more query types for you to explore. Have fun!

--- Shoshanna

//BATSData.cs

using System;

using System.Collections.Generic;

using System.Linq;

using System.Text;

using System.Net;

using Xignite;

namespace BATSLastSale

{

class BATSData

{

private Uri serviceUri;

private XigniteBatsLastSaleContainer context;

public BATSData()

{

serviceUri = new Uri("https://api.datamarket.azure.com/Data.ashx/Xignite/BatsLastSale/");

context = new XigniteBatsLastSaleContainer(serviceUri);

context.Credentials = new NetworkCredential(YOUR_LIVE_ID,YOUR_ACCOUNT_KEY);

}

public IList<LastSaleQuoteEntity> GetBATSLastSaleData(string symbol)

{

IEnumerable<LastSaleQuoteEntity> query;

query = context.GetLastSale(symbol);

try

{

return query.ToList();

}

catch (Exception ex)

{

Console.WriteLine("BATS ERROR {0}", ex.Message);

throw;

}

//Program.cs

using System;

using System.Collections.Generic;

using System.Linq;

using System.Text;

using System.Net;

using Xignite;

namespace BATSLastSale

{

class Program

{

static void Main(string[] args)

{

IList<LastSaleQuoteEntity> BATSList;

BATSData dataClass;

Boolean quit = false;

string Symbol = String.Empty;

{

Console.Write("\nEnter stock symbol or q to quit: ");

Symbol = Console.ReadLine();

if (Symbol == "q")

quit = true;

else

{

dataClass = new BATSData();

BATSList = dataClass.GetBATSLastSaleData(Symbol);

if (BATSList != null)

{

Console.WriteLine("BATS Data for {0}", Symbol);

foreach (LastSaleQuoteEntity entity in BATSList)

{

Console.WriteLine("Symbol:\t\t {0}", entity.Symbol);

Console.WriteLine("Company Name: \t {0}", entity.CompanyName);

Console.WriteLine("Last: \t\t {0}", entity.Last.ToString());

Console.WriteLine("Delay:\t\t {0}", entity.Delay.ToString());

}

while (!quit);

}

Data Hopes for 2011

2011-01-03T01:00:00Z

The next generation of data-driven decision making is rapidly approaching. It's an exciting, scary, and frustrating time; it's exciting for developers and organizations that have new access to quality datasets; it's scary for businesses and organizations who have wrapped their business models around limiting access to data; and it's frustrating to information users who don't quite have the right set of tools yet for integrating all that data into meaningful, useful insights.

Recently, I started using a simple taxonomy to describe the value trajectory for data-driven decision-making:

These three categories roughly correspond to the value curve for data. Information is the first order derivative of data. It is a factual representation of data in a simple to understand form that expresses a position. It is compiled from one or more data sources. An unordered list of universities is data. The same list ordered by Total Cost to Graduate becomes information. And even better, correlating that total cost to the predicted salaries of graduates - that's insightful. (See Bloomberg Businessweek report "What is your College Degree Worth" for a great example. I was pretty happy and surprised to find my old alma mater ranked 15^th on the list. Not bad for a regional engineering college. )

So what is insight? I went to the source, Dictionary.com, and found the following definition:

in·sight

-noun

1. an instance of apprehending the true nature of a thing, esp. through intuitive understanding: an insight into 18th-century life.

2. penetrating mental vision or discernment; faculty of seeing into inner character or underlying truth.

Given that definition, the largest value is created when data is combined with analysis and meaningful visualizations to create new insights. Yet, surprisingly, we spend the most effort on the least valuable task - organizing, cleansing, and cataloging data. It's a necessary cost, but this can't be the final outcome. As we walk the value curve, the tools at our disposal become infinitely more costly and complex, and the insights more difficult to obtain.

So what's on my hope list? We begin to recognize the true value of providing unfettered access to high quality information, our tools become more powerful and easier to use, and we're all able to become insight creators and consumers. I'll give that hope 10 years to manifest. For the next year, I'll be watching for some small and even large leaps towards that direction.

Data:

Preparation - Preparation involves compilation, cleansing, and enrichment. While there are tons of companies that will help, I'll be looking for data cleansing services that can be used to enrich published datasets. The complexity here is enabling those enriched datasets to be republished.

Correlation - I'll be looking for Semantic Web and RDF to break the confines of academia and research.

Datasets - The most important ingredient. Following in the footstep of the European Commission's Eurostat service, Data.gov is in the process of publishing all data generated by the Executive Branch of the Federal Government. NGOs, like World Bank and the UN have already bought in to publishing their data. If there's an easy way to get programmable access to IMF's data, I can't find it. Tools like www.needlebase.com can scrape data from websites and create structured datasets, but I do wonder what anyone can do with that data.

On the commercial side, I'm seeing two trends - an increase in the number of specialty content aggregators, and awareness by some of the leading aggregators that the game may be changing. Thomson Reuters, Bloomberg, and LexisNexis have been content aggregator category leaders for a long time. These big three all require custom pricing and negotiations to get access to their premium datasets and feeds- beyond the reach of the average, researcher, or information worker. LexisNexis, D&B, and Associated Press are now starting small experiments with simpler and easier pricing and access. The trend is good. Let's see more.

Publication - There's a rush to commercial data markets, and that's a good thing. I happen to be involved in one myself. They're doing us all a service by compiling and publishing catalogs of multiple different types of datasets and making them simple and easy to access. This is a nascent market. There will be lots of movement and quite a few new players. My hope is that instead of fracturing, we begin aggregating catalogs.

APIs - Go hand-in-hand with publication. Open data protocols, like OData, need to evolve to support more data types. Geo and binary data are a must. So is richer metadata for things like semantic and temporal tagging. And having uniform APIs and semantics over all the various datasets are a necessary ingredient in moving into information and insight creation. We're all spending too much time hard-coding those mappings.

Information:

Mash-up tools - Remember Popfly? I'm sadly predicting that codebase doesn't get resurrected, although I get asked about it almost daily. Google gets a shout-out for creating an entire ecosystem around Google maps. Yahoo Pipes and JackBe Presto use a wires metaphor for connecting feeds and datasets. They're quick to use and great for power information workers and developers. Presto has some pretty good looking charts and graphics at the end. But, my favorite data mashup tool? Nothing beats Microsoft Excel; especially with the PowerPivot add-in. I'm not alone in my opinion. Even the big data community is moving to Excel. (Calculating Excel Models on an HPC Cluster)

Charting and Visualizations - Alas most great information visualizations that I see are hand-crafted by graphic designers. That doesn't scale. There a slew of control vendors for Sliverlight and ASP.NET AJAX controls. They're gorgeous, but they're also data agnostic. We've got great datasets just waiting to be visualized that could benefit from gorgeous visualizations. Will the two constituencies meet?

Insights:

Temporal and predictive analytics - This is where life gets interesting and where it gets complex. A good background in linear algebra, machine learning, and statistics are necessary to build your own analytics. Microsoft StreamInsight is particularly capable at running complex event processing applications, and if you know LINQ you're more than halfway there.

There have always been specialty companies who are willing to do the work for you, though. You can purchase access to financial prediction models from newcomer Recorded Future.

Marketing research is rife with analytics tools and models from well-known names like IBM's SPSS, Cognos, Coremetrics, and Unica, SAS, and Teradata with their acquisition of Aprimo. And now that we're all focused on web site effectiveness, companies like Perfomable, and Experian Hitwise will help you crunch your data. The ones I'm particularly interested in are the ones that won't only crunch your data, but will publish the models back to the ecosystem, so that we can all get smarter and better. Naïve, I know.

My final hope for 2011 is that our public and private education system starts training students in advanced mathematics and analysis. This is an open field and a great opportunity to innovate for the future.

Happy New Years,

Shoshanna

Windows Azure Marketplace DataMarket - Information as a Service on any platform
Resources: Whitepaper | Forums | DataMarket Blog

Twitter: @shoshe