PDC 2009: New topics on Context-aware Computing

Hi there,

I’ve been quite for some time on this blog. The usual day-by-day job (ru) is making most of my time, but the progress on context-awareness research is slowly growing :)

same as year ago.... http://blogs.msdn.com/semantics/archive/2008/09/25/windows-7-context-awareness-on-microsoft-pdc-2008.aspx

anyway today I wanted to say added to Windows 7 and to once-heard promise of WinFS, this time at PDC 2009 we are proud to announce another brick in the puzzle of Microsoft Context-aware Computing vision brought to live:

Microsoft Semantic Engine

Just look at this PDC 2009 Talk:

Microsoft Semantic Engine

See how the Microsoft Semantic Engine brings search, structured query, and analytics together in a simple, unified Search-Discover-Organize usage model. Learn how it addresses the need to have unified access to structured and unstructured enterprise data through easy to use analytical tools. Also learn how to enable business insight to support decision making at all levels within the enterprise. Get an overview of the Semantic Engine, its architecture, the product and its APIs.

• Tags: Servers, SQL Server

Of course this time it should be the most intriguing session. This one will be lead by Naveen Garg, head of Program Management in Microsoft Semantic Engine team (part of SQL Server Org which once incubated WinFS). I can’t say more on this topic until PDC, but be sure, this will be a VERY interesting topic, really!

Expect more news – now from PDC – join us there!

In Reply to CNN Interview with Gordon Bell on His New Book “Total Recall”

First of all, let me share my biggest congratulations to Gordon Bell (MSR) with publishing of his great book “Total Recall”!

Let me just remind you, Gordon was the one behind the great project of Digital Memories, the “MyLifeBits” project. It is an actual “Memex” project brought to life with Microsoft Research and Gordon Bell as the soul and main author and driver of the project.

The actual reply here is to a recent CNN Interview with Gordon Bell regarding his new book, “Total Recall”, that summarizes the results of experiment Gordon Bell took part in for last few years, the MyLifeBits software project, and all the related research articles, experiments etc. done in this area.

Particular, the question that I would love to comment on is the following one:

CNN: Are you worried about losing your memory?

... Forgetting is not a feature, it's a flaw. I don't think forgetting is an important feature of human memory. I think it's important to be able to remember things accurately.

In this reply, Gordon Bell told about opportunity to remember things accurately. For me, it is a slightly important topic that needs to be fully addressed in a more formal way like an article but for the purpose of this writing and with regard to readers of this blog here are few ideas that briefly comment on this suggestion.

I. In Physics, there is understanding that every process can be described, modeled precisely using Math and Physics formulas. However, it is also known that to be totally accurate everything should be known totally precisely. The opportunity to describe everything precisely in Physics comes down to quantum mechanics and here is a moment that there is an Uncertainty principle postulated by Heisenberg. To be short, the point is you’ll never know either precise position of electron or its precise momentum. Thus being showing something precisely, or, as Gordon Bell names “accurately” is technically impossible. This leads to question, if it is impossible to record some processes (and our life that we record with computer power is about series of different processes, after all) precisely, what options do we have?

II. My answer to this question comes from the so-called “General Semantics” approach, first introduced by Alfred Korzybski in his “Science and Sanity, an Introduction to Non-Aristotelian Systems and General Semantics” in 1948. It is that it is impossible to record things accurately. It is also that what we record always will be based on our opinion of the facts we record. And if use computers then we just build them with our opinions in minds. Thus what is important that a real digital memories system will have it’s own opinion that will be based on opinion of system’s author, or, if system will be intelligent enough, on opinion of its owner.

This answer is based on the following principles of General Semantics (let’s name process an event in this particular discussion):

• The map is not the territory itself – which means that any particular event is not equal to its representation, where territory is an event and map is representation
• Every particular event can have a human representation of it being objectified; but there are different object representations given by different parties;
• Every particular event has n of characteristics, or properties representing it where ; thus each event can have m of objects representing it where   and each object highlights different sets of event properties;
• Every object representing a particular event is an abstraction of this event given by a specific point of view or need because of which some particular event properties are highlighted in this object;
• Every object can have its own object representation; because amount k of event properties showcased in particular object is countable but number n of event properties is uncountable; ; this leads to fact that amount of all possible abstractions of event uncountable.

One of the fundamental principles of people intentions is that if there are several opinions on something the most wise thing we can do is to find common things in these opinions in order to precisely describe that exact something. Thus the most interesting question for me is that actually, a recording system should be able to:

I. Be able to record situation of user from different perspectives/opinions

II. Be able to interact with other digital memories systems used by other people in order to build a better/more correct combined image of the situation

And here comes, that a simple recording system turns into a much more complex context-aware system.

The question comes how based on these ideas we could build a better digital memories system?

Information Overhead Hypothesis

Introduction into Context-aware Computing Vision

When looking on to the past we can see that amount of information available for average person was growing each century but the most dramatic growth was established in the XXI and XX century. Also the less fast growth in previous centuries was easy to handle by average human mind the latest breakthrough was too fast for human brain to adapt. Growth of basic required education as well as growth of information flow around us (books, newspapers, Internet) combined together brought us to a situation when an average person doesn’t process all the information but rather uses the templates to process it automatically. The templates are provided as part of education as well as via aggressive advertising and popular shows on TV, radio, books, Internet.

Another interesting trend is that amount of people dealing with information as their primary job was also dramatically increased in XIX and XX centuries (and is growing even more today). A new job category, knowledge or information worker, appeared on the jobs market.

The third trend appeared as part of information overhead problem is that amount of tasks people process daily also dramatically increased from hundreds to hundreds of thousands of tasks.

Task, or activity, as stated in Activity Theory, is described as dynamic, collaborative, multifaceted, it exists at different levels of granularity, exists across places[1]

It’s the way people solve different problems. The problem of a growing number of tasks leads to a growing complexity in a life of an average person while solving personal and work problems.

The information overhead that humanity is facing now thus consists of inability to efficiently process large information blocks solved by applying templates (which turns off thinking), growing amount of people whose primary job is dealing with information (hence information workers), and growing amount of activities.

[1] “In Supporting Activity in Desktop and Ubiquitous Computing” by Stephen Voida, Elizabeth D. Mynatt, and Blair McIntyre (printed in “Beyond the Desktop Metaphor”, MIT 2008).

MSR Project Tuva: Augment Feynman Physics Lectures Videos with Contextual Information

Hello there,

Wanted to share with you my excitement on the Project Tuva by MSR. Check it here: http://research.microsoft.com/apps/tools/tuva/index.html

The goal of project is to understand how we can augment video with contextual information.

Let me return back in my memory to about almost 5 years ago when I was a student in my institute and learned physics. Feynman lectures were a big help for me to better understand physics and look at it from a different perspective.

Let me recommend you to look at the project Tuva and you should definitely watch Messenger Series on "Prof. Feynman lecture on Natural Law example - Gravity Law" at Cornell University especially the introduction video is a lot of fan :) For those who ever read Feynman books it will remind how fun, spectacular was this well-known professor.

Enjoy!

How Microsoft Well Executes on Its Vision?

Hello there, continuing on the topic we discussed in previous post.

This time, let's look onto Microsoft's Information Worker Vision. As you might remember, the Microsoft Center for Information Work team with Russ Burtner, UX Lead Designer, had put together a Productivity Vision back in 2005 for next 5 years.

Long Zheng over his IStartedSomething.com did a great overview of that video... See here: http://www.istartedsomething.com/20070314/microsoft-2010-rtc/

Now, 5 years later, where Microsoft is?

Surprizingly, Microsoft did a lot... Voicemail in Exchange, Unified Communications, Tablet PCs going mainstream... Of course everything here required a lot of direct involvement of Partner network and the results are really interesting...

Read Long Zheng's review: http://www.istartedsomething.com/20090713/reflecting-microsofts-2010-vision-from-2005/

Communication History from Longhorn Prototypes in 2004: Now Possible With New Windows 7 in 2009?

Had a conversation with Yochay Kiriaty, who leads Windows 7 Evangelism for Developers in DPE and he had let me know about the new Windows API Code Pack that was just released a week ago.

In short, Windows API Code Pack is a project inside Windows SDK Team with goal to provide .NET Developers with managed APIs for Windows. One of the cool new things supported by this Pack are Libraries and Explorer Browser Control.

Do you remember the Communication History concept from Longhorn Prototypes in 2004?

So as I was excited by that prototypes about ugh 5 years I wanted to look if we can Finally do that same stuff in 2009 but with real technologies (I mean it; no WinFS guys).

This is what I was able to achieve with half day of coding (BreadcrumbBar is from VistaBridge Library v1.4):

The screenshot above should show a filtered view of a “Communication” library. The reality is (as far as we do not have any WinFS and there is no Communication Library and messages are not stored in it) that I’ve created a saved search (and it’s really local only as it involves name of my Exchange Mailbox in scope), named it as “Received Messages” and set the scope for my Inbox folder. I did the same for Sent Messages search, too.

The list with filters under “Communication Links” is showing real content of “Filters” folder stored inside the manually created “Communication” library. The messages you see on the screen is my real world copy of Inbox folder of my Exchange mailbox so I cleared personal info from the screenshot. The Explorer Browser Control is used to show saved search results; each time a new item arrives to Inbox in Outlook this search result is automatically updated so actually you can work with your inbox through my app without need to go to Outlook.

Also I didn’t want to focus on Back, Forward buttons as well to Command Bar buttons – hopefully you’ll forgive me :)

The Navigation, Details, Preview, Commands panes are hidden by default (in order to mimicry the Longhorn 2004 Prototype User Experience). I also changed the gray background for the window to a Glass one (it was the goal for Longhorn, too).

So far, it seems that we can make the 2004 Longhorn’s promise a reality also 5 years left from that time and the future is not about just this stuff anymore…

-the Best,

Daniel

First public Location Platform Driver is available – Find Your Location Based On Your IP address!

Hello there colleagues,

I’m sorry for not writing too much last time, too busy with regular job that is you now in this crazy time becoming extremely important :)

So far, I wanted to say that also excuse us and our partners across the globe for no public drivers supporting new Location Platform yet, we are working hard to make these drivers happen, consider to hear more public news in coming months!

This time, however, I want to highlight the community focus on innovation, and, particularly, let me share with you my excitement that first public, community, Location Platform Driver is available!

Dan Griffin, MVP, worked hard to deliver a driver that uses IP address of user’s machine and Live Maps functionality to give you your location. Of course it is not very concrete but kudos to Dan it is the first such driver that gives you opportunity to play with new Location Platform in Windows 7!

You can learn more about it here, and grab here:

http://laptoplojack.codeplex.com/

The coolest thing is that you do not have to use GPS (And thus buy one, connect to your device etc) to get your approximate location and start creating location-aware applications today!

And remember that you can use Windows 7 Navigation Application to see your location on the map :)

Dan, Bravo!

MSR TechFest 2009

Hi there colleagues!

I’ve been keeping silence for last few months. It’s because we are working on a project around context aware systems and augmented reality, bringing ideas together.

Also I can’t disclose more details let me share with you a few links on AR-related projects that were presented @ MSR TechFest 2009 just these days!

Demos:

Sticky Notes in AR

http://research.microsoft.com/en-us/events/techfest2009/demos.aspx (pre-last demo)

unfortunately no video :(

Project Videos:

Hope you’ll find these projects interesting :) And you if you got your ideas on how to marry Augmented Reality and Context Aware systems share your aspirations in this blog :)

First Context-Aware Platform for Windows is launched today!

I want to share with you our happiness here at Microsoft for Chris Karr’s new project, Jarvis, first deliverables.

Chris Karr is working in The Laboratory for Understanding Collaborative Technology in Northwestern University, IL, US. His work is concentrated around the problem of context-aware solutions and about a half of year ago he shipped a context-aware plugin for Mac OS X, known as Pennyworth. Pennyworth is a system that connects to several data sources like your HDD, your apps etc and tries to transform this information into knowledge – facts. The power of the system is exactly what we outlined in our presentation on Context Awareness here at Microsoft when we talked on this topic at World Usability Day 2008 in Moscow, Russia. Pennyworth has learning subsystem that makes good job in learning user behavior patterns and tries to predict his context, what he is doing (activities). The power that comes from this learning is hard to underestimate: computer can know act on behalf on you, helping you in your everyday tasks.

So let me welcome the new Context-aware system for Windows, Jarvis. Jarvis is based on .NET platform, UI is written for WPF. System has several sensors:

As you can see, Jarvis knows what app I’m working in, it is aware of power (either I’m connected to “Wall” or I’m on “Battery”), knows networks I’m connected to, has list of apps that are currently running, it even knows what devices are connected to my device. This is how Jarvis hears us.

That’s how it asks us if its prediction is correct:

and this is how it shows its prediction in Taskbar:

and here is how Jarvis is predicting (actually) the context:

what’s really cool is that we can define Rules for Jarvis to manually control it’s behavior

but we also can work with Learners and help Jarvis to self-predict in which context I am now so that it could act on my behalf which is really cool.

Few scenarios coming to my mind for Jarvis (and Context-Aware Computing in general) – Presence (if I’m working actively in some app and it’s say Outlook not a game then my presence should be set to “Busy” etc), setting different backgrounds and themes for different locations (Based on network interface) etc.

Let me just add that we have in Windows 7 the specific Sensor and Location Platform that makes writing context-aware applications a bit easier, and combined with Jarvis power Windows 7 will help you to turn your applications to context-aware ones as easy as it could be :)

You can read more about scenarios in our presentation on Context Awareness in UX here.

And you can start trying out Jarvis by downloading it from Jarvis site.

UPDATE: Chris asked me to point out that Jarvis is in very early stage and there are many steps beyound initial build and a product compaged with current state of Pennyworth for Mac OS X context-aware system. For example Rules that I've mentioned are not yet working as some other features, too. Follow "Context Blog" for updates.

Chris, it’s great job!

Windows 7 Navigation Application Released!

Hello to all!

As you remember two months ago just a week after PDC 2008 I’ve created a sample Windows 7 Location Aware WPF Application. You can consider it as a mashup:

What It Showcases?

1. How to access Windows 7 Location Platform data in your .NET application
2. How to show this data with Virtual Earth 3D and track movements automatically by subscribing to Location changes from Location Platform API
3. How to use WPF Ribbon Control and VE3D WinForms control to make easy Natural User Interface for interaction with app

Requirements

1. Windows 7 Beta
2. .NET Framework 3.5 SP1
3. Virtual Earth 3D Control (go to maps.live.com and click 3D to install control)
4. Windows 7-compatible Location Provider (say GPS)

Thanks

Let me say thanks to Yukinori Morishita from Redmond for his implementation of LocationProvider that I used to subscribe to changes of Location data. Thanks come also to Prasanna Padmanabhan (PC|3, Sensor and Location Platform team Test Lead) who gave me detailed explanations on Location API design for Windows 7 M3 and Windows 7 Beta without whose help Yuki’s LocationProvider would be much harder to implement.

Download

Download from MSDN Code Gallery

Application Site

Windows 7 Navigation Application @ MSDN Code Gallery

All additional info on how to work with this app will be hosted on this site.

New Russian blog on developing context-aware systems on Windows!

Friends and colleagues, Happy New Year!!

Now when Windows 7 is available for everyone, it’s now time for developing on Windows 7!

Now we are creating a new dedicated blog on Windows 7 where we will talk about how you can develop context-aware applications using Windows 7!

This blog is in Russian so if you do not speak Russian just continue reading the blog “Thoughts and Ideas on Semantics and Contexts”. The Russian one will have the same posts as in “Thoughts and Ideas” blog.

 UPDATE: Thanks to Denis Mak, here is a link to Russian blog: http://blogs.msdn.com/wincontext/

Happy New Year! Windows 7 Beta is here!

Hello and Happy New Year!

Windows 7 Beta is here!!!

· Get the Windows 7 Beta: Download the Windows 7 Beta from the MSDN site.

· Test for compatibility: Ensure your new applications are compatible with Windows Vista; that is the best way to prepare for Windows 7.  For ISVs already compatible with Vista, encourage your local application developers to begin testing their applications on the Windows 7 Beta bits. Check out the Windows 7 Application Quality Cookbook to help with this.  Early application testing will ensure a smooth migration. Help them learn more about Windows 7 using the Windows 7 Developer guide, and optimize performance using the Windows Performance Toolkit.

· Innovate with Windows 7: Increase awareness regarding what’s possible with Windows 7 and encourage your customers and partners to innovate and get logoed.

So now you are able to easily start trying out Windows 7, and in coming months  there will be sensors on market to use with Sensor and Location Platform. Expect to receive SDK shortly :)

Other news

Materials from Microsoft Tech.Day 2008 @ MSU are posted to live site!

As you know we (Microsoft Russia folks) with help from Redmond Windows Product Group (PC|3) talked about Windows 7, it’s Sensor and Location Platform and how they can together enable you to write cool new Context-Aware applications, at Microsoft Tech.Day at MSU last year in December.

So the page was updated and materials were posted :)

here you go: http://microsoft.cs.msu.su/events/Pages/TechDay2008.aspx (warning: Russian language!)

Few interesting posts to read about Sensor and Location Platform in Windows 7

and, btw, found a very interesting article on our lovely Windows Sensor and Location Platform (again in Russian) here:

http://ko-online.com.ua/node/39327

also you might find it interesting to look onto these posts (both are in English):

• http://www.patrickverbruggen.com/2008/11/20/windows-7-sensor-and-location-platform/
• http://www.informationweek.com/blog/main/archives/2008/12/microsoft_comes.html

Point for Discussion: Privacy for Windows 7 Sensor and Location Platform

And if you are thinking about Privacy and critics of how it was implemented in Windows 7 for Location Platform check out this post: http://www.aclunc.org/issues/technology/blog/location_tracking_in_windows_7.shtml

I think it is important here to comment:

in Windows 7 we tried to make Sensor and Location Platform a system that will consider all information coming from sensors as PI (Personal Information). This leads to fact that user should be in control for usage of this information. Thus by default if you insert a new sensor into your Windows 7 PC it will be disabled and if any application will want to access it Windows will ask you if you want to enable this application to access this sensor or no. But, the implementation is that you can control if sensor will work or won’t work. You do not have granularity of kind when you can control access to sensor data by apps. This might be considered as a huge problem, but

THERE IS NO WAY TO IDENTIFY THAT APP WAS NOT REPLACED BY VIRUS. THERE IS NO WAY TO IDENTIFY IF APP THAT YOU GRANTED ACCESS TO SENSOR DATA WAS NOT CHANGED.

It’s where simplicity of creating new apps for Windows makes it also impossible to easily track all apps in the system, IMHO.

What do you think? How would you implement such a system?

Few links on Virtual Earth 3D Control

Hi there,

I’m working on porting my Location Aware WPF Application to Virtual Earth 3D Managed WinForms control and found that there is not so much available information on its usage.

Let me share with you several links you might find useful. Here is a quote from one of my colleagues from Redmond’s Virtual Earth team:

The best available documentation for using the managed control directly is on the VE3D blog here:

http://blogs.msdn.com/virtualearth3d

You will probably want to start here:

http://blogs.msdn.com/virtualearth3d/archive/2008/04/19/creating-the-most-basic-windows-form.aspx

Moving the camera to a position is covered here:

http://blogs.msdn.com/virtualearth3d/archive/2008/10/22/camera-control.aspx

Adding pushpins and other geometry is here:

http://blogs.msdn.com/virtualearth3d/archive/2008/04/19/geometry.aspx

All of the samples are here:

http://blogs.msdn.com/virtualearth3d/archive/2008/09/25/current-samples.aspx

Other blogs:

• http://ve3d.spaces.live.com/blog/
• http://blogs.msdn.com/kristoffer/

Created a Group on Windows Live to store all materials on Context Awareness from Microsoft

http://cai.groups.live.com/

you will find here all materials from Microsoft on Context Awareness including slides from WinHEC, PDC, World Usability Day and Tech.Days.

I will update this group when new slides will be available and post into this blog as well.

We’ve set up also an initial discussion on what the Context Definition is – feel free to join the conversation!

Have a great weekend! :)

Videos of our talk on Context Awareness at WUD 2008 are published now!

Happy to share with you that videos of our speech (in Russian, sorry) on Context Awareness technologies that we made on World Usability Day in Russia on November 13 this year were published:

You can find English version of our presentation in the previous post.

Wanted to say thank you to RusCHI (Chapter of SIGCHI) and ACM for invitation to World Usability Day Russia 2008!