It’s been 1.5 years since I published my last blog entry. It was about the fundamental problems mobile application developers face today: intermittent connectivity and the lack of stable and discoverable addressing. These problems aren’t specific to mobile device world - we have to deal with these communication barriers all the time. We need to share data between computers at work and at home, computers on the Internet and computers hidden behind firewalls and NATs, between internet sites, providing cloud storage (Spaces, Flickr, Facebook). I’m pretty sure everyone could come up with good examples of situations when they needed to access data, but couldn’t because of some sort of connectivity barrier. How many times did you have to use email as the only available mechanism for sharing data between people and devices? Does emailing documents to yourself again and again sound familiar? On .NET Compact Framework team we built store-and-forward messaging channel on top of e-mail, so application developers could use it. Although that is an interesting solution, it was quite a stretch, a desperate attempt to adjust the only broadly deployed and generally available technology to address the asymmetric connectivity problems. And it was really pushing the limits of the channel.
So, how things changed in 1.5 years? I December 2006 I joined the Live Mesh team. We are building a v1 product that has a potential to help people solve many, if not most of the connectivity and data availability problems I’m personally passionate about solving. The product brings together your data, people you share data with and various devices you use to process the data into a single “Mesh”. The bold statement is that you can access your data anywhere from any device. We provide multiple alternative access paths to your data through the Mesh. I’m working on the Accounts team – we are responsible for keeping track of users, devices and applications in the system, security, authentication and authorization.
There are multiple perspectives to look at Live Mesh.
First, I view the Mesh as a collaboration tool between individuals. Every person in the Mesh has a unique Identity. We require every Mesh user to have a Windows Live ID. This is how we identify people in Mesh, that’s what you use to sign into the system and that’s how other people in the system recognize you. Your Identity is the main security principal in the system. If you use multiple Live IDs, each Live ID will only give you access to different data set, associate with the signed in Identity. You won’t be able to access data associated with other Live IDs, unless you explicitly share/sync it.
That’s how I plan to use Mesh: I want to share my pictures with my family and plan to use it as a collaboration tool to work on documents with other members of Accounts team.
Second, I view the Mesh as a set of Devices people own or use. Device ring you see on the front page is a great visual representation of this view. So, what do we call a Device in the Mesh? For single user devices, like phones or PDAs, that’s obvious. For potentially multi-user devices, like PC (multiple people can share the same PC, each person may have different OS account) - the device is identified as a {physical computer, OS account} tuple. For example, if I have 2 different Windows user accounts on my PC, switching Windows user would imply switching between Mesh devices. I and my daughter share the same physical PC, but use different Windows accounts, there will be 2 distinct devices in the Mesh, there will be no way to tell if they in fact represent the same physical device. Note that Device is also a security principal in the Mesh environment and device can be delegated by a user to sync data on user’s behalf, even when user is not signed into Mesh. Each Mesh Device obtains a unique thumbprint from Windows Live ID that’s used for device authentication purposes (Devices are also authenticated with Windows Live ID, similar to Identities).
Mesh is a sync platform. Every time you add Device to a mesh, you create a new sync endpoint. You can then chose what subset of your data you want to sync with the particular device by mapping Mesh Objects (such as Live folders) to the device.
Now how about the Live Desktop? Is that just another Mesh device? Is that a physical computer Microsoft runs for me in the datacenter? Well, not exactly. Live Desktop is a service that provides you with a projected view into your data stored in the cloud. Your cloud storage is always a sync endpoint by default. You never explicitly add it to your mesh. And it’s always online.
There are yet some other perspectives to look at the Mesh I plan to blog about in the future.
Check out Live Mech technology preview and stay tuned...
- Roman
This posting is provided "AS IS" with no warranties, and confers no rights.
Technorati Tags: LiveMesh
A while ago, when .NetCF v1 wasn't released yet, we could only guess what kind of applications our customers would build. People on the team were coming up with various ideas of mobile applications they would like to build on top of our platform and try to figure out which platform features would be most valuable. We had application building days, when everyone on the team would go and work only on their personal application projects.
One of the really simple ideas that came up then and remained unimplemented was the "Lunch Launcher". Many folks on the team prefer local restaurants to Microsoft cafeteria. Getting several people to go out to the same place often requires lots of coordination over the phone or e-mail, especially when people are away from their offices, in different buildings. Usually someone wants Indian food, someone had Indian three days in a row and wants to go for Mexican and someone else have strong cravings for a slice of bad pizza...
As most people on the team have some sort of Windows Mobile device connected to WiFi or mobile carrier network, we thought it would be nice to have an application that allows one person to send a lunch invite with a list of a few good local restaurants to other people and let them vote where to go to. When the poll is finished, its' results (the place of choice) are communicated back to everyone and everyone hopefully is heading to the same place. Sounds trivial, doesn't it?
However there are a couple of problems that make this application idea really hard to implement using existing connectivity technologies. First of all, how do I address someone else's device? Device IP address is not stable and is not discoverable... This makes "push" scenarios and true P2P not feasible. Second, devices lose connectivity all the time and someone may be off-line when the message is sent. How do I make sure my message reaches the other person's device, if we may not be on-line at the same time?
Interesting enough, simple, well-known technologies exist today that can help solve both of these problems. Every connected Windows Mobile device can send and receive e-mail, i.e. it has a stable, discoverable address associated with it - e-mail address. E-mail also works pretty well in the on-line/off-line scenarios (occasional connectivity). So, at some point Mike Zintel started promoting an idea of using e-mail as a web service transport and putting a programming model on top of it. Hmm, e-mail!? I was really skeptical about this at first, as probably many of you are now. But let me ask you this - how many times did you use e-mail to transfer some data from one computer to another? I find myself doing this far too often. For example, when I need to get some document from my machine at work to read it later at home - I send it as an e-mail attachment to myself. If I want to get a couple of pictures from my home computer to my work machine - I send myself an e-mail. I even use e-mail to get documents from my desktop machine to my laptop, when my laptop is off-line. It's quicker than sharing the file, turning another machine on, going to the share and copying it over. Sure, there are some limitations. But there are a number of scenarios when it comes pretty handy. Now, you may ask what does it have to do with .NetCF and web services and how does this help application developers?
Well, .NetCF team has been very busy since we shipped CF 2.0. A number of exciting things are happening here. Earlier Mike gave away some "hints" about what's coming: "the NET CF team is working on WCF subset for the next release, with some interesting support for transiently connected networks." I think it's time to elaborate on what that really means.
We are bringing a subset of WCF (Windows Communication Foundation, formerly known as Indigo) to devices. WCF provides a new unified programming model for building connected applications with managed code. The cool thing I like the most about WCF is its' extensible channel architecture. It allows building applications that can work on top of completely different transports and protocol channel stacks. Using WCF programming model, now I can send and receive data in my application equally easily using HTTP, TCP or e-mail. In many cases I can maintain identical or very similar communication logic code - sending and receiving the data works on top of various channels (only the initialization part will be different, as different channel stacks need to be constructed).
WCF is built around the concept of a Message object. The structure of a Message object loosely represents a SOAP envelope, and consists of two distinct parts: the message's body and an extensible collection of headers. The body is application-defined data, and headers are added/processed by infrastructure (and can also be used by the application). It is this structured primitive which is the core of WCF power (and which departs from the classic networking primitive of a byte array). It allows for extensible interoperable protocols to be built, and combined together in different ways. (See the following blog entry for more details).
The WCF programming model support will be limited to channel layer messaging (no service model support on the device) which means your program will be sending and receiving messages and it will be up to the application to correlate messages with each other (no RPC semantics). However, we are working on a custom proxy generation tool (along the lines with SvcUtil) that would allow consuming WCF services running on the desktop by the messaging client on the device without the need to modify the public-facing service.
Transport Channels perform the actual send/receive of the Message to a network resource. Layered Channels perform a function based on the Message passed in, and then delegate further modification and transmission to other channels in the channel stack. Some examples of Layered Channels include Protocol Channels that use Message headers and infrastructure Messages to establish a higher-level protocol (such as WS-Security). .NetCF will support a subset of WS-Security to enable message-level security and mutual certificate authentication.
We'll expose all necessary extensibility points so people could plug in their own transport and layered channels, while maintaining a unified programming model. We'll deliver a couple of transport channels out of the box, but we expect the community to fill in the gaps and build a variety of custom channels. The channels we plan to deliver out of the box are Http and... yes, e-mail (for WCF on desktop and device). Our friends from WCF team played around with building custom e-mail transport channel before and proved that it's feasible. We decided to take this experiment one step furhter to address concrete problems mobile application devlopers are facing today.
Thanks to the e-mail infrastructure, now you can host a service app on any machine or device that can reach your e-mail server (Exchange). It provides a totally symmetric communication model. Exchange server acts as a relay that allows communicating between devices that aren't on-line at the same time. We built a demultiplexing scheme that allows multiple applications to share the same e-mail account (the only limitation - multiple instances of the same application sharing the same e-mail account can't be on-line at the same time. Think of this as a network port sharing problem). This works through firewall/NAT boundaries. And with AirSync "Always Up To Date" technology, it should allow prompt message delivery once network connectivity is available. We are going to leverage powerful Exchange 12 web services under the hood on the desktop side Thanks to WCF layered architecture, channel implementation details will be hidden from application developers.We are working with Exchange server team to make overall user experience nicer and enable separation of regualr e-mail traffic from "service" e-mail traffic.
So, finally we were able to build a "Lunch Launcher" app. We also built some interesting P2P casual games (like Poker), that tolerate the high latency of the transport. We see this channel as a powerful notification mechanism for mobile LOB applications - now you can easily "push" important notification messages to occasionally connected device on the mobile network. For example, a Mobile Sales application could easily push information about an item recall or inventory change to all devices in the field. Even if some of them are off-line, they'll pick up the message once the network becomes available:
I can also see this channel used as a light-weight notification mechanism to trigger more heavy weight data sync (through SQL Everywhere / Sql merge/replication or RDA) and avoid expensive polling.
We don't expect people to always use e-mail as a primary transport channel, if a good stable network connection is available. However, in many cases it can be used as a backup. Thanks to the unified programming model, switching from one transport channel to another is fairly easy. The e-mail channel will provide an off-line message store on the device, so application developers won't need to implement complex logic to detect network connectivity. Thus app developers can just call Send() and forget about the message - it's placed into the off-line queue (Outbox) and gets physically sent out when it becomes possible.
We'll leave it up to you to come up with more cool app ideas. And we'd love to hear from you what kind of mobile WCF applications you'd like to build. As usual, your feedback is very important to us.
Stay tuned...
- Roman
This posting is provided "AS IS" with no warranties, and confers no rights.
Parametric polymorphism (aka Generics) is one of the most interesting new features added to .NET CLR and languages in version 2.0 (Whidbey). There are many good articles describing the feature and its’ implementation for the full .NET CLR. For those who interested in the design fundamentals of generics I’d strongly recommend the article by Don Syme and Andrew Kennedy from Microsoft Research, who originally designed and implemented .NET generics.
Many people were asking what .NET Compact Framework generics story is. Although generics were not included into our Whidbey Beta 1 release, I’m very excited to confirm that .NET Compact Framework version 2.0 will also include generics support. Some people may have already had a chance to play around with .NET Compact Framework generics (latest Visual Studio 2005 Community Technical Preview Releases are available to MSDN subscribers).
Here is a quick introduction into the feature from Seth Demsey.
As I was actively involved into generics work for a past year or so, I’d like to share some details about specifics of .NET Compact Framework approach to generics support, underline some limitations and differences from the full .NET CLR approach.
What to expect?
First of all, it’s important to understand that generics implementation goes deep into the CLR. Unlike C++ templates, generics is not just a compile-time concept, it’s an integral part of the type system, reflection, debugging and so on.
Our main goal for this release is providing core language feature compatibility with C# and VB. Generics quickly gain popularity in the .Net community, customers love them. So, it became pretty clear that we couldn’t ship the product without generics support. As usual, we had to make certain trade-offs to minimize the overall impact on the rest of the runtime. We identified and targeted the subset of generics functionality which is absolutely necessary for C# and VB runtime support. We don’t position generics as a “performance feature” for this release, as typical usage patterns remains largely unknown. We’d definitely benefit from the customer feedback on generics usage in .NET Compact Framework apps, which would allow us to introduce targeted optimizations.
That said (I hope I set everyone’s expectations accordingly), the results of our work look pretty promising. We were able to provide a fairly complete generics story which includes:
- C#/VB language and runtime compatibility in regards to generics, with only a few limitations listed below. Both generic types and methods are supported. All kinds of constraints are accepted (see note on constraints in the next chapter).
- Full port of Base Class Library generic collections classes, Nullable<T>, functional programming elements (delegate-based APIs), etc.
- Limited reflection support. We only implemented a subset of reflection functionality for generics, which is needed by C#/VB runtime.
- Full debugger support.
Limitations
- Expansive generic recursion through fields is not supported (for both value types and reference types):
class Foo<T>
{
Foo<Foo<T>> field;
}
As far as I can tell, this pattern does not have any practical use. This will compile (if Foo is a reference type), but will result in a load time exception against .NET Compact Framework.
To disallow this pattern and avoid infinite loading sequence, we impose a limit of 1024 unique closed constructed types per generic type declaration. Other kinds of generic recursion, which are commonly used in practice, are fully supported by .NET Compact Framework, such as:
class C : IComparable<C>
class C<T> : IComparable<C<T>>
- No support for variance modifiers. Though variance/co-variance forms part of the overall ECMA spec for generics, and is implemented for the full .NET CLR, it is not used in Base Class Library or by C# and VB.
- .NET Compact Framework does not perform IL verification and metadata validation on device. Thus, we don’t validate constraints for normal execution (we assume constraints are relevant to verification). So, in the absence of verifier, constraints will not be validated by the loader, except for reflections scenarios. Reflection will still verify the constraints when binding to generic parameters. This means that all kinds of constraints can be used in the code; they will be validated at the compile time, but not at the runtime (except for reflection scenarios).
- To minimize the working set impact, we may limit the number of type parameters for generic types/methods to 8. Please note that C# allows type parameters from an enclosing class to be used in a nested class, but compiles it away by adding extra type parameters to the nested class definition. For example the following C# generic type declaration:
class Foo<T, U>
{
class Bar<V> {}
}
will be transformed by the compiler into:
Foo`2<T,U>
| .class public auto ansi beforefieldinit
| Bar`1<T,U,V>
| | .class nested public auto ansi beforefieldinit
| |___
|___
Thus the number of type parameters of the nested type includes the numbers of type parameters of all enclosing types. This should not exceed the limit either.
- Reflection functionality is limited for generics, in particular for generic type or method declarations, open (uninstantiated) generic types/methods and formal type parameters. Examples of unsupported functionality for open types:
Type.GetField, Type.GetMember Type.GetMethod, Type.GetProperty , etc. But in any case, there is not much you can do with a member of an open type, even if you could reflect on it. Using reflection methods, which are presently in .NET CF Base Class Library, but don’t work with generics will result in NotSupportedException being thrown at the runtime.
.NET Compact Framework team strives to improve ECMA standard compliance of our product with every new release. As generics were introduced into the standard, we provided a comprehensive generics support in .NET Compact Framework v2 product. From the runtime perspective .NET Compact Framework supports most of the generics features, specified in the standard with only a few minor limitations listed above. Please note that these limitations were introduced only due to time and resource constraints for this particular release of the product, based on requirements prioritization. We’ll continue to work to fill these gaps in the future releases of our product.
Implementation insight
There are a few details about internal implementation of generics in .NET Compact Framework, which may be valuable to know to use generics more efficiently in your application.
There are a couple of approaches we could have taken to introduce generics into the product:
- Representation and code sharing. This implies sharing of execution engine data structures and JITed code between different instantiations of generic types/methods. We were considering boxing all valuetype type arguments and thus uniformly sharing JITed code (generating only one copy of the code). However, this approach requires a fare amount of JIT work to transform IL-sequences and does not make life much easier after all. It also introduces a perf hit, due to boxing/unboxing overhead for valuetypes, which we are trying to avoid in the first place. This approach would undermine one of the main competitive advantages of .Net generics against Java generics – the execution efficiency, as Java compiler substitutes type parameters with Object, resulting in boxing/unboxing overhead.
- Representation and code specialization – each distinct instantiation of generic type results in a separate execution engine data representation and JITed code specific to that instantiation. The CLI does not support partial instantiation of generic types and generic types may not appear uninstantiated anywhere in metadata signature blobs. Considering this fact, fully instantiated (closed) generic types and methods, are not very different from concrete types/method and can be represented by existing internal data structures with minimal modifications. This implementation can be improved in the future to enable some degree of JITed code sharing.
Full CLR’s approach is a combination of sharing and specialization. Data structures for different instantiations of generic types/methods are duplicated, but they share JITed code between compatible instantiations, where possible.
.NET Compact Framework implementation is going to be fully specialized for this release. What does this mean for developers? If I have a generic type declaration:
class Foo<T>
{
public static T field;
public void bar(){...};
public static void baz(T arg) {…};
}
and somewhere in my code I create 4 different instantiations of this generic type:
Foo<int> f1 = new Foo<int>();
Foo<double> f2 = new Foo<double>();
Foo<MyType> f3 = new Foo<MyType>();
Foo<Object> f4 = new Foo<Object>();
4 different internal execution engine data structures will be created in memory. It’s the same as loading 4 completely different types from the working set perspective. Note that this has nothing to do with the number of objects created – only the number of different unique closed constructed types loaded matters. As with regular, non-generic types, I don’t have to create an object to trigger loading (creating the internal execution engine data structure of the generic type), it’s sufficient to reference one in the code, for example by accessing a static member (Foo<int>.baz(5) or Foo<int>.field = 5).
Now, if I invoked method bar() or baz() at least once for each closed constructed type, it would be JITed 4 times and 4 potentially different pieces of JITed code will live in memory – it doesn’t matter if type arguments are value types, reference types or would be considered compatible by full .NET CLR.
The same is true for generic methods:
class Foo
{
public void bar<T>(T arg){…};
public static void baz<T>(T arg) {…};
}
The first time closed constructed generic method is referenced in the code, unique internal execution engine data structure is created to represent it. So, if I have code like this:
Foo.baz<int>(5);
Foo.baz<string>(“Hello”);
Foo.baz<MyClass>(new MyClass());
3 internal data structures get created, method baz() will be JITed 3 times and 3 instances of the JITed code will live in memory.
Please refer to Mike Zintel’s article for more details on .Net Compact Framework Memory Management.
Thus, when using generics one needs to be aware of the potential JITed code size impact. If there is very large number of closed constructed types/methods per generics type/method definition, app may start experiencing a JITed code size pressure. On the positive side, specialized JITed code is typically more efficient as exact type parameter information is always easily accessible.
This posting is provided "AS IS" with no warranties, and confers no rights.
