MikeZintel's WebLog

Why Sync?

MikeZintel — Tue, 06 May 2008 06:48:00 GMT

For several decades after the invention of the computer, the dominant constraint was processor cost. While mainframes had, in relative terms, impressive storage and local I/O capacity, they were primarily designed to do useful work on every clock cycle. Indeed the humans who were running around mounting and unmounting tapes according to instructions in a job control stream were trying to keep the processor from ever executing a single very expensive NOP.

It was very Tron.

With the advent of minicomputers, computers became much more useful to people. The economics of cheaper processors and storage meant that a department level manager could buy a computer rather than waiting for access to corporate iron. Multi-tasking operating systems, in conjunction with terminals – the precursors to networks, began to move from a technology designed to keep the machine busy to a technology that was enabling interactivity in the human computing experience. The economics still required people to share a processor and storage (and experts).

Workstations were an early version of what would eventually become the personal computer. The workstation was designed exclusively to make the individuals using them as personally productive as possible, by supplying them with processing, graphics, local storage and then network storage. The magic of the workstation was that the era of optimizing for the machine was coming to an end. I believe that is was the productivity of the workstation that enabled developers to experiment and refine networking and its associated metaphors and practices (virtualized peripherals, remote shells, RPC, collaborative / distributed development, etc.) that paved the way for the Internet itself.

The personal computer again fundamentally changed the economics of computing. While the first PCs were a step backwards in functionality relative to the workstation, the price was right and getting more right with each passing year. The first 10 years of the personal computer were driven by the novelty of rapidly cheaper computing and the search for uses for it. Later the PC would combine workstation class operating system sophistication with a TCP/IP stack and affordability and give birth to the modern Internet.

It’s interesting to consider that it was roughly 15 years ago that we were debating the merits of adding, or not adding, TCP/IP and Winsock to Windows 95.

The next opportunity that affordable personal computing created was access to information. The industry got a few years out of enabling people to type their recipes in WYSIWYG and then print them on newly affordable “graphics” printers. But we wanted more. We wanted to know things. We didn’t know it at the time, but there was a pent up demand to publish. Encyclopedias, universities, newspapers and television were still in control of knowledge and its distribution.

The next economic constraint was the cost of storage. I remember when we were generally happy when a new operating system install took less than one quarter of the hard drive, and data loss due to drive failure was an ordinary part of the personal computing experience. Thankfully the term head crash has disappeared from our jargon completely.

CD-ROMs started to show the promise of cheap storage. Usenet was showing us that people really wanted to be heard. And email, the power of communication and collaboration within an intimate group controlled by individuals.

For 10 years between 1990 and 2000, the capacity and reliability of everything associated with the PC increased at an amazing and increasing rate, even as it got cheaper at an increasing rate. By 1998 I had so much affordable, fast local storage and dirt cheap processor cycles that I could render digital video in real time. At the time it wasn’t obvious, exactly, why this was better than an analog tape but damn it was way cool. PVRs came later and showed us one reason it was cool.

“Overnight”, the default directions of everything in the industry started to change.

By 2002, we were beginning to take high capacity, small and mechanically robust laptop hard drives for granted. Affordable desktop PCs had storage capacity and reliability undreamed of by workstations in their prime. Video games could assume large amounts of space. Solid state storage was enabling portable devices to store music, photos and video. Indeed the storage wave just kept going even as the clock speed wave crested and began to roll back. This was not conventional wisdom 10 years ago.

Of course the biggest trend of the last 10 years has been the browser. The web march has been stunning: from simple access to a huge, barely “administrated” and human friendly namespace; to forms, catalogs and ecommerce; simple publishing via hosted services and rich client authoring; even more simple (if less rich) publishing via blogs and wikis; to highly interactive (but still not very rich) sites that are the anchor point of sharing, collaboration, publishing and discussion.

The web has received a lot of well deserved focus on ease of use enabled by a simple and clever networking layer and presentation layer. Indeed the browser sets a new standard for it just works experiences, and technology that doesn’t meet that bar will not be adopted. As is often the case, the reality is that considerable effort on the part of developers goes into making the experience work and feel great on top of a platform that has, as we say, evolved over time.

The evolution of (cheap) storage has had an equally important impact on the web. The combination of the natural partitioning flexibility of the URL space, racks of (cheap) PCs and simple software redundancy have enabled the creation of immense reliable enough stores. Because these stores are “centralized” it is feasible to do data mining, indexing and transformations on them. This is the engine behind search engines. As this technology becomes more approachable to more developers, it is powering more experiences on the web.

It’s tempting to assume that web client server wins everything. But I don’t think so.

I love the web. I’m using it now. But I still spend more time using Windows applications. At the moment I’m listening to very high quality glitch free music being streamed from my hard drive, through an iTunes client and into my headphones. I have a few folders open where I did some photo editing with Photoshop earlier today. I’m using Office to author this blog and enjoying best in class spelling and grammar checking. Messenger is open. I routinely use Premier, Lightroom, Dreamweaver, Nero, and various scanning programs.

The industry’s newest, coolest, ease of use standard setting, cleverest mobile browser ever powered device, the iPhone, has a local application for text messaging, calendar, photo album, camera, YouTube, stocks, maps, weather, clock, calculator, notes, mail, music, the phone itself and of course, the shell. The reason for this is that local applications can have experiences that are highly tuned to the device capabilities and the preferences of the customers using them.

Of course what is really happening it that the industry is moving toward the best of both worlds. Many of the local applications I describe above are invisibility connected to the web. This is nature of what we’ve been calling this Software + Services.

What doesn’t seem to have happened yet is much coherency when I want an experience that spans mobile, web and PC. Mobile and (typically one) PC work together over USB sync, and Facebook has done a decent job with their iPhone targeted (browser) client but for the most part, I have to “pick” an app or a site or an operator, use it and accept another silo of (my) information. But since lots of them are truly unique and useful, I use lots of them. And I learn to context switch. A lot. I am forced to maintain a complex mental map of where my data is, how I get to it and how to move it around.

Even the web, “the alternative to complexity”, is becoming complex. Sites are competing to store your information and as a side effect they aren’t motivated to allow you to store it across sites. I don’t think I could find all of the photos I have uploaded to the web if I wanted to.

I think that the Live Desktop and Windows Mesh client experience (and more, coming …) are a visceral demonstration of the power of having a coherent (synchronized) view of all of your information from every endpoint. Of course, these are just the initial experiences we have built on Live Mesh Platform. I can imagine many more.

The success of Live Mesh depends in large part on the core platform and the developers being successful, in the real world, to build “it just works” experiences.

Live Mesh was my first deep exposure to REST. I have to admit that the power of simple operations against an URL space has impressed me. We have tackled very complex problems using the model and it has held up very well. I’m happy to tell you that this elegance will extend to the developer model (soon).

I’m looking forward to it.

Mike Zintel

Product Unit Manager, Mesh and Storage Platform

I want to thank a few people: Dave and Amit for creating the opportunity to work on this. Ray, Jack, George and Colleen for their deep insight. Abhay and his team for making it cool and showing me what is possible.

Abolade, Alex, Dave, David, Mike, Tom, Tony, Vlad and many other people on my team for designing and building the right stuff long before I knew what it was. Todd and his team for making it work on the Internet.

We’re not done yet and we’re hiring. mikezin@microsoft.com

Technorati Tags: LiveMesh

Microsoft XNA Framework Referenced Here, Etc.

MikeZintel — Thu, 06 Apr 2006 08:47:00 GMT

Scott Holden mentioned that I didn't reference the Microsoft® XNA Framework® in my previous posts crawl this one too, thus missing out on the search engine action. Actually it was a secret test to gauge the degree to which search algorithms can infer based on semantics.

Still pretty bad.

So, let me fix this by sharing a photo of Chris Muench, at GDC, showing off his game, written using the Microsoft® XNA Framework®.

We're going to give an internal demo of these games at an upcoming "all hands" Developer Division meeting. It's cool.

We're getting ready to release NET CF V2 SP1 to beta. We're finally close to making good on our commitments to support for Windows CE 4.2, performance tools and headless support. Thanks for your patience.

In other news, the SQL folks are going to announce today a product called Microsoft® SQL Everywhere®. This is a compact local database for use on PCs and Windows Mobile devices. It will enable developers to use the same database and APIs to develop occasionally-connected applications that sync with SQL Server 2005 across all Microsoft client platforms. The database will have similar install footprint as SQL-Mobile does today. Very similar. Very familiar.

Some folks have been asking about this for a long time.

The NET CF team is working on WCF support for the next release, with some interesting support for transiently connected networks.

I got an interesting answer to an interview question a few weeks ago. "Why do you want to work at Microsoft?" The candidate described the anti-Microsoft buzz on campus. He then said he listened to a presentation that a Microsoft employee gave at campus, and decided to "put some independent thought into this". I have to admit a certain guilty pleasure to reading mini-microsoft; it’s like an accident scene. But I hope people can see through this stuff. There are a lot of great folks who care here. I've been here 11 years and I think it's a better company now then during the "good old days

Mike.

This posting is provided "AS IS" with no warranties, and confers no rights.

It's Cool.

MikeZintel — Fri, 24 Mar 2006 04:14:00 GMT

I have a theory that everyone gets into software to write video games or build robots. Many discover that it's hard, they don't "got it", most of the paying jobs are for accounts payable systems and that rent costs money. I have another theory that photographers are just creative people with no drawing talent. I'll admit to having no real data on this, other than it being true for me personally.

One good solution to the art problem is Photoshop, and for the game problem to build tools and entice the folks who "got it" to use them. Frankly seeing people with different/better skills than you build very cool stuff on your tools in the best part about working on developer tools.

About 2 years ago I challenged my team to make sure that as D3DM (DX for Windows Mobile) got rolled out on devices, NET CF was the best choice for writing a game app on the platform. Success would be measured by adoption and great customer feedback. At last year's MEDC, D3DM was highlighted in a keynote with the game, Pocket-Jongg, written by Chris Muench. Chris also gave a talk at that event on the details of writing D3DM games in managed code.

That was fun to watch.

Now the challenge is to make sure that once hard core C++ game developers try .NET, that they leave delighted. This will be a learning experience for all of us.

I've been working on operating system level stuff for long enough that I can get excited by seeing the right text on a screen (although, never by a single light, even if it blinks; can't do hardware). So I thought I was really excited when I saw a NET CF team home brew game, "Racer", running on Windows Mobile, Xbox and a Windows PC.

But then I saw Chris's game, with some help from artists in the Xbox team, on Xbox.

It was cool.

And then I saw this.

And it was cool.

Accounts payable never looked this good.

Check out the news from our embedded reporter, Rob.

Mike.

This posting is provided "AS IS" with no warranties, and confers no rights.

A Market Within A Market (was .NET in HD)

MikeZintel — Tue, 14 Mar 2006 07:37:00 GMT

In the late 90s, I worked on the IrDA infrared connectivity technology. My team built the protocol stacks that shipped with Windows CE and Windows 2000 and descendents, and I was an advocate of the technology within Microsoft.

IrDA is a success. It continues to be useful as simple way to transfer files and contacts between devices and PCs, and to use phones as modems.

That said, the thing I was most passionate about failed. I wanted to create the most seamless experience for moving photos between digital cameras and PCs. Keep in mind that 8 years ago, it was trivial to draw a crowd by taking a digital camera to the zoo. And my internal demo of a pre-production digital camera beaming images to Windows NT5 was very cool indeed.

There was a “competing” internal technology called Windows Image Acquisition (WIA) architecture. WIA abstracted various lower level buses, normalized the command and data transfer, and exposed the camera photos through a standard API that can be used by many different programs. In theory, WIA and IrDA would coexist, but I disliked it. A WIA camera was installed via the PnP system as a new device, and controlled via APIs and control panel.

I argued that we had created an essentially new model of connectivity, completely free of state and configuration and completely ad-hoc. No drivers to install, no settings to get wrong, and wrapping this in PnP was ugly from an engineering perspective and bad for ease of use.

I was quite passionate about this, and the company empowered me to ship it in Windows 2000. I even lined up some camera vendors to support it. And I was sure I would win in the marketplace because of the fundamental beauty of what we had built.

It was beautiful.

I don’t know for sure, but I suspect that, today, if my one unit of the one camera model that I know actually shipped with this feature wasn’t broken, I could point it at my XP laptop, hit send, and all of my photos would be transferred to a folder. The folder would then automatically open to an Explorer photo preview model. This might even still work in Vista.

Slick as can be.

But it failed. Worse, partners who bet on me failed. I didn’t make life easier for consumers. I left Windows with legacy (but good quality) code to support. And I now am a regular and happy user of WIA to import images from one of my two scanners into any number of image editing tools that I use on a regular basis

Why am I telling you this story?

The software platform business is fundamentally different from an innovation and marketplace perspective than, say, the 3M model of product incubation, entrance, success or failure and withdrawal (at least as it’s taught to me by B-School types; I’ve never worked at 3M and one thing I’ve learned is that you never quite get it from looking outside in).

First, you can’t really withdraw things. At the scale of Windows, even if a small percentage of folks take a dependency on a technology, then you want to continue to support them.

For a long time.

DOS apps I wrote in 1980 continue to run on XP today. I’m sure if the folks writing DOS knew this at the time, they would have made this task a bit easier.

.MODEL LARGE ; large model

.CODE

PUBLIC _TestForKey ; BOOLEAN TestForKey(void)

_TestForKey PROC

mov ah, 1 ; Key waiting check

int 16h ; Keyboard BIOS service

mov ax, 0 ; Assume no key waiting

jz Return ; correct, return(FALSE)

mov ax, 1 ; Key waiting, return(TRUE)

Return:

ret

_TestForKey ENDP

It's important to get things right for the long term.

Second, the net value to people of a feature is not measured in isolation. Frankly I don’t want to choose between 10 different ways to do the same task and decide which one I like best. I want one way to discover and start programs, one way to cut and paste, one way to organize things, one way to add and manage devices and administrate my machine, etc. Once I’ve learned how to do things, I want to do them the same way in as many places as possible. Perhaps most importantly, I want programs and hardware I buy anywhere to work anywhere.

You might think we could have done a better job on some of these things, and I’d agree, but it’s not for lack of trying. Our software is used, literally, by hundreds of millions of people, in every country in the world, to do an incomprehensibly large number of different things. An interesting consequence of listening and acting on a huge number of feature requests, each that make sense in a scenario, is that you can accidentally end up doing a bunch of overlapping things. Frankly the search for the fully normalized horizontal, discoverable and easy to learn and live with solution to problems is not an easy task.

Some of use who have been around for a while remember a simple happy time when you wrote some code, put in on floppy, took out an ad in Byte, and made a fortune. Personally, I’m going to ignore the fact that this wasn’t at all my experience trying to bootstrap a software company, and remember it this way.

Why am I telling you this story?

I’ve been spending the last few months doing “cross group” work, to get something new lined up and right for the long term.

We spend a lot of energy and time at Microsoft these days doing cross group work. On a great day, this can be an enlightening and enjoyable process, as we have an amazingly diverse, talented, curious and interesting group of folks here. I can’t imagine a faster way to gain rich insights into technology, customers and the marketplace. On a grumpy day, cross group work feels like a game of whack-a-mole that has gone on too long, as the shear number of opinions and innate complexity can be overwhelming.

And this work has been “secret”. Not so much because we want to save the surprise, but because we want to make sure that we don’t announce things until we have very high confidence that we will be able to build, ship and support them, that they are the right solutions and that we understand how they sit relative to other potentially overlapping efforts.

I once wrote about what we knew Vs what we didn’t know when we designed the .NET Compact Framework.

I’d like to tell you now what I know about the game space. I know that very high end games are written in C++ , with a significant art component, and are getting more expensive to build. I know that C# improves programmer productivity substantially and has very solid adoption and customer loyalty. I know that at the low end, various technologies are making “casual” game authoring much easier, and that devices are a high volume target for these games. I know that the set of technologies used to write games varies significantly across platforms, and that it’s hard to migrate code and skills between consoles, PC and devices.

I also know that working with the Xbox team has been among the most enjoyable and productive cross group work that I’ve done. I know that my team and the XNA team within Xbox, have been burning the midnight oil to allow us to demonstrate the feasibility of games written in managed code running on a .NET CLR on a final 360 dev kit. And I know that we’ve demonstrated the same game binary (almost the same; oh so close) running on the 360 kit, Windows and on Windows Mobile.

In HD.

Fast.

Stay tuned.

For the record, the right solution to the photo exchange problem proved to be tiny solid state drives that are PnP’d as a local filesystem.

Mike.

This posting is provided "AS IS" with no warranties, and confers no rights.

It's a Completely Cool, Multi-Purpose Blog

MikeZintel — Thu, 03 Nov 2005 08:44:00 GMT

I recently watched, for about the 10th time, the movie True Stories. And, for about the 10th time, I started telling anyone around me who would listen about this great movie.

I find it fascinating that different folks see different themes in the same movie. Indeed, depending on my mood, I can see different movies.

True Stories is about small towns, suburbia, religion, the technology business, metal buildings, community, shopping malls, love and marriage, progress, music, other stuff. The movie cuts between David Byrne's monolog narration, as he drives through a town in Texas, lots of lip-syncing to music by the Talking Heads, and various snippets of small town life.

Most of the characters in the movie are eccentric. This includes a man who advertises for a wife on television (played by John Goodman), an executive who uses food as a metaphor to explain the changes that technology will have on society, a paranoid preacher hooked on conspiracy theories and a pathological liar.

It's pretty clear during the clip of "Love for Sale ", that Byrne is being condescending to advertisers and people gullible enough to fall for advertising. Yet despite the fact that the story is basically about the boring lives of bunch of oddballs, I find a more subtle and optimistic message intertwined.

The movie's viewpoint on impact of technology on society is particularly forward looking. The almost unbelievable global change driven by the wide availability of PC computing escaped Bryne, who was commenting on the impact that a semiconductor company was having transforming the local community. Yet the exaggeration, intended to be a comical spoof, has proven to be shy of reality.

Small town life really is boring. And folks who choose to live in small towns are generally isolated enough from urban trends so as to appear unsophisticated and even intolerant. But my small town friends are not lacking in intelligence, creativity or ambition. The need to self-entertain leads to a lot of eccentric hobbies and a sensitivity and openness to simple pleasures. Most folks get that talking about the weather isn't intellectually groundbreaking, but value having a collectively low barrier to entry to basic human interaction. As far as open mindedness goes, my experience has been that in a big town, you advertise your tolerance with a parade, and in small town, parades are not a political events and privacy is valued and respected.

The fundamental gag in True Stories is that Texas is flat, ugly and boring. Indeed the closing scene is a girl amusing herself with a sequence of different walks as she travels down a straight, seemingly endless road through the prairie. And the strange characters are a result of life in a boring, isolated Texas town. But they all seem sincere, decent, complex and genuinely entertaining.

I liked them.

Mike.

This posting is provided "AS IS" with no warranties, and confers no rights.

The Proper Division of Labor in an Engineering Org

MikeZintel — Thu, 20 Oct 2005 08:35:00 GMT

My job: blogging.

These folks: real work (ie setup, versioning, JIT, GC, loader, reflection, debugging, type system, interop, BCL, globalization, Windows Forms, DirectX, networking, crypto, ADO.NET, XML, etc).

Mike.

This posting is provided "AS IS" with no warranties, and confers no rights.

QA Has Signed Off on .NET Compact Framework V2

MikeZintel — Fri, 30 Sep 2005 02:29:00 GMT

The .NET Compact Framework Quality Assurance Team signed off on V2 yesterday. We've been working on this for a long time and I'm excited to get this thing shipped. Soon.

I'd like to share some insight into what we've been doing for the last 3 years, and why.

The first half of the project cycle is divided into milestones, each about 3 months long, split between a coding and a stabilization phase. Like most of the techniques used to manage multi-person software projects, the goal of milestones is to find and fix bugs early and keep the code in good health.

If I check in a code change that introduces a subtle bug (pretend that the team would actually let me check in code these days) on Monday, and on Wednesday I get a bug report on an apparently different bug, there's a decent chance I'll be able to remember what I did, and make the mental connection, and correctly fix the root cause. The probability of a correct diagnosis is far lower two months later. And if other folks have "fixed" the side effects in the interim, possibly by unknowingly taking a dependency on the incorrect behavior caused by my bug, fixing the root cause then creates a bug ripple through the system. When you work on systems with 10s, 100s or even 1000s of developers, you have to pay attention to this.

Small projects tend to behave like clay models. Changes are predictable and visible. Large systems behave more like a building or even a living organism. Each step of building construction establishes a quality baseline for the next one. A crooked foundation ripples through the process until the kitchen cabinets can't be installed without a gap on one side or the other. The cost of moving water pipes during a remodel vastly exceeds the cost to run them initially, it's often to impossible to move things like chimneys and drains and you can add only so many new plugs to a circuit.

I think about the code of a complex software project in terms of its "health" at any given point in time. It's much cheaper to keep it healthy than is it is to cure it, and after recovery from a major illness, accident or surgery, it often doesn't come back to optimal health.

There's also a team efficiency and morale reason to keep the system healthy. If you are trying to debug complex behavior in your code, and the components that you are running on and with are also unstable, the amount of complexity soon becomes overwhelming. If this continues for very long, folks tend to give up on the really hard bugs, since they can't find them and there's a good chance they might just "go away" in some future build anyway. This phenomena extends to external customer satisfaction too. It only takes a handful of memory leaks, unexplained crashes and hangs, incorrect program execution or data corruption before folks assume that all of their bugs are really system bugs. We hold ourselves to high standard around these kinds of things.

So milestones are a simple tool to make sure if the code goes on a bad drinking binge, we can send it to rehab.

Sometime around the middle of the cycle, the system becomes stable and complete enough that you can start to collect feedback. If you do this too early, the feedback tends to be obsolete and customers get frustrated because the system is still churning. If you wait too long, you can't act on it. We collect feedback through our early adopter programs, beta tests, MVPs, newsgroups, internal app building exercises, MSDN forums and bug reporting, industry events, web searches and formal industry research.

During the second half of the multi-year cycle we start to pay a great deal more attention to our automation harnesses and associated dashboards. In terms of automation, we run a continuous build, with a "nag auto-mailer" that catches build breaks and sends email, ideally in time to fix them before the nightly automated build. Every day, a new build is created and minimal set of automated tests, called build verification tests, are run against a matrix of hardware and operating systems. The failures are logged, analyzed by engineers that day, and bugs filed. The entire test suite and automation needs the same kind of health program as the code under test. If the suite generates too many false positives it will be ignored, yet the coverage needs to be as good as possible to find bugs early.

We also have suites, automation and dashboards around stress and performance. The coverage on these needs to be broad, to prevent ongoing performance tuning (and the occasional foundation wall move) from "training" to a small set of benchmarks that don't translate to real world gains, and to squash nasty stress bugs early.

We have ship criteria around these dashboards and we watch these and bug trends to help figure out when we're ready to ship.

At the end of every milestone, and prior to final builds, we run what's called a full test pass. That is a process that runs all the tests on as many combinations of hardware and operating systems as we can muster. This is a multi-week process.

The thing we have been working hard on for the last few months is compatibility. We want customers to deploy the next version widely, and key here is to provide both enough new value and to make sure they have a pleasant upgrade experience. I think our performance gains alone will be reason to upgrade, in addition to the new APIs we expose. We have an automation harness that uses reflection against our assemblies to validate that we are in sync with the .NET Framework on Windows. We also run a compatibility test suite to validate that our behavior is compatible. Both of these are necessary starting points, but the only way to really succeed is to test real world applications.

To that end, we have assembled, with your help, a catalog of about 600 application. We've manually tested about half of them, covering a wide range of application types and complexity, and have fixed all but a handful of really tough problems. This has been the last dashboard that we've been watching closely and I'm happy that it says we're ready.

And you've been telling us that we're ready too. Thanks.

A few V2 stats:

Bugs opened, reviewed and fixed or closed : 11,831

Test executables run in final test pass: 546,940

Test cases run in final test pass: 3,548,056

Devices used in final test pass: 94

Global versions tested: CHS, CHT, ESP, FRA, ITA, JPN, KOR, PTB, USA

Apps tested for app compat: 326

Performance scenarous tested: 230

This is what we've been doing, with your help, for the last 3 years. I hope it delights you.

Mike.

This posting is provided "AS IS" with no warranties, and confers no rights.

WPF/e and the .NET Compact Framework

MikeZintel — Fri, 23 Sep 2005 07:12:00 GMT

I'd like to share some thoughts on integrating WPF/e, announced at the PDC, and the .NET Compact Framework.

WPF/e is a C++, cross platform implementation of a subset of WPF (Avalon). It will be able to execute XAML, in addition to exposing dual mode COM interfaces that make it programmable from COM and script languages.

This makes perfect sense for deployment in browsers, but less so for .NET developers on devices.

I've been struggling for a while trying to understand how to add a great, easy to author, graphics animation infrastructure to the .NET Compact Framework. We made good progress on enabling games with the DX object model in V2 (Whidbey), but there's still considerable investment needed with this approach to build a game. I'm equally concerned about the "disjoint silos" between Windows Forms apps and full screen only DX apps.

The implementation of WPF on the PC was too large to port to devices.

I'm optimistic that WPF/e is the solution to this. The next step for the team is to investigate the feasibility of integrating this technology deeply into the .NET Compact Framework. Ideally, this would enable the hosting of a control built in WPF/e in a Windows Forms application, in addition to enabling full screen WPF/e applications. In both cases, the programming experience would be .NET.

Mike.

This posting is provided "AS IS" with no warranties, and confers no rights.

Ignition System Envy

MikeZintel — Thu, 08 Sep 2005 08:02:00 GMT

I'm pretty excited to try out my new toy, the Zintel-O-Tron 6 Channel Combination Model Rocket and Fireworks Ignition System.

I built this thing on the weekend from parts I bought at the local Radio Shack.

At least I was excited, until Mark Prentice, who works with me on the .NET Compact Framework, basically told me that my design point was lame ...

From: Mark Prentice
Sent: Friday, September 02, 2005 1:28 PM
To: Mike Zintel

I bet using http://www.seetron.com/ssc.htm and a couple of hobby servos at http://www.hobby-lobby.com/servos.htm, a bit of wood and nails for a simple peg board switch, a PPC with serial out, one could use NETCF to write the fireworks scheduling app…

Mark

Mike.

This posting is provided "AS IS" with no warranties, and confers no rights.

This is Not a Technical Post on the .NET Compact Framework

MikeZintel — Fri, 26 Aug 2005 23:28:00 GMT

I've been on vacation.

If you're very lucky, you know a place that feels like it's the calm in the center of the universe. For me, Little River Lake, in White Rock, Nova Scotia is just such a place. We've spent time there every summer for about 20 years.

The highlight this year, beyond seeing friends and family again, was fireworks. The key to this was the discovery of Fireworks F/X, a few ~~miles~~ km down the road. Every summer I search for fireworks. And I've usually been able to find small, single shot units at the local hardware store. And for quite a few years, the locals have told me that I need to go to Grand Pre for the good stuff.

I stole this description of Grand Pre from a tourist site:

Adjacent to the town of Wolfville is Grand Pre, or "Great Meadow", stretches east from the Acadian dykes which excluded the tides of the Minas Basin. Grand Pre National Historic Park marks the site of the expulsion of the Acadians in 1755, and it has become a symbol of the preservation of the Acadian People.

For over three hundred years, the village of Grand Pre has provided a home for immigrant farmers and artisans. The first Acadian settlers traveled east from Champlain's original settlement in Annapolis Royal in 1680. This group of industrious farmers engineered the complex system of dykes along the shores of the Minas Basin and nurtured what would become the agricultural heartlands of Nova Scotia.

Today, Grand Pre is the home of a growing number of artisans as well as a National Park and a variety of tourist facilities.

Not the first place I expected to find a great selection of the "good stuff". Besides, I could never get more information than "it's in Grand Pre".

But this year Bob gave me a web site and a phone number. Soon, I had directions: "Turn right at the intersection and look for the sign." The place looks like a farm. And the folks there seem like typically friendly and helpful Nova Scotians, but their sentences are peppered with words like aerials, shock tubes, batteries, bomblettes, fountains, cones, strobes, firing system and cakes. And they have the good stuff.

And the more good stuff you buy, the cheaper it gets, so you can load up.

Cool.

So here was the plan. First, find a suitable rock in the center of the lake. In the daytime, take a big tub full of dirt and leave it there. Shuttle a cardboard box full of explosives, a "BernzOMatic ignition system", a fire extinguisher, a few flashlights and a camera and tripod to the rock.

At dusk, head back out ...

and wait for darkness.

Doug1 was in charge of ignition.

Doug2 was in charge of fire safety. In particular, he was responsible for ensuring that any burning pieces of space junk didn't fall into our rubber boat or cardboard box of fireworks.

Here he is, keeping a close watch on the boat and box of fireworks.

It took us a couple of tries to really get the logistics right. On the first night, we didn't even start until dark. That was the first mistake we corrected. Then we changed rocks from a small round one to a medium sized flat one. We also figured out a way to secure the boat line to a medium sized flat rock. Finally, after firing the first set of fireworks, we debated the need to soak the spent carcasses in water prior to loading them into the gasoline powered rubber boat. We debated this over a beer, on the far side of rock. When we turned around to leave, the pile of junk had spontaneously burst into a nice, but slightly smelly campfire. Clearly this was just another example of a scenario with watercraft, the darkness of night and explosives, made safer by beer.

The view up close is different, but very good.

In most of these photos, it looks reasonably bright on the rock. That's a result of the fact that I was using a combination of time exposures and a flash in the same image. It was pitch dark. In the photo below, the camera self timer tripped the flash just as Doug was putting some distance between himself and the fuse he had just lit. A second later we heard Doug hit the rock and mumble something about a blinding light (paraphrasing from Doug's exact words). Fortunately, Doug, the Maglight and the torch were not damaged.

I'm still struggling to understand how we could have made that scenario safer with beer.

We did pull off a few synchronized ignitions (a cake and a roman candle) with no injuries.

It was a blast.

As always, the hard part is saying goodbye. Thanks again folks for making us welcome.

I'm back to work now, and we're wrapping up V2 of the .NET Compact Framework and planning some cool new stuff for V3. Our backwards compatibility, stability and performance are all looking very good. I think you'll like it.

There is still great opportunity on the team for experienced and passionate systems levels people, with expertise in operating systems, compilers, graphics and/or networking.

Check out the video:

http://www.finalizer.net

Mike.

This posting is provided "AS IS" with no warranties, and confers no rights.

You can be fairly sure Microsoft is not advocating that you take any safety guidance from me.

Just Don't Do It

MikeZintel — Wed, 29 Jun 2005 08:48:00 GMT

I recently had the unenviable task of giving a presentation to the toughest of all audiences, Microsoft developers.

These are folks who, during the agenda slide, generally ask questions about why the framework is so slow, big, ugly, etc etc. I typically refer them to read some geeky blog posts and say things like "it's really just fine for the vast majority of applications".

This time I decided to use a bit of reverse psychology.

I opened with a story about how, in grade 5, the local fireman gave our class a fire safety talk. They explained that aerosol cans were dangerous. They showed us how to find out which aerosol cans were flammable by reading the warnings on the cans. Then, to drive the point home, they demonstrated that you can light the spray from one of these cans with a lighter and create a ~~stunning~~ dangerous flamethrower.

Then they told us not to do this.

In grade 5.

Mike.

This posting is provided "AS IS" with no warranties, and confers no rights.

From The Dept of Battlebots Built on .NET

MikeZintel — Tue, 07 Jun 2005 16:38:00 GMT

www.finalizer.net

Mike.

This posting is provided "AS IS" with no warranties, and confers no rights.

MEDC 2005

MikeZintel — Tue, 24 May 2005 06:52:00 GMT

I'm a bit slower on my MEDC blog than my peers. I'm still using my trusty Nikon F3 while my peers have moved on to some newfangled thing called the "digital camera". I had to wait for a week for the film to come back before I could spend 3 evenings scanning it. For the life of me, I can't understand what's so great about digital cameras.

One of the Las Vegas A/V crew spotted the Nikon, probably after hearing the signature clang phwaap of the mirror and shutter during otherwise quiet talks, and after exchanging the secret "film guys with old cameras" handshake, we talked for 20 minutes about our favorite old gear. It was great. I gave him a card with www.zintel.net scribbled on the back of it.

I've been to Vegas 3 times. The first was during my trip to California to visit my dad. I was about 14 and living in rural Nova Scotia. I remember walking up and down the Strip all night, jaw and eyes wide open, gazing at everything. I didn't know it at the time, but that was probably the point where I was committed to leaving Nova Scotia in search of bright flashing lights. Ironic that I ended up in cloudy Washington State, and the Strip, for all its grandeur, has pretty much been flattened and replaced with stuff with way more flashing bright lights.

My next trip was about 10 years later. I was pretty offended by the whole experience that time. I found Vegas to be a spiritually hollow, shallow, uncaring and impersonal place. And all that money, which really should have been collected in taxes to help the poor, was just being wasted. The place was a crude attack on my socially enlightened Canadian sensibility. In retrospect, I think the real problem was that I was nearly broke, and broke people just shouldn't go to Las Vegas. The real currency of Vegas is, well, real currency.

This was a great trip. A few of my Canadian friends would explain my new found admiration for Las Vegas in terms of the corrupting influence that America on me. But I can't believe it myself.

Everything is Vegas is grand.

I like to make sport with my friend and colleague Larry Morris, who runs the Windows CE group, that this was The .NET Compact Framework show. I don't work on operating systems anymore, so I can abstract literally thousands of person-years of work into a single box called "Operating System" and make fun of folks who don't think all day and night about language runtimes.

To put this in perspective, consider that my team has been working on NET CF for over five years now. It took us a little over three years to release Version 1. This was much longer than we had hoped (and estimated, and committed to our management), and was a long time to go without any customer feedback. Ironically, we're building a tool (an "app", in Larry's mind) to build apps, yet we had very little time and energy left to actually build non-trivial apps. So we guessed what would be important to apps writers, and we did "OK". But nothing works as well as real customer feedback.

Immediately after you release a version 1 product, where your success depends on the successful adoption of an ecosystem of partners (and some competitors), there is an initial burst of activity from early adopters. These are brave folks who just love new stuff, or folks who want to get ahead of the pack. Early adopters always suffer a bit of pain and the send the product team into short term crisis trying to help. The initial burst of feedback and the challenge of fixing problems and making partners successful is pretty exciting, and the product team and customers learn a lot quickly. The product usually evolves quickly. Early adopters kept us busy for about 8 months after we released. Many of these folks were at MEDC, and we are grateful. You can't get started if you can't get started.

After that wave, there is a disturbing relatively quiet time where each new win requires convincing more conservative decision makers to take a risk. Without a critical mass of already successful cases to point to, this is a lot of work and proceeds slowly. You pretty much sell yourself, your team and your product seemingly without effect for what feels like forever. We are also grateful for every win during this phase, and frankly, things go a little easier for them than the early adopters.

Finally, assuming you on track to win, adoption seems to spike with no apparent catalyst. Previously conservative decision makers become concerned that they will be left behind and see enough wins that they can manage risk appropriately. For us, this phase started around fall of last year and continues to this day.

Concurrent with great adoption, we are preparing to complete, and were talking about Version 2, our "customer feedback release". We pretty much know we got this one right since we basically did built (most of) what customers using the product asked us to build. People were eager to hear about it, because they asked for it.

While in Vegas, I read Hotel Babylon. This is the perfect Vegas read, in no small way to let yourself feel morally superior to the ultra rich and famous and in part to excuse any of your own bad behavior. But it also gave me some insight into the logistical complexity of the luxury hotel industry. It made me appreciate the bowl of fresh blueberries, strawberries, raspberries and pineapple that I enjoyed in the middle of the desert quite a bit more and gave me some appreciation for the effort that people put into making the stay comfortable.

I would like to share two moments that I experienced there. I worked on the original Windows CE product in 1994. We dreamed of all the cool mobile scenarios and cool devices we would be enabling. But we developed on big, slow prototype boards with small dim screens. When I finally saw the first real device, it was big and slow and had a small dim screen. I decided I didn't have the patience to wait any Moore and moved to work on the PC. But the iMate JAM was pretty close to the device I was dreaming about, and I was holding a real one in Las Vegas. I was in awe of Microsoft's stamina to see things to success.

Second, I asked one of our MVPs, Alex Feinman, who gave a great talk on COM interop, if he was happy with our quality. His response, "it's nice to use stuff that works", made me very happy. We don't talk about quality in public too much because frankly, you should just expect it. But we do work very hard to make sure that the platform is solid, correct, compatible and as bug free as possible. This process starts early in the cycle with good written design and peer review, ongoing automated correctness , stress, performance and application compatibility testing for all features, beta and early adopter feedback and a post-mortem after each release and after a serious bug slips through the process. It's a lot of work, and it makes us happy to hear that you can trust the product.

COM interop frightened me from the beginning. COM is an evolutionary technology, which means that there is no definitive specification to code and test against. The team really wanted to provide advanced features such as ActiveX hosting, but I dragged my feet. I wanted us to take smaller steps and be sure the technology was solid before expanding the scope. Well, things don't always go as planned ...

Alex demonstrated hosting a Macromedia Flash player as a Windows Forms control. I have to admit, it was cool. Damn cool.

Bill was there.

But I think that Bill was overshadowed by the .NET Compact Framework demos.

My team missed the Blue Man group. I heard that it was very good. We went to Penn and Teller instead, which was also good.

Mike M got us tickets. Mike seemed to know his way around Vegas pretty good, and was a great host. Mike is on one of the folks who you don't see much, but plays a critical role in making your experience productive by writing the code that makes the remote debugger work. Tricky stuff too. Mike got the good room.

After Pen and Teller, we decided to walk from the Rio back to the Mandalay Bay. We could see the Mandalay in the distance so we just started walking toward it. After about a half hour of walking, we could still see the Mandalay Bay, but it wasn't getting any bigger, or closer. We got fooled by the fact that everything in Vegas is grand. It turned out that the walk was long but great. There's lots of bright flashing lights to look at on the Strip.

An unexpected bonus for me was the fountain outside of the Bellagio. This thing was just mesmerizing. We went back the next night, playing a cool game we invented called drink a beer, watch the fountain, go in for a beer, back out to watch the fountain. At least I think we invented it. The fountain cycles at 10 minute intervals. It's a fun game.

Chris Muench, another MVP, showed how to write a D3D Mobile game in managed code. I gave the team a goal to make sure that C# was as good a choice as C++ for writing DX games on devices. Since Chris has the coolest game around, I'm assuming they have have been successful.

Chris Tacke shared his experience getting OpenNETCF bootstrapped. Chris refused to even step into the photo enabling light for even a moment. It took me half an hour on Photoshop just to get it this good.

Scott gave a talk on NET CF memory management. Scott explained all the work we do on your behalf so you don't have to worry about memory.

Roman gave a great talk on .NET Compact Framework performance engineering. He explained that if you aren't worrying about memory, you just aren't doing it. Roman got the best score for a new speaker. We knew Roman was a great engineer but we were pleasantly surprised to learn that he is a great motivational speaker too.

I love this one. We just get faster, and faster, and faster. And we're not done yet.

Last but far from least, Mark and David gave a great talking on debugging applications. This was the last talk of the show, and they packed the room.

Thanks for helping us make NET CF great.

Mike.

This posting is provided "AS IS" with no warranties, and confers no rights.

Blogging Because I'm Not Speaking at MEDC

MikeZintel — Wed, 04 May 2005 22:26:00 GMT

Daniel Moth sent me a note telling me that he would be at MEDC (great news), and that he was looking forward to my "talk". I want to set expectations here. I'm on a panel. I'm taking my cue from Jonny Moseley on Warren Miller's Fifty, when he introduces a freestyle event: "This is how we get it done these days. It's much too dangerous for me to actually be out there".

A panel is what you sign up for when you are too tired [ frightened | unprepared ] to do a talk, but you want to get a cool shirt. I'm actually honored to be on a panel. It seems only yesterday I was skipping school and firing home made potato(e) guns from inside an abandoned railway car, and now I'm on a panel. Cool.

The only reason I have time to write this is because folks I work with are sweating it out right now, reviewing their slides, practicing in front of mirrors, taking conflicting advice from lots of people on the manner of their speech, posture, dress, font size, etc etc. My job is to wander into their offices, and give them advice on the manner of their speech, posture, dress and font size. It's a good job.

Be sure and check out the talks by Richard Greenberg, Roman Batoukov, Scott Holden, David Kline and Mark Ihimoyan from my team.

I'm excited about MEDC. You have made the .NET Compact Framework a pretty hot topic. Seth Demsey and Neil Enns will be on stage with Bill at the keynote and they're going to show off some cool stuff.

I've been involved in some preparation for the panel. The general theme is around helping explain what tool is best for what job. Frankly, I'd prefer if the panel format was such that I just asked you questions on this topic. We've effectively built a toolkit with the .NET Compact Framework, with an almost dizzying numbers of choices. I'll defend this, given that there are lots of different programming problems, solutions, and even skills sets, conceptual models and biases about the best way to solve a given problem.

That said, I'm struggling to think about how I would position the following technologies, all of which we support today:

Relational tables, transactions and networked replication at the SQL-Mobile level; including data-binding to GUI elements
Relational tables, as ADO.NET DataSets, serialized and transmitted over network via web services
Hierarchical data, stored and manipulated as XML and the XML class libraries
Programming language objects, serialized to the file system, moved across the network via web services and data bound to GUI elements
Simple remote method calls over web services
“Push replication” via an ActiveSync file copy between a Windows PC and device
Low level network access
Store and forward messaging via System.Messaging/MSMQ

I look forward to talking to folks at MEDC to learn what works in the real world (i.e. outside Redmond - I've seen it on several vacation trips).

It never hurts to get noticed by the boss.

I had mentioned in a previous post that we were thinking about the problem of transiently connected networks for our next release. We're still thinking about it, most recently about the importance of addresses.

People focused on networking technology (like me) tend to under-appreciate the importance of addresses to customer adoption of a solution. Addresses are the proxies that people use to identify other people or organizations. Key qualities of great address, for a large network, are discoverability, lifespan and administration. Phone numbers are examples of addresses that are highly discoverable, stable for a long length of time and formally administered by an organization. Instant messaging addresses are slightly less discoverable, stable enough and created ad-hoc with formalized collision detection. URLs and email addresses are the two other very successful addresses, both with distributed administration and collision avoidance schemes.

The other critical quality of an address is the number of addressable endpoints. Phone numbers are excellent in this regard. Any one of over 1 billion landline and 1 billion wireless phones can address any other phone, simply by knowing a phone number. Those of us from the PC networking and Internet world often marvel at the continued success of SMS text messages, given their very high cost, limited bandwidth and boring user experience. Yet no other technology offers a set of addresses, and associated underlying protocols, with the qualities that customers value.

It sometimes comes as surprise that phone numbers could be superior to any one of several addresses used on the Internet. Despite the technical elegance of the basic TCP/IP machine address / application address, IP addresses are nearly useless to customers. While the shortage of V4 addresses has received considerable attention, the impact that the various techniques to work around this problem (DHCP, PPP, NAT) have had on the inherent peer to peer nature of the TCP/IP is underappreciated. The modern internet consists of a relatively small number of machines that have fixed addresses and a much larger number of machines that have very short lived addresses that are reused from a pool. The IP address of the latter group of machines is effectively useless as an endpoint address, doing little more than satisfying the technical need for a legal local address in a purely “connect out” environment.

Various dynamic name server technologies, like DNS and WINS can hide transient addresses behind a name with a longer lifespan. This is not likely to ever be practical on an Internet scale, but works fine for isolated networks within organizations. The effect of this is to create many new address spaces, with similar looking addresses but no interoperation among them.

URLs, since they build of DNS names, suffer from the same disadvantages. The URL space is extremely large and flexible, but it is difficult or impossible for me to offer you a URL that you can reliably address on my corporate machine, home or PocketPC.

IPV6 may eventually solve this, by assigning everyone an address with the qualities of a phone number. Dynamic DNS servers and/or distributed name resolution algorithms will make these addresses friendlier to humans. Unfortunately, the rollout of IPV6 to internet or phone system scale is a many year exercise, and even when the IP address space is suitability discoverable and long lived corporate security concerns almost guarantee that true peer-to-peer connectivity between any two machines in the world will never happen.

There is set of business and technical issues around mobile devices and operators that also impede true TCP/IP level peer connectivity. It is reasonable to plan for the vast majority of devices to be connect out only at the lower layers. A peer to peer user experience will almost certainly have to be created by using “relay servers”, like mail and instant messaging do today.

Anyway, the main thing this thought exercise has reinforced for me is that I have to hire some people, smarter, and harder working than me, so I can slack off and shoot potato guns. If you're interested, contact me at mikezin@microsoft.com. We're also look for hotshot graphics engineers too. If no one applies, you'll just have to use whatever stuff I can think up.

Finally,

I was late for work this morning. After my family left in the morning, and then my immediate neighbors left, I cranked the volume up to 11 and watched Mike Oldfield's Art in Heaven DVD. If you like Mike Oldfield, or progressive rock, this is a great disk. The concert was New Year's Eve in Berlin, and it makes me sad that I was huddled in the basement retesting the generator and checking the fresh water supply. But before you take musical advice from me, consider that I can lay down Pink Floyd, Pet Shop Boys, Al Stewart, B-52s, Neil Diamond and The Irish Rovers on the same tape (electro-mechanical playlist like thing) and think it sounds pretty good.

See you in Las Vegas.

Mike.

This posting is provided "AS IS" with no warranties, and confers no rights.

From the Dept of Information Retrieval on .NET Compact Framework Network Performance

MikeZintel — Thu, 31 Mar 2005 02:18:00 GMT

"How fast is the .NET Compact Framework?"

I have only recently organized enough thoughts and words to be able to answer this question with something more useful than "it depends".

The .NET Compact Framework, in the vast majority of cases, is fast enough. I know this because I've seen lots of complex applications running on the platform that are scalable and snappy. I've also seen apps that are dog slow. I used to claim that RPC's downfall was that it isolated the developer from the network but did not isolate the customer from the network. In a similar way, I fear we've isolated, or perhaps just delayed, developers from the pain of running on transiently connected, resource constrained hardware with lots of data and other apps running. Developing for devices, using any tool, is still a specialty. It's clear to me that we need to get faster (and we are), but we also need to do a better job at providing tools that give better visibility into the dynamic behavior of the system. Equally important, we need to provide some architectural guidance around best and worst practices ("don't read from a local table, convert to XML, serialize and send to another process via IPC, read into the DOM, modify it, pass it back, ..."). And finally I'm counting on the wider community to help us, and the community, to understand what works and what doesn't in practice and what we need to fix.

We presented the following information at a recent internal project review. I hope you'll be happy with the gains we've made to date and the work continues.

But this post isn't about the performance of the .NET Compact Framework, exactly. Lately, people, sensing that I'm feeling somewhat confident about general performance, have started asking me "How fast is networking with the .NET Compact Framework?"

"It depends."

I've like to be able to write an entry about networking performance with lots of tables of numbers and details and stuff. But I don't have the numbers. But I think I can do better than this post about nothing. I'm still trying to think up a way to plug my web site. I haven't thought of anything clever yet.

But I am prepared to ramble on for a while about a few things I've learned about the general problem of network performance. We will get better at specific measurements in the future. If you have better data, please share it.

Let's start with simple model of two boxes connected via a serial bus with hardware flow control. The bus has a fixed maximum throughput, measured in bits per second. The bus will deliver data to the destination as fast, but no faster, than the target can receive it. The sender can just push bits out the bus as fast as they will be accepted by both the bus and the target computer. The test app has a loop that generates blocks of data that repeat the sequence www.zintel.net and sends them to the bus. The receiver consumes and discards the messages as they are received.

The first thing we would likely measure on this setup is the overall throughput; how many bytes / second can we move between the machines? Let's say we're not happy with the result and we want to start optimizing. Where should we start? The first step is to isolate the bottleneck. In this case, either the pipe is at capacity, the sender can't fill the pipe, or the receiver can't empty it. Working on the wrong thing will have no effect at all on the performance. The first easy check is CPU utilization at each end. If it's low, then the pipe is likely the bottleneck, or there's a timer somewhere in the code path that is not allowing the CPU to fill the pipe. If it's close to 100% at either end, then that is the place to start optimizing.

It's quite helpful to do some "order of magnitude" math to see if the measured results are in any way reasonable. Let's say we assume both machines above have a clock rate of 1 billion instructions per second (1 GHz) and the serial bus has a rate of 100Mbit/s. The serial bus throughput in bytes / sec is roughly bits/s / 10, or 10 Mbyte/s.

Bus Rate Bits/s	100,000,000
Bus Rate Bytes/s (/10)	10,000,000

Let's assume that the sending machine requires 10 machine instructions, per byte, to generate the formatted www.zintel.net data buffers, and that the cost to send through the driver stack is 5 instructions / byte. With this information we can calculate the number of machine instructions that will be consumed by all net processing on the send side per second.

Instructions / Byte - Data Manipulation	10
Instructions / Bytes - Send/Recv Path	5
Total Instructions / Second	150,000,000

Since we know the clock rate of the machines, we can calculate the amount of processor load on the sending machine devoted to networking. In this case, it's 15%.

CPU Clock Rate / Second	1,000,000,000
CPU Utilization	15%

Clearly the bus is easily going to be saturated. The receiver is even more lightly loaded, as it is simply discarding the incoming bytes.

Bus Rate Bits/s	100,000,000
Bus Rate Bytes/s (/10)	10,000,000

Instructions / Byte - Data Manipulation	0
Instructions / Bytes - Send/Recv Path	5
Total Instructions / Second	50,000,000

CPU Clock Rate / Second	1,000,000,000
CPU Utilization	5%

Data compression would be a great optimization in the case. Assume that we count compression as an additional 25 instructions / byte on each side, and we achieve a 25% compression rate. For the sending side (the bottleneck), the math looks like this:

Bus Rate Bits/s	100,000,000
Bus Rate Bytes/s (/10)	10,000,000

Instructions / Byte - Data Manipulation	35
Instructions / Bytes - Send/Recv Path	5
Total Instructions / Second	400,000,000

CPU Clock Rate / Second	1,000,000,000
CPU Utilization	40%

The sender processor utilization is still only 40%, but the effective throughout of the system has increased by 25%, or 25Mbytes/s.

It's interesting to do this math on a 56KBits/s link and a 200 MHz processor, not unlike what you might see on a PocketPC on a (most excellent) GPRS network.

Bus Rate Bits/s	56,000
Bus Rate Bytes/s (/10)	5,600

Instructions / Byte - Data Manipulation	250
Instructions / Bytes - Send/Recv Path	10
Total Instructions / Second	1,456,000

CPU Clock Rate / Second	200,000,000
CPU Utilization	1%

I had to crank up the instruction / byte count to about 250 before I could get a 1% processor load. Clearly, compression is going to help, and CPU utilization is not a major concern.

If I substitute a 10Mhz LAN in the same system and don't change anything else, the processor can't even keep the pipe full. Focusing on the overall code path length / byte is the right optimization in this case. It's conceivable that expansion, say, by always sending the same block size, might even help (and be a disaster in the previous case).

Bus Rate Bits/s	10,000,000
Bus Rate Bytes/s (/10)	1,000,000

Instructions / Byte - Data Manipulation	250
Instructions / Bytes - Send/Recv Path	10
Total Instructions / Second	260,000,000

CPU Clock Rate / Second	200,000,000
CPU Utilization	130%

The next thing to consider is the size of transfers, as effective throughput x transfer size generally equals customer foot tapping delay, with a threshold of between 10-30 seconds generating a customer support call.

	4800	56K	10M	1G

1K	2s	0.18s	0.001s	0.00001s
100K	208s	18s	0.1s	0.001s
100M	208333s	17857s	100s	1s

The key thing to note here is that at the bottom end (like on, say, GPRS networks), data size has a huge effect on responsiveness.

Finally, it's worth considering latency separately from throughput. You might expect the time it would take (in seconds) to send a single byte would be equal to 1 / effective bandwidth. In practice, multiple different protocol layers add extra bytes as envelope, and the entire package needs to be delivered before the single byte can be processed.

I'm going to break one of my strongly held principals, "measure twice and cut once", and speculate about where I think the .NET Compact Framework is going to "net out" on networking performance today.

At the managed sockets level, the .NET Compact Framework doesn't add much processor overhead to the code path. If you replicated the simple two box and water pipe test above, even in the CPU bound case, I would expect we would be reasonably close to a native C++ solution. In the real world, of course, developers like do interesting and useful things with data, and that takes processor time. As the amount of C# processing increases, the delta between managed code performance and C++ will increase. I can imagine a worst case scenario (say, a fully loaded database engine written entirely in .NET Compact Framework - something I wouldn't recommend doing) where we're 50% slower. Fortunately we are not generally CPU bound in most real world networking scenarios today.

In the more typical slow network / fast CPU case, at the managed sockets level, I expect there is also little cost of choosing C#. You would have to do a great deal of data processing to create a situation where the processor couldn't keep up with the network.

From both a throughput and latency perspective, web services changes the equation. The thing that makes web services great for real world interop, text readable messages, also adds considerable envelope size to the data. Swapping out managed sockets for web services, in the fully loaded, CPU bound case, would likely drop your performance considerably.

The best way to decide what technology to use is to try a simple test prior to writing lots of code. The key is to make sure the test scenario is representative of the ultimate configuration. If you develop a client / server app that, for example, requires a server round trip to validate every field, then latency is going to be a key issue. Be sure and try this on a slow network. But if you are downloading, say high scores, or small maps, then the convenience of web services might outweigh then envelope cost, even on the slow network, and will be in the noise on a fast network.

As compared to the bus scenario above, TCP/IP adds another level of complexity. Where a bus can be conceptualized as a water pipe, a TCP/IP connection looks more like a water pipe, feeding a pressure tank, with dozens of different size pipes coming off on it, some of which feed an outdoor fountain, which drains into a sump, which elevates the water to a dozen gravity feed storage tanks, each with dozens of different size pipes coming off on it ... Frankly, I'm amazed the thing works at all. The "backpressure" on a TCP sender is essentially a heuristic on the send side about how fast the network, end to end, can deliver packets. If the network delivers IP packets in roughly constant time, and damages or drops few of them, then TCP will stabilize at a relatively efficient transfer rate. If the packet delay is changing widely, the error rate high or constantly changing, then TCP will become confused, thinking that it is overwhelming the network, and will throttle back the send rate. The bad news is that there is nothing you can do about this. The good news is that the TCP vendors, us included, are always working to optimize this.

The reason I bring this up is that it can bite you on GPRS, and result in 50% decreases in throughput and corresponding increases in latency. At some point, you need to switch your application to a store and forward model / database replication model, where the UI is essentially a view on the local data, and an asynchronous replication engine periodically tries its best to keep the local and remote databases in sync.

The speed of your "first hop" TCP/IP network connection usually doesn't have much to do with the effective bandwidth, unless it is the slowest link (like, say, GPRS, for example).

If you are using TCP, learn about NAGLE. If you introduce any stop and wait behavior in your system, be wary of the huge effect on throughput (but not latency) that that can have.

I'm getting ready to go to MEDC in Las Vegas. I cut myself a pretty good deal this year. I'm on a panel discussion, which gets me one of those cool "speaker shirts", but I'm not giving a talk, which takes lots of preparation and frankly scares me silly. Both my team and some our MVPs are working on new material for the show. I look forward to seeing you there.

"We're all in this together".

Mike.

This posting is provided "AS IS" with no warranties, and confers no rights.