mikechampion's weblog

Web Services Standards and .NET Interoperability

Successive versions of the .NET framework closely track the evolution of the WS-* specs as they progress from publication, to submissions to W3C or OASIS, and ultimately as W3C Recommendations or OASIS Standards. See for example the list of supported interoperability specs in .NET 3.0 (which shipped with Vista) and .NET 3.5 (which shipped with Visual Studio 2008), noting how the later release referenced the final standard for several specs whereas the earlier versions referenced the original submission. 

Not only does Microsoft closely track the standard version of specifications, Microsoft invests in helping other companies do similarly.  One way is by sponsoring  “plugfests” where implementers can get together and test what works and what doesn’t while products are still being developed.  Don’t believe me, take a look at accounts of the recent web services plugfest from other “coop-etitors” in the SOA industry.  For example Stefan Tilkov  interviewed Harold Carr to assess the state of interoperability between Sun’s Metro product and WCF.  Harold notes:

.NET 3.5 (which is released) is based on the standard versions. Metro 1.x (which will ship later in 2008) will be based on the standard versions also.

Harold's own blog has details on exactly which of the ratified standards were used successfully to interoperate with .NET 3.5. 

These ongoing  Plugfests are a great way to increase real world interoperability, and we hope to see even broader participation in the future. See the scenarios and frequently asked questions page for details on how to participate. All interested parties are invited irrespective of which platform they develop for and whatever their business relationship (or not!) with Microsoft might be, with no legal agreements to be signed.

Here is a list of Web Services standard specifications (i.e. the versions of these specs that are OASIS standards or W3C Recommendations) that are shipping today in WCF, Microsoft’s application platform framework: 

• SOAP 1.2
• W3C Web Services Addressing 1.0 - Core, SOAP binding, WSDL binding
• MTOM
• WS-ReliableMessaging 1.1
• WS-Security 1.1 (also SAML Token Profile 1.1)
• WS-Policy 1.5 / WS-PolicyAttachment 1.5
• WS-SecurityPolicy 1.2
• WS-SecureConversation 1.3
• WS-Trust 1.3
• SAML 1.1 Token Format
• WS-Addressing 1.0
• WS-Coordination 1.1
• WS-AtomicTransaction 1.1
• UDDI 2.0
• WS-I Basic Profile 1.1
• WS-I Basic Security Profile 1.0

Other ratified standards are supported elsewhere in the Microsoft product line, e.g. the DMTF WS-Management standard is supported in Vista.

It’s fair to point out that not all potentially relevant specs that can claim to be standards are supported in .NET (or the native Windows XML libraries for that matter).  The clearest example of a ratified version of a spec that Microsoft and many others chose not to implement is XML 1.1.  This was a difficult decision, since some changes in XML 1.1 make it better aligned with the evolving Unicode standard than XML 1.0 is, but others  are “breaking changes” that would actually degrade real world interoperability.  In this case, principled pushback on a standards committee has led to a proposed new edition of XML 1.0 that has improved Unicode alignment without sacrificing interoperability.

We encourage implementers to participate in the ongoing work to demonstrate and improve real world WS-* interoperability, and for customers to let us know about additional scenarios and specifications where we need to focus in the future in order to make cross-platform services ever more valuable.

WS-Bandwagon or WS-JustRight?

My previous post used WS-Management to illustrate the larger point that  "the WS technologies are taking hold, deep down in the infrastructure, doing the mundane but mission critical work for which they were designed."  Perhaps because  WS-Management is one of the more controversial bits of the WS-* infrastructure, that example stimulated more pushback than the other example I used -- the increasing success of the WS-* based identity metasystem (which is a lot more newsworthy at the moment).   But it's easy to justify the value of standards and open source work in the identity space, where the prevalence of phishing indicates that the web does not have a good native story that falls out of the REST architecture.  Let's look more closely at the harder question of why WS-Management exists and why it is gaining traction.

 A couple of points in reply to comments on the earlier post: 

• It overlaps in functionality with WSDM, another family of specs, and this overlap is a source of friction between the MS and IBM web services stories.  I won't get into the history or politics here other than to speculate that WS-Management is a classic example of the Pareto Principle -- it does quite a bit less than WSDM (maybe 20% of the functionality and complexity of WSDM), but its relative simplicity makes it easier to implement while being good enough for 80% of the important use cases.  So, even in the allegedly pathological WS ecosystem, evolution favors the simple and functional over the the more highly but complexly designed.
• Why didn't something even simpler and more RESTful evolve, which Patrick Logan's comment suggests would have been easy?  I don't know the history, but it's clear that REST concepts were in fact leveraged to keep the spec relatively simple.  Patrick also pushed back that that it is a "shell of a solution with little agreement on what goes into the shell." REST is also a "shell of a solution", and WS-Management adopts the same basic approach to allow late binding to the resources in a catalog rather than tight coupling to a predefined set of objects.
• Why not just use HTTP rather than WS-Transfer?  I created a bit of contention by asserting that WS-Man is designed for use "in situations where the web-scale alternatives really don't fit, such as deep within operating systems or in the firmware of chips".  A number of people pointed out (probably correctly, but I don't really know) that an HTTP implementation would "fit" in the same amount of code or memory as a WS-Transfer implementation.  That being as it may, I meant "fit" in the architectural sense -- I thought the common REST argument that HTTP is an application protocol indicated that it would not "fit" so deep down in the stack as the firmware on a chip or device.  More significantly, HTTP does "fit" on top of TCP/IP, but WS-Management was designed for use in situations where only UDP network services are available.  
• Whether or not the designers were simply riding the WS bandwagon rather than thinking hard about RESTful alternatives,  WS-Management sits on a number of WS specs, not just WS-Transfer.  By building on WS-* they could leverage services that HTTP does not provide.  WS-Eventing provides asynchronous "push"  notifications, WS-Enumeration supports the traversal of collections of resources, and WS-Security, WS-Trust, and WS-SecureConversation provide a security infrastructure.

WS-Management leverages some key benefits of SOAP such as its protocol neutrality while adopting some of the REST abstractions to avoid the complexity of an RPC API for all possible management service invocations.  In other words, it exemplifies what Jon Udell calls WS-JustRight for its intended domain: "I stand by what I’ve been saying all along, in a variety of places including this InfoWorld cover story: there’s WS-Heavy, there’s WS-Lite, and for every situation there’s a WS-JustRight that may rely on elements of one, the other, or both. The Richardson/Ruby book brings much-needed clarity to the WS-Lite end of the tolerance continuum, and that’s a great thing. But when we celebrate one end of the continuum, why must we deprecate the other?"

WS-* and the Hype Cycle

There's a persistent theme talked up by WS-*ophobes that it's all just a fad, rapidly sliding down toward the "Trough of Dilillusionment" in the Gartner Hype Cycle. I've come to the opposite conclusion after six weeks back in the web services world.  The WS technologies are taking hold, deep down in the infrastructure, doing the mundane but mission critical work for which they were designed. 

Let's consider one example, WS-Management, which I had barely heard of when I started in CSD Interoperability. It's stated purpose is:

To promote interoperability between management applications and managed resources, this specification identifies a core set of Web service specifications and usage requirements that expose a common set of operations central to all systems management. This comprises the abilities to
• Get, put (update), create, and delete individual resource instances, such as settings and dynamic values,
• Enumerate the contents of containers and collections, such as large tables and logs,
• Subscribe to events emitted by managed resources, and
• Execute specific management methods with strongly typed input and output parameters.

At first glance this appears to duplicate widely deployed bits of the web.  For example, it depends on the oft-criticized WS-Transfer spec, and some are advocating using Atom and the Atom Publishing Protocol rather than WS-* for describing collections and subscribing to notifications of their contents.  On closer examination, WS-Management is widely used today in situations where the web-scale alternatives really don't fit, such as deep within operating systems or in the firmware of chips.    For example:

For IT Professionals, Windows Server and Microsoft Operations Manager will enable the management of heterogeneous software and hardware systems using WS-Management. For consumers, Windows Vista will support the discovery of and interaction with Web services-enabled devices, such as printers, digital cameras, and home control systems.

Those devices may or may not be "on the Web"; increasingly support is built into the firmware of the devices. For that matter, the heterogenous software systems being managed may include HTTP servers.  Furthermore, this is not a Windows-specific technology, it is actively developed and supported across platforms by HP, Intel, and others. 

Another example where WS standards and technologies are quietly taking root is in the area of identity management.   Both the major identity management stacks (WS-Trust / WS-Federation and Liberty) depend on Web services technologies, and there is no credible "pure Web" alternative. The "identity metasystem" supported by  Windows Cardspace and several other vendors and open source projects that to some extent bridges these stacks is getting real adoption, inching WS-* up toward the "plateau of productivity."

Perhaps we are seeing a repetition of the pattern that made XML ubiquitous.  It was originally conceived as "SGML for the Web", but that vision has never materialized -- non-wellformed HTM, Javascript,  and proprietary formats such as PDF and Flash predominate on the Web itself. XML has become pervasive as a convenient format for less glamorous tasks, such as configuration files, summaries of site contents (RSS and Atom), and lists of lists (OPML).  Likewise, at the peak of the hype cycle the web services technologies were promoted as part of a grand vision.  The post-Vista world may not look exactly like the 2001-vintage Longhorn vision, but WS technologies are doing the unglamorous work for which they were actually designed. 

In short, from what I have learned recently, the trajectory of WS-* isn't pointing toward oblivion, it looks headed toward the same sort of pragmatic ubiquity that XML has achieved.  That's not to say all is rosy; there is lots of confusion and dissension, again just like there was in the early days of the Web and XML.   Likewise, "ubiquity" doesn't mean that the WS technologies are the best or the only option across the board,  but that it they are increasingly available as a very viable option when developers need protocol-neutrality, security, identity, management capability, etc.

The Secret of LINQ Design

A team within Microsoft ran an "app week" recently to build applications that implement customer scenarios using a variety of LINQ technologies.  The feedback on LINQ to XML was uniformly positive. The participants were not XML geeks, but more like our target audience: developers who come across XML now and then and need to deal with it without undue fuss. This group found that they were immediately productive without having to read the documentation, because the APIs worked the way one intuitively thought they should.  This set off an interesting internal thread speculating on why LINQ to XML was successful in achieving its design objectives in a world (and to be honest, a company) where this is not the norm.

I saw The Secret of Apple Design and was struck by some similarities with how LINQ to XML came about. The article quotes extensively from design guru and former Apple exec Donald Norman.  For example:

"[Jobs] had a single, cohesive image of the final product and would not allow any deviation, no matter how promising a new proposed feature appeared to be, no matter how much the team complained. Other companies are more democratic, listening to everyone's opinions, and the result is bloat and a lack of cohesion.

Thie quote points to the first similarity -- the existence of a designer with the skill to make hard decisions and the authority to make them stick. In LINQ, the Steve Jobs role is played by Anders Hejlsberg. Superficially, the LINQ design teams  look a lot like W3C working groups, with regular meetings where requirements are interpreted and fine-tinued, and proposals for alternative ways of achieving the requirements are presented and discussed, sometimes for months.  In sharp contrast to the designed-by-committee XML technologies such as DOM, however, LINQ has an architect who makes decisions, not a chair who drives consensus.  In the LINQ design meetings, all opinions were considered, and previous decisions open for reconsideration, but there is no pretense of democracy.  When it came time end a discussion, there wasn't a vote, or weasel-wording to paper over disagreements, there was a decision.

The quote also highlights the importance of conceptual integrity. Anders began what is now called LINQ to XML with a clear vision for how a modern XML API could work, with C# functional constructors and LINQ queries providing an essentially functional approach to working with XML.  Lots of ideas were proposed, and lots of people complained when they were removed or rejected, but the goal of internal cohesion and alignment with LINQ was paramount.  Of course, XML reality intervened in various unavoidably ugly ways, but much design effort was put into minimizing and isolating the ugliness.

The third similarity I noted is an insistence on simplicity over feature richness.  Quoting again from the article:

"The hardest part of design ...is keeping features out." Simplicity, [Norman] says, is in itself a product differentiator, and pursuing it can lead to innovation.

Rolston agrees. "The most fundamental thing about Apple that's interesting to me," he says, "is that they're just as smart about what they don't do. Great products can be made more beautiful by omitting things."

 A year ago, I wrote a post on what LINQ to XML would not do, such as guarantee that a tree in memory is well-formed, support XML 1.1, offer more than bare minimum support for DTDs, round-trip entity references, and so forth.  Likewise, the design team decided to remove the streaming input API that had been extensively discussed and prototyped.  In many such cases, Anders decided that the functionality benefit wasn't worth the complexity cost.

Of course, the real secret to successful design is the skill and tenacity of the designers and the implementers; giving a chief designer the authority to purse cohesiveness and simplicity just enables success, it doesn't assure it. Likewise, LINQ's technical success doesn't guarantee real world success. For example, for all  its aesthetic flaws, DOM has been an undeniable success story in helping make XML and dynamic HTML reasonably usable by developers, and has helped the world move from mostly proprietary to interoperable data formats over the last 10 years.   It is possible that the same ugly compromises that promote committee consensus actually help overall adoption.  Likewise, just as Apple's design aesthetic pays off because it appeals to an upscale market willing to pay a premium for form and fashion on top of functionality, perhaps LINQ to XML will appeal mostly to a niche in the developer community.  We shall see ... but I bet a couple of years on this and I'm convinced that this technology will be widely adopted on the Microsoft platform and will spark a round of competitive innovation on other platforms. 

Accelerating Evolution: LINQ News from Mix 2007

There is a lot of interesting (and once confidential) stuff that came out of the Mix conference this week.

Jon Udell's  "Watching Anders Hejlsberg reinvent the relationship between programs and data" ... offers an enthusiastic summary:

A lot of the time, when we use the web, we’re effectively performing joins among data sources. You visit one site to look up some data, then you grab some of it and plug it into another site. If you’re lucky, somebody will have built a mashup, on a third site, to facilitate that join. But what if your browser had the data manipulation chops to help you do that mashup directly? I’m hoping that technologies like Silverlight and LINQ will enable things like that to happen.

Jon doesn't elaborate a whole lot on what "reinventing the relationship between programs and data" means.  Fortunately, Anders' LINQ presentation itself is now online.  Some points he makes about the new relationship between programs and data include:

Data != Objects  - Now you can query data without it being in a database ... LINQ offers "native support for queries as a first class concept". C# and VB now offer the same expresive power as SQL and XQuery built right into the languages.  LINQ unifies and brings to a higher level of abstraction the whole notion of data-driven programming. 

Imperative->Declarative - Programmers can focus on "what now how" without having to learn a new language such as SQL or XQuery.  Imperative approaches have run their course; we won't see 10x productivity improvements anytime soon, nor will compilers be able to automatically adapt imperative code to new hardware.  But the declarative approach will support 10x improvements in the performance of data-driven applications and the "what not now" approach allows compilers to adapt to multicore architectures and so forth.  The key is to adapt programming styles to a more declarative, query-oriented style.  LINQ supports this.

OO programming languages have no notion of projection, you have to do it with tedious declarations and imperative transformations.  LINQ makes this much easier, with anonymous types and functional construction.  In the demo, Anders usees this approach to transform XML data into business objects with very little code, and no schemas or code generators.

I'd elaborate a bit on the last point.  Lots of technologies exist that leverage some explicit mapping from raw data to programming objects; that mapping might be a conceptual model, a schema, an ontology, or an adapter.  For example, XQuery is used to integrate and query over diverse data sources, but only after the data has been explicitly transformed from its native representation (e.g. SQL tables) to an XQuery data model.  LINQ to Objects and LINQ to XML, on the other hand exploit the IEnumerable<T>, aka monads/monoids, that is implicit in almost all collections of data, or exposed as axes in XML documents.  To me, that's the essence of LINQ's reinvention of the relationship between programs and data: conceptualizing data in an object graph, database, XML document (or Amazon.com catalog,  LDAP directory,  ad infinitum) allows developers to write smaller, cleaner, more parallelizable, etc. programs with much less concern for the plumbing.

Putting all this into a realistic application, Aaron Dunnington and Tim Scudder of the Data Programmability / XML team gave a Mix Presentation that exploits the XML features in Silverlight clients and LINQ, etc. on the server.  Their demo is called "The Socializer", showing how Silverlight applications can bring all sorts of social networking data together and display it in a visually appealing way. Technologies used in the demo include Silverlight, C# 3.0,  LINQ to Objects, LINQ to XML, RDF, FOAF and more. 

Needless to say, all this Silverlight love from unexpected parties isn't going unchallenged.  The most popular statement of the opposing side seems to be the fine rant from Mark Pilgrim. His most trenchant bits are unquoteable, but the gist of it seems to be that Adobe and Microsoft (and Sun's "alternative to Ajax") indicate the return to the bad old days of proprietary technology and vendor lockin.  Astonishingly enough I see it differently: The purely standards-based web is quite functional and interoperable for relatively static content, but the "Web 2.0" demands for browser-centered applications that are dynamic, attractive, and portable are very difficult to meet with currently standardized technologies.  As Dan Ingalls of Sun put it:

AJAX sort of deals with all of the old way of doing things. It makes it simpler, which is great, but underneath it’s still all this junky HTML, Document Object Model, CSS, all that stuff, where 30 years ago, we knew how to do that stuff cleanly with a dynamic programming language and a simple graphics model.

That's NOT to say that Ajax is dead.  As far as I know, there was a lot of Microsoft Ajax support announced at Mix, and more in the pipeline.  I hope and pray we (inside and outisde MS) don't repeat the IE6 mistake of declaring "junky HTML, DOM, CSS, all that stuff" obsolete and stop maintaining the standards and implementations.  It will live on, and improve for some time... while LINQ/Sliverlight/Apollo/Flair/whatever mature and potentially drive a new generation of web standards.  But a new generation of web standards won't be invented by committees, but by developers, then polished  by competition ... then committees can come along and tidy up.  Just like Web 1.0 came about.

It's interesting watching various pundits and analysts read the tea leaves of the numerous LINQ subprojects, Silverlight versions, and incubation projects such as Volta to figure out the "Microsoft" master plan for pulling it all together.  Just as customer pain with today's technology drives diverse innovation across the industry, so it does within Microsoft, and the process resembles evolution in action more than intelligent design by a supreme architect.  I don't think we're seeing a silly season so much as a Cambrian explosion of new ideas from all sorts of places, and Father Darwin alone knows how it will end up.

Reporting for duty on WS-Deathstar

After an enjoyable and extremely educational 2 1/2 years on the core XML team in SQL Data Programmability at Microsoft, I've moved to a position in the Connected Systems Division's Interoperability unit.  Responsibilities include representing Microsoft on the W3C, helping with web services standards partnerships, and generally helping the world understand the method behind the apparent WS-Madness, or at least the subset of it that Microsoft endorses.

"Huh?" you might ask.  Why voluntarily join the doomed crew of WS-Deathstar?  Or maybe the WS-* crowd are not evil, they're  weenies who get sand kicked in their faces? As with XML itself, the point isn't whether "any damn fool could produce a better data format" it's whether the web services specs provide a pragmatic foundation for actual interoperabiliy. Just as XML quietly became pervasive while its numerous limitations were being flamed by the cognoscenti, the WS technologies are quietly taking root in all sorts of places where interoperable identity, security, reliability, management, etc. concerns trump geek aesthetics.  There's no disputing that skillful people can implement systems with all these properties without WS-*, but there is much to be said for building these capabilities into the infrastructure of Windows, .NET, Websphere, Java, Apache, etc., in a standardized and interoperable way. I don't pretend that goal has been achieved, but I'm convinced its achieveable.

 Another part of my reason for moving is that LINQ to XML is close enough to being done that I can declare victory and move on, hoping that its success will rebalance my karma after the beating it suffered from my role in DOM :-).  I'm still involved in the project to some extent, backing up Ralf as he assumes the role of LINQ to XML PM and doing my bit to evangelize the LINQ XML story in Orcas and Silverlight.

But there are some more substantive attractions to WS-Darkness.  First, the overall WS-* framework doesn't deserve its bad reputation among XML and Web geeks. It's hard to call anything designed by that many committees elegant, but there is an underlying conceptual integrity in it that I've come to appreciate over the years. That's not exhibited by all the specs, to be sure ...don't ask me to defend XSD or WSDL!  There's no disputing that current interoperability above the SOAP level can be problematic, but much of this can be traced to incorrect implementations and problems mapping XSD to underlying object systems. It will take longer to get full interoperability at the higher levels of the WS stack, but I'm convinced that it will be much more practical to complete the work that is underway than to start over with some new paradigm, or to return to the bad old days of one-off data contracts and armies of consultants needed for everything.

 The stock phrase in the WS-* specs is that they are "designed to be composed with each other to provide a rich messaging environment".  I think the term "Aspect-oriented messaging" better captures the underlying architectural principle that the WS specs can work separately or be composed together to provide a variety of message services: 

[SOAP 1.2] provides a much simpler and more powerful aspect-oriented mechanism for extending the SOAP processing model. New aspect-oriented features can be added to SOAP by defining additional SOAP headers and how to process them. Such features can range from transport-level features such as security and reliable messaging to business-level features such as spending-policy enforcement to personalization and differentiated service. Bottom Line: Users who consider SOAP to be just a low-level RPC mechanism should review SOAP v1.2 to understand its potentially profound architectural implications for distributed computing.

"Ironick" indeed, that's by the same Nick Gall whose Web of Services for Enterprise Computing whitepaper got so much link love from the WS-skeptics. Gall may or may not have come to regret his previous enthusiasm for SOAP's architectural principles, but in any event the main point of his position paper isn't  isn't so much about WS architecture as it is about W3C's philosophy: "While there is nothing wrong with [the various WS-* specs]  --in fact there is great value in having the major middleware vendors finally agree on middleware standards after so many years of proprietary middleware protocols--the problem is that such work has nothing to do with web architecture or the W3C."  That may be a defensible position, but the fact remains that the W3C management and staff have worked hard to give at least some of the more fundamental and web-related WS specs a home at that organization.  

Which leads to my second point:  "It's a tool, not a religion".

I have spent time around job sites. I can promise you that you'll find circular saws, chop saws, and table saws. Is there overlap in functionality and potential application of these saws? Absolutely. Why have all three then? Well, because each is particularly good for certain types of cutting chores and less optimal for others. Contractors, like mechanics, want a broad set of tools available so they can have the optimal tool for each task.

Why then do we, as software engineers, have to work so hard to reduce our toolbox to the ultimate tool? There is this tendency to look for the Swiss Army language with supporting framework that will be the optimal solution for every problem in the world. The simple reality is that such a language and framework can never be created.

I've always been annoyed by the zealotry manifested in the "SOAP vs REST" debate.  The overhyped WS marketing and the RESTifarian pushback in 2001-2002 or so had the effect of people "flipping the bozo bit" on each other wholesale.  We're just now seeing some exploration of the sensible middle position -- the native Web technologies are powerful and deserve first class support in the services-oriented tools and specs, and at least some of the WS technologies are useful on the Web as well as the enterprise, especially in the realm of identity and security.  People on the WS side are finally understanding the the potential of the SOAP GET binding, and maybe the REST folks are starting to understand the oft-maligned WS-Transfer  ("HTTP over SOAP over HTTP") is not quite so pointless in a multi-hop, multi-protocol world.

Finally, the supposed coup de grâce in most skewerings of WS-* is to point out the complexity of the web of specs and frameworks.  The best response I've seen is by Charles Zedlewski:

But I think the kvetching over the WS-* complexity is pretty overblown. What struck me looking at Innoq’s diagram is how similar it is to the Java class library (or any other major language’s class library for that matter).

...  you don’t have to learn all the inner workings of either standard to make good use of it. You either:
1) Learn and choose to leverage only the most useful bits of the standard that are relevant for your project. ... Or you:
2) Utilize some combination of tools and platforms that hide the complexity of the underlying standards.

Anyway, I'm having fun getting back up to speed on all this and getting back in the blog debates I swore off awhile ago. 

Convergence Zones

I had a lot of time to think about Elliotte Harold's call for XML predictions on the way home from Redmond Wednesday night.  We got several inches of snow, which is rare here and the highway folks just can't deal with . There were massive traffic tieups, and lots of time spent staring off into the snowflakes. Most of us commuters were caught off-guard by the snow, since the forecast was something like "cloudy with a chance of snow showers".  That's not inaccurate, but not very helpful ... like most of those "successful" beginning-of-2006 predictions pundits are bragging about this month.

Our unexpectedly intense snow yesterday was created by a Puget Sound Convergence Zone:

Those northwest winds will collide with the Olympic Mountains. Part of the air flow will be deflected east down the Strait of Juan de Fuca, while the other part will be deflected down the western side of the Olympics. When the northern branch reaches the I-5 Corridor and the Cascade Mountains, it will then be forced to the south. Meanwhile, when the southern branch reaches the I-5 corridor and Cascade Mountains past the southern side of the Olympics, it will then turn to the north.

Eventually, the south-flowing branch and the north-flowing branch will converge. When that happens, the air has nowhere to go but up. Rising air will lead to convection. That will lead to cloud and storm development.

That seems like a good metaphor for why technology prediction is hard.  The interesting stuff happens in the "convergence zones" where different streams of ideas and technologies collide and create upward convection currents (driven partly by hot air and vapor I suppose).  The World Wide Web arose out of a convergence of the internet and SGML-ish markup;  the Internet Bubble arose out of a convergence of this enabling technology and the vast numbers of PCs in every home and office; XML arose out of a convergence of the capability for universal interoperability spawned by the SGML and the Web,  and the demand for a Web-like experience in a whole range of document and data products that was accelerated by the Bubble. 

But not every wind / technology convergence creates a convergence zone.   See the "Meteorological Mumbo-Jumbo" section in this article about our weather yesterday for an explanation of how this weather pattern was different than the other 20 recent occasions where the winds were about right:

This time, the cold air is, at least initially, coming from the Gulf of Alaska. Normally, that pattern doesn't make for much snow because the air has a long time to moderate over the warmer ocean waters. But the air mass up there is SO cold ... the air was still cold enough to snow by the time it made it here.

In technology, lots of things -- technologies, personalities, zeitgeist -- have to come together at just the right time.   Why was "Dynamic HTML" a bit of a yawner in the late '90s but "AJAX" a big hit in 2006? Why did the Newton flop and then the Palm become a hit a few years later?  Why is YAML a backwater but JSON the Next Big Thing this year? I don't pretend to know.  I do think, however, that we know enough about the XML technology landscape and the forces that interact with it and each other to at least talk about some likely convergences and divergences.

First, as the weather truism goes, the best prediction of tomorrow's weather is today's weather.  2007 will look a lot like 2006.  Duh.  Overall, XML has become a fairly mature and slow-changing technology because there is a 10-year legacy holding back any radical change, e.g. a mass migration to RELAX NG, JSON, LINQ, or whatever.  We will see a gradual migration to XSLT 2.0, growing interest and support for XQuery (as a query language!), continued improvements in XML Schema 1.0 interoperability and growing interest in 1.1 as it solidifies, and so on.

Second, there's a strong prevailing wind that puts XML almost everywhere, even places where it might not belong (e.g., multi-gigabyte log files!). But the pervasiveness of XML means that the "XML community" has less and less in common.   The streams that collided with Mt. NonInteroperable and reconverged to create XML 10 years ago have shifted because Mt. NonInteroperable has been greatly eroded.  People care more about interoperable documents OR data OR messages than interoperable documents AND data AND messages, so the fact that XML underlies all of their interop stories is not particularly interesting anymore. What predictions does this pattern imply?

•  Assuming that nothing pops up to reconverge these separate streams, we'll probably see a continuing decline in the popularity in generic XML forums, books, conferences, etc., and more integration of domain-focused XML topics into more specialized venues.
• Likewise, we'll probably see an increase in the use of niche XML-related technologies, such as RELAX NG for purely textual documents and JSON for web applications, in the domains for which they are well-suited. 
• Conversely, we'll see a decline of interest in "one size fits all" approaches.  For example, it looks (from the outside) like the W3C Efficient XML Interchange people are focusing on one problem -- the fact that XML is too bloated for many limited-bandwidth scenarios -- and not trying all that hard to come up with something that is smaller / faster / cheaper / more universal than XML for all scenarios. 
• Finally, the breadth of XML's adoption comes at the expense of depth. This suggests that we'll see more of the pattern that Jon Bosak decried in his XML 2006 keynote: "business users will never adopt a solution that depends on an additional XSLT pass because it would require them to learn something new (never mind that this “new” thing had been widely employed in other contexts for years)." Those vendors may know their customers, and suspect that they really don't want to learn yet another XML technology just because it would offer an elegant solution to a somewhat peripheral problem.

Third, like the wind XML is mostly invisible -- it's hidden behind the firewall, embedded in ZIP files, or just called something else. What's more, most people really don't want to see the XML that surrounds them. They want to see readable documents, updated feeds, processable objects, and delivered services.  The better the XML is hidden, the more the bulk of the world likes it. Being out of sight, it's also out of mind.  Few noticed that "Asynchronous Javascript And XML" actually refers to the XmlHttpRequest API much more than XML content, so the substitution of JSON for XML as the typical bits on the wire format caused little concern. Actually that understates the case -- most people are happy to use more familiar technologies instead of XML or in front of XML.  What might we predict from this pattern?

• I'll make a (self serving!) prediction that the LINQ approach of focusing on what is common across data formats will get more mainstream traction than will the notion that users want specialized tools for their XML data.
• Tools that make it relatively easy to consume XML directly into programming objects such as LINQ to XSD will continue to mature technically and be adopted by pragmatic developers.

What about some of the XML technologies that have been swirling around out there ...  What landfalls may we expect in 2007? Again, the whole point of the argument here is that only incremental change can be foreseen unless XML technologies get tangled up in some larger crosscurrents.  So, the Semantic Web will remain highly domain-specific (e.g., the biomedical arena where well-established taxonomies and ontologies really can be leveraged by OWL inferences, SPARLQ query processors, etc.).  I can't think of any larger social forces that might collide to provide upward convection for the semantic web technologies other than some low-probability event such as a major terrorist attack being thwarted as a direct result of Homeland Security's semantic technology investment.  How about SVG, or XML 1.1, or some other major XML technology that hasn't gotten mainstream support (ahem, partly because of decisions in Redmond)?  I don't foresee any major shifts in the winds here ... but then again I didn't know about the Puget Sound Convergence Zone the other day until the streets near home were iced over.

What great technological, economic, or political forces can you envision changing the XML climate in the next year?

Since Don put me up to it ....

I don't really want to perpetuate the 5 things meme, but DonXML asked nicely (and I'll take the opportunity to shamelessly plug some favorite people, products, and organizations):

• I came to the software industry rather late in life after a mis-spent youth in Political Science at the University of Michigan.  I discovered that it was more fun writing the code than writing the dissertation, not to mention the fact that the job market for junior programmers was immensely larger and more lucrative than the market for assistant professors circa 1980.  Furthermore, programming gave me the luxury of staying in Ann Arbor for 25 more years.
• My longest lasting hobby is playing the acoustic guitar. I  learned much of what I know from Frank Salamone, who sadly is no longer able to play. I spent the accrued vacation pay from my last job on a Martin 000-28 when I started at Microsoft.  The piece I most enjoy playing is Leo Kottke's The Fisherman.  I try, with minimal success, to play various stuff by John Renbourn.
• My musical tastes run mostly toward the classical/traditional. A survey of the dying iPod (Apple seems to have rediscovered the virtues of Planned Obsolescence) doesn't show a whole lot of written after 1759 (when Handel died) and only a handful  after 1827 (Beethoven).   All time favorite recordings include Parkening Plays Bach, Trevor Pinnock's Goldberg-Variationen, and Derek Bell's Carolan's Receipt.
• My current favorite "leisure" activity is, as one of my kids puts it, getting torn up by nasty thorny things.  I'd give it a more pretentious label, "natural area restoration", which around here involves a lot of struggle with Himalayan Blackberry on behalf of the Sammamish River Stewards. Back in Ann Arbor, is was a struggle with European Buckthorn on behalf of the Friends of Dicken Woods. 
•  Based on the countries I've actually visited, if I had my life to live over I might do it in either Norway or Australia.

Being paranoid of this kind of response, I won't inflict the meme on anyone else.

The JSON vs XML debate begins in earnest

After seeing  Douglas Crockford's talk on JSON at XML 2006 recently, I figured that some sort of great debate between XML and JSON advocates was brewing.  I had been waiting for Elliotte Harold's rebuttal of what Crockford is missing, but haven't seen it yet.  What has happened is that Dave Winer got off a  rant against JSON as reinventing XML's (and more specifically XML-RPC's) wheel:

God bless the re-inventers

Gotta love em, because there's no way they're going to stop breaking what works, and fixing what don't need no fixing

Crockford gets off a very pithy response: 

The good thing about reinventing the wheel is that you can get a round one.

There are a lot of other very interesting observations in that comment thread.  Some points that hadn't been obvious before to me include:

• There seemed to be a lot more JSON fans than XML fans in that thread (maybe because the original post was just a wee bit inflammatory)
• JSON may be something like 100x faster to parse than XML in today's browsers (but I doubt very much if the best JSON parsers are anywhere near that much faster than the best XML parsers ... it would be interesting to know!),
• JSON parsing ends up with something akin to a  typed "business object" rather than an untyped DOM tree
• To do that in XML requires yet another layer or two of cruft (a schema and a databinding tool)
• The bottom line argument for JSON comes down to elegance -- it does what is does simply and cleanly, and simply refuses to worry about many of the things that complicate XML such as metadata (attributes), comments, processing instructions, a schema language, and namespaces.

As a 10 year XML veteran, and informal minister of propaganda for the "XML Team", aren't I supposed to leap to XML's defense?  I just can't summon the energy.  First, XML's lack of elegance, presumably the result of its genesis in a series of committee rooms and compromises, is unarguable. One just can't look at it without remembering the old saw that a "camel is a horse designed by a committee." I'm not sure if that's a problem, since I like the comeback "but a camel is a lot more sturdy beast if you are exploring the unknown."  Still, horses' elegance generates a lot more enthusiasm in the rest of the world than camels' sturdy pragmatism does.

Second, it's also hard to argue with the proposition that the XML wheel could be a lot rounder.  As far as I can tell, there is just about zero enthusiasm among XML cognoscenti for re-opening the debates from a decade ago about which aspects of XML (or SGML) are "features" and which are "bugs" -- it depends on who is doing what, and whether there are enough people doing it to matter.  By making evolution toward something more simple and secure impossible, the XML community made something like the JSON revolution inevitable.  That doesn't mean that the revolutionaries will win;  after all, JSON's own limitations will become apparent only once is tested in scenarios that were not anticipated by its designers . At a minimum, a few key victories for the revolutionaries might motivate the old guard to make some needed reforms.

Third, the fact that the argument for JSON comes down to a matter of elegance doesn't bode well for its ultimate success.  It's hard to forget the famous lament of a LISP devotee reflecting on its lack of success against the much less elegant C/Unix/etc. competition: Worse is Better.  I wonder how many of the legitimate challenges that people are finding easier to solve with JSON than XML (especially the infamous cross-domain data mashup problem imposed by XmlHttpRequest's security model) mightn't be solved more expediently with some small tweaks to the XML infrastructure rather than a wholesale adoption of a disruptive innovation such as JSON. 

Finally, in the larger scheme of things it doesn't matter.  What does matter is that there be standardized, widely supported means for making data interoperable across applications, platforms, programming languages, and time.  Life would be easier for us infrastructure implementers if there were a single, stable standard, but it's unrealistic to expect that XML 1.0 would be the last word on the subject.  We will cope with whatever happens -- small tweaks to address critical bugs that JSON illuminates, multiple de facto data interoperability standards,  guided evolution of XML to be a better universal data interchange format, or wholesale revolution to produce a better world.

It would be nice if the JSON vs XML debate does not go the way of the "REST vs Web Services" perma-talking-past-one-another-fest.  Ultimately they are very likely to end up in the same place.  As Eric Newcomer puts it:

once a technology passes the hype cycle and becomes adopted, all its warts and bumps become more obvious as we find out what it is really good for, and what it is not...we should no more propose Web services as the right solution for everything than we should propose REST as the right solution for everything.

I hope nobody proposes JSON as the right solution for everything, and that the debate reminds us that XML is not the right solution for everything. One way or another, this debate will take us toward either a couple of rounder wheels better fit for their specialized purposes, or inspire a next-generation wheel that really does better than either do well today.

 Quick update with some other interesting links: (Winer's piece made the top of Techmeme while I was typing ...)

Dare Obasanjo

The obvious reaction was to make the Google and del.icio.us announcements into a REST vs. SOAP or XML vs. JSON story since geeks like to turn every business decision into a technology decision. However if you scratch the surface, the one thing that is slowly becoming clear is that providers of data services would rather provide you their data in ways they can explicitly monetize (e.g. driving traffic to their social bookmarking site or showing their search ads) instead of letting you drain their resources for free no matter how much geek cred it gets them in the blogosphere.

Simon Willison

The sweet spot for JSON is serializing simple data structures for transfer between programming languages. If you need more complex data structures (maybe with some kind of schema for validation), use XML. If you want to do full blown RPC use SOAP or XML-RPC. If you just want a light-weight format for moving data around, JSON fits the bill admirably.

What do we lose from not using XML? The ability to use XML tools. If you’re someone who breathes XSLT that might be a problem; if like me your approach when faced with XML is to parse it in to a more agreeable data structure as soon as possible you’ll find JSON far more productive.

Potential at the Trailing Edge

Lots of people linked to the happy news last week that Jon Udell was joining Microsoft, so I didn't bother.  I have previously recommended  his great interview with Anders Hejlsberg.  This is a clear, concise, hands-on demonstration of LINQ (including LINQ to XML) that feels like Anders stopped by your office to explain it in person.  I have to credit the interviewer for a lot of that.

I met Jon Udell at the XML 2003 conference, and blogged about it in my previous home on the web. I remember one interaction well: I had (a year or so before joining Microsoft) just bought a Mac Powerbook and noticed that he and most of the other really cool people were also using one.  I asked if this the first sign of a trend toward OS X... he replied something like "no, just a fad."   A couple of years later I noticed that many of the  bleeding edge people seemed to be switching to Ubuntu Linux and others were agonizing about whether to switch  ... so I guess he was right!  And I've given the PowerBook to my daughter and gone back to Windows.  Oh well.  I can't exactly pull off that Mac Guy style with my age and physique anyway.  Maybe I'm worse than uncool; I use a Mac Mini at home now ... but commit the heresy of running Windows Vista on it.  [The Mini is a phenomenal piece of equipment: very small physical size, lots of horsepower, extremely quiet, relatively cheap ... I wonder how much of Apple's newfound market share comes from people like me and some colleagues that I know who  buy their hardware and run our software?.]

Anyway, the tagline here comes from a piece in Udell's new weblog:

Part of this career change goes beyond switching employers. The disconnect between the geek world and the civilian world has really been bugging me lately. Leading edge aside, there’s so much potential at the trailing edge that languishes because nobody helps people connect the dots. ,,, Smarter methods of communication. More powerful data analysis and visualization. Surprisingly simple kinds of integration.

That really resonated with me.  It is similar to the LINQ to XML vision in that we see lots of people in the "civilian world" who could benefit from the data interoperability that XML enables but don't have the tools to use it effectively, and no particular interest in learning a whole set of specialized XML technologies just to do a single job.  Although I would hardly call it "trailing edge", LINQ is all about connecting the dots between existing programming languages and the different data sources that one comes across.

More generally, and more to the point of what Udell will be doing, there does seem to be a problem in our rather fad-driven industry:  Lots of people are contending over who gets to lead the parades, but not very many are coming along behind to sweep up after the horses and elephants. I can't blame the "civilians" for wanting to avoid those streets until nature does its slow but sure cleanup job, but there are a lot of good ideas that have been generated and proven (not to mention bad ideas that have been discredited) in the debris of yesterday's parades. The example closest to my heart is XML I suppose - it's no longer hip (and the hipsters are off exploring new ideas such as JSON), there is a lot of, uh, compostable material left behind that needs to be disposed of properly, but there's much of value if  users can get the kind pragmatic advice on what works and what is overkill that Udell does so well.

Bleeding edge geekdom is lots of fun, but working on the trailing edge has a pleasure all its own.  What works today will probably still work tomorrow, but it can work better if we incorporate the formerly bleeding edge stuff that actually proved itself.

The Model T and the Prius: Simplicity vs Complexity, yet again

My favorite conundrum, the difficulty of being simple, pops up everywhere I look these days. 

• OpenXML document format vs the Open Document Format
• Point: OpenXML is so complex no one else can implement it.
• Counterpoint: Its complexity is due to the existing features of MS Office, which are reflected in the old binary format and faithfully preserved in OpenXML.
• W3C XML Schema vs RELAX NG
• Point: XSD is hard to read, hard to write, hard to understand and should be scrapped. 
• Counterpoint: Namespaces, URIs, DOM, and others are also horrible specs, but real people use them all to get the job done.
• WS-* vs REST
• Point: Stand up to the forces of WS-Complexity and speak truth to confusion.
• Counterpoint: Or stick with WS-* if you value security, your career, and other such boring enterprisey stuff.
• XML vs JSON
• Point:   XML is not well suited to data-interchange because is ugly, inefficient, and doesn't match the data model of most programming languages.
• Counterpoint: XML tools are pervasive; the best ones don't force developers to confront random complexities unless their features are actually required.

I had an epiphany shortly after buying a Toyota Prius recently:  The thing is an absolute Rube Goldberg machine under the hood, but is if anything simpler to operate than an ordinary car. By contrast, the Ford Model T was so simple in design that it came with all the tools needed to maintain and repair it, and a generation or two of backyard mechanics learned how to do just that.  It got 25 miles per gallon, better than Ford's typical product today. The Prius, on the other hand, is so backyard mechanic-unfriendly that the 12 volt battery is sealed away under the trunk; there is apparently no way even to use it to perform that one bit of auto repair that almost anyone can do, jump-start another vehicle.  But that complexity returns a big benefit - a Prius has all the interior size and environmental comforts of a conventional midsize sedan, but twice the fuel efficiency of the simple, uncomfortable Model T (and 3-4x that of Ford's current best seller).  The complexity is mostly hidden away from the driver; the only  leaks in the abstractions that make it look like an ordinary car are the tricks one can use to optimize fuel economy.

I noticed that Len Bullard recently made essentially the same point I am trying to make:

So it is with the modern combustion engine: it works, it is efficient with respect to the power delivered for the fuel and the weight of the overall vehicle, thus adding more efficiency, but it is not something that the average backyard mechanic can rebuild. The tools required, the parts required and the knowledge of how they work in combination exceed his or her resources, understanding and nerve.

Complexity is just as much in the nature of the evolution of systems as simplicity is desirable. At some point, the demands of an environment or a market require complex solutions. While simplicity is a goal, it can also become a religion just as harmful as fundamentalism when pursued with a sword. Complex systems can do what simple systems cannot do. The goal that all systems be accessible can be met with open standards, but the goal that they be powerful, workable and light might not even as the principles over which they are built remain the same.

Now Donald Norman, the guru of high design, weighs in:

“We want simplicity” cry the people befuddled by all the features of their latest whatever. Do they really mean it?  No.

But when it came time for the journalists to review the simple products they had gathered together, they complained that they lacked what they considered to be “critical” features. So, what do people mean when they ask for simplicity? One-button operation, of course, but with all of their favorite features.

Joel Spolsky elaborates:

In the case of the iPod, the way beauty is provided happens to be through a clean and simple design, but it doesn't have to be. The Hummer is aesthetically appealing precisely because it's ugly and complicated.

I think it is a misattribution to say, for example, that the iPod is successful because it lacks features. If you start to believe that, you'll believe, among other things, that you should take out features to increase your product’s success. With six years of experience running my own software company I can tell you that nothing we have ever done at Fog Creek has increased our revenue more than releasing a new version with more features

Gadgetopia summarizes -- "you need features, but they need to appear  simple to the end users". 

There's a name for the discipline of working within a network of complex constraints to produce something simply usable - engineering.  It's not easy to get the complicated system of batteries, motors, charging systems, and drivetrain in a modern hybrid car to work together efficiently, but some very clever engineers managed to do that.  Those who dismiss various XML technologies, or XML itself, because of ugly complexities or unpleasant constraints may someday look as prescient as the folks in the auto industry who killed off the electric car in the 1990's, only to be humbled by the engineers who made the hybrid system practical a few years later.

OpenXML, WS-*, XSD, and XML for that matter are not doomed because they are complex, nor are they destined for success because they attempt to hide their complexity from the end user.   Those who build infrastructures and applications on top of them are challenged to do good engineering to make their purported features actually work and to appear simple to the ultimate consumers.  In the long run we'll probably end up more or less where automobiles are now - complexity under the hood that would inspire Rube Goldberg, but engineering quality that makes it appear simple to the driver.

Reasonable people can disagree on whether we are currently on the right track to making this happen.  If we are, the yelping in the blogosphere might be dismissed as backyard mechanics lamenting the fact that "enterprisey" applications can't be built with hand tools anymore.  If the critics are right, we could plausibly be in the midst of a disruption akin to the 1970's consumer revolt against Detroit-made gas guzzling behemoths in favor of smaller, better-designed imports. We shall see, but the answer is not at all obvious despite various "Mission Accomplished" declarations by one faction or another.

 Here what  I (personally ... obligatory disclaimer about not speaking with the xmlteam hat on) think about the technologies listed at the top of this post with respect to the simplicity/complexity dilemma:

XML 2006 Observations

I could only attend half the conference due to a family health issue, but here are some thoughts on what I did see.  The links are mainly to the conference program; I believe the entries will eventually link to the actual presentation slides and submitted papers.

Roger Bamford’s keynote  spent much time showing how the high-level architecture of a typical enterprise application 30 years ago was not that much different than today:  3270 pages vs HTML pages, CICS transaction monitors vs app servers, and a back end DB that is a scarce resource that the rest of the system tries to offload.  There was a discussion of how “share nothing” approaches worked then and now to achieve scalability, but he noted that people who truly understand this were scarce then and remain so now. 

He got to the main point by arguing that today’s real problems have much to do with a single DB having to support applications with very customized views via side tables to define extended attributes that apply to a customization but are not in the core schema.  He noted that XML is more extensible than SQL, and proposed that this core problem could be solved more easily by storing customized data an XML store.  That implies XQuery as the query  / middle tier programming language … or actually “XML + XQuery(P) + Apache + REST == no middle tier” 

His vision of how to bootstrap this seems to be:

- Bamford (personally?) has started something called the FLWOR Foundation  to produce an open source XQueryP implementation called Zorba.

- Oracle’s main DB product will be completely interoperable with it at the XQuery code level

- The people with low end requirements use Zorba, and those with enterprise-scale requirements use Oracle.

- I guess the rest of the world will re-engineer to be code compatible with Zorba "correctly" implement the XQueryP standard, and thus we get interoperability.

Very interesting, especially since I bought into this basic vision big time in 1999, which motivated me to take a job at Software AG because of their Tamino XML database.  That hasn't worked out particularly well for Software AG, but obviously Oracle has immensely more resources, and maybe the time is ripe now even if it wasn't then.  It's quite different from Microsoft's vision, however.  We stress the interoperability of data more than attempts to create standards for portable code.  We work hard to make our platform and tools the best in the industry, and the fact that many are proprietary is not a drag on interoperability since it is standard data -- which generally means XML these days -- flowing across applications and platforms.    A distributed application may use Java and XOM on one node, Linux and XSLT on another, and Windows and LINQ on another ... and any of them may use XQuery to get data out of a database ...  but so long as they speak XML to one another each node has no need to care how the others produce and consume the XML.  Let the most productive tool win!

I spent the rest of the first day chairing the Enterprise XML Computing sessions. First up, Dana Florescu and Ralf Lämmel shared a session that I introduced as “starting from different ends of the declarative / imperative continuum and ending up in about the same place”.  That is, the XQuery people start with a declarative query language and are adding imperative features to make it more easily usable for scripting, etc., whereas the .NET people are taking imperative languages and adding declarative query features to make applications more scalable / streamable / parallelizable / etc.  Dana talked about XQueryP and gave a great presentation, with concrete use cases, realistic assessments of the competition (including LINQ to XML!), etc.    Ralf talked about how the LINQ to XML team is thinking about extending LINQ to XML to gracefully handle streaming use cases.

There were some good audience comments -- and Dana and Donald Kossman met with me later to reinforce the point -- arguing that XQueryP can do all that stuff in less code (just one nice compact string rather than all those method calls), and will have [someday] an optimizer to figure out whether a particular query can be streamed or not.  I think we agreed at the end that we are aiming at different audiences – XQueryP advocates seem to be going after the people who see XML as the center of their data world, like to learn new languages,  and demand proper computer science solutions;  whereas with LINQ to XML we go after people who use all sorts of data, want to change one thing at a time, get a lot of help from Intellisense to build queries rather than having to type in an arcane string, and not necessarily worry too much about the underlying theory.  LINQ to XML is "only" an API and not a declarative language processor / optimizer so we have no option but to ask the programmer to decide whether to write a query over a loaded tree or over a dynamic stream.    

Next, Ralf and Igor Peshansky of IBM gave what amounted to compare/contrast talks on how strongly typed XML and web services features are being incorporated into the .NET and Java languages. If you read Ralf's blog you know what we are doing. See http://alphaworks.ibm.com/tech/xj  for the Java side of things.

My summary (as chair, trying hard to be neutral) was:

Similarities:

- Both start from the position that current XML processing is too hard and the programming language – XML impedance mismatch needs to be bridged.

- Both provide a more OO-ish experience to their users than is possible with existing XML APIs.

- Both provide automated facilities to map XSD types into language types

- Both can be thought of as competing with XQuery in the programming environment, but don’t undermine XQuery’s value as a query language. (Apparently the XJ folks get "I thought that's what XQuery does" questions a lot too!)

Differences:

- LINQ is being put into the core of .NET whereas XJ essentially creates a domain specific language for XML on top of Java.

- LINQ is the query language, XJ incorporates XPath as the query language

- LINQ provides a common programming model for objects and XML, XJ is disconnected from ordinary Java objects, i.e. you can't query over object graphs or join across XML objects and ordinary Java objects. 

- C# doesn’t incorporate XML syntax into the language, XJ (and VB9 of course) does.

- The LINQ framework doesn’t have any special support for web services, XJ does (that was the focus of Igor’s talk).

In the last session of the day, Paul Downey described the work of the Schema Patterns for Databinding WG and how it’s doing useful work (but don’t get no respect).  One notable quote:  “databinding tools don’t all suck, they just suck in different ways so they don’t interoperate.”  I'm just not in a position to know much about or comment on Microsoft's thoughts on this ...

Finally, there was a panel discussion of XML Pipeline Processing.  Sam Page described the concept and some real world use cases. Norm Walsh described XProc standardization.  The canonical use case is for applying the numerous inclusions, validations, and transformations needed to build the DocBook documentation, and having the different people who work on different platforms be able to get the same result.  Norm did a good job of deflating some overheated expectations (e.g. that it is a minimalist alternative to BPEL).  Still, having attempted to evangelize an XML pipeline product while at Software AG, IMHO the principal response in the real world is going to be: “I can do that in 10 lines of C#/Java/Python/Perl/whatever, why on earth should I do it in 100 lines of XML?”  Also, the WG has agreed to disagree on what the underlying data model might be, so an actual pipeline may consist of nodes that exchange XML text, DOM trees, PSVI serializations of some sort, or whatever. Thus, anyone wanting to write a pipeline component will have to write it differently for every supported implementation… although pipeline end users could write one “script” and have it produce the same result no matter what the implementation.  Nevertheless, this looks like a model W3C effort: they're standardizing existing practice rather than doing a "design by committee", finding the intersection of what is known and needed rather than the union of everyone's hopes and dreams, and working in a fast and focused manner.  It looks like it will do one thing well, and if you need that one thing done, this will be the spec for you.

The only presentation I saw on Wednesday was Douglas Crockford’s “JSON: The Fat-free alternative to XML” (Slides are already online at http://www.json.org/xml2006.ppt).  A couple of things I took away from the talk:

- JSON took a very different approach than XML in a couple of areas: there is no version number because the spec is declared stable forever; it takes a non-draconian “be liberal in what you accept and conservative in what you produce” philosophy to be friendly to supersets (such as YAML www.yaml.org, which is big in the Ruby world)

- The JSON folks are agitating for the infrastructure vendors to support a JSONRequest API http://www.json.org/JSONRequest.html  that addresses some limitations in XmlHttpRequest for AJAX environments. 

- The JSON people have an interesting response to the question of why it doesn’t have namespaces, which echoes the feelings of many namespace-haters in the XML world:

• Every object is a namespace. Its set of keys is independent of all other objects, even exclusive of nesting.
• JSON uses context to avoid ambiguity, just as programming languages do.

- There is a JSONT spec that defines a way to declaratively transform JSON to HTML, XML, etc. http://goessner.net/articles/jsont/

I'm not sure if Crockford was positioning JSON as a better-XML-than-XML for data interoperability across applications. I had thought of JSON as “the real X in AJAX”, i.e.  for communicating between browser and server in a Web application, but not a serious contender for interoperability purposes.    Will we see JSON substituting for XML in SOAP messages, RSS feeds, etc.?   Will people store JSON persistently? That will require JSON-flavored APIs, schema/”data contract” languages, query/transformation languages, apps, etc.  If this starts to get momentum (I’m talking about momentum as a data interop and storage format, not the momentum it clearly has for “Web 2.0” server-browser communication), things could get interesting.  I'm not even sure how I feel about that ... Maybe the time is ripe for the stuff we talked about on the sml-dev mailing list years ago.  YAML evolved out of that list, and JSON is essentially a subset of YAML.  I'm sure that lots of us at in the MS Data Programmability team are pondering what we should think and do about JSON.   More later.

Rough Spots in the LINQ to XML Learning Curve

[minor editorial updates 11/13] 

We've been doing some formal usability testing on all the LINQ components over the last couple of months and have learned a lot about what people find challenging. The results have generally validated LINQ's story as a common programming model for all types of data, but they've also identified some things that people find hard.  Some of these may be fixed with API tweaks, but for the most part they indicate what we have to do a better job of explaining.

I've come up with the following list of things to keep in mind when working with LINQ to XML. Some will be explained in this post, others in subsequent posts, and we'll try to make sure that the documentation calls them out.

1. Clear your mind of SAX and DOM when approaching LINQ to XML; there are many similarities of course, but they can lead you astray.  XPath, XSLT, and XQuery are closer in spirit to LINQ to XML, but of course the details are very different.
2. Remember that XElement and XDocument are similar, but not interchangeable when you are loading a document from a data source.  XElement.Load() loads everything under the top-level element, XDocument.Load() also loads any markup before the top-level element.
3. You must grok IEnumerable<T> to use any of the LINQ tools effectively; this abstraction plays essentially the same role in LINQ as the relation does in the relational model.  In LINQ to XML, its is the axes of the XML tree, not the nodes of the tree, that expose IEnumerable and hence can be queried.
4. People who see an IEnumerable have a tendency to do a foreach to iterate over it and use lots of if statements to decide what to do with the data values. Avoid that temptation (except when foreach is unavoidable, e.g. when writing to the console), and use the standard LINQ operators to select, filter, sort, join, group, etc. the data.  Once you get used to this discipline, we believe you'll be more productive and your code will be more understandable, and more likely to be correct.
5. In most scenarios we have investigated, there is a simple, clean, and CORRECT way to do what you need to do with LINQ queries and functional constructors.  If you find yourself writing a lot of tricky imperative code to deal with straightforward XML data, step back and decompose / refactor / reconsider. Remember that the execution of queries is deferred until you actually get the data, so there is no performance penalty to decomposing a complex query / transformation into understandable pieces.

One overall pattern we've detected is worth noting: The less you know about current XML technologies, the faster you learn LINQ to XML.  The intuition of participants in our studies who know DOM, SAX, XmlReader, etc. is that the problems we posed them should be hard to solve, so they tended to roll up their sleeves and start writing a bunch of imperative code. Those who didn't know much about XML except that it is a tree of elements and attributes could use their SQL and C# intuition and found more elegant solutions.  On the other hand, the really experienced developers fairly quickly learned the core LINQ design patterns and began to apply them effectively to new problems. This was very heartening, since the core LINQ to XML value proposition is supposed to be make XML accessible to mainstream developers who don't want to learn the complex details of its syntax or master the numerous XML processing tools that exist today.

Getting to the less cheery news, there is plenty of stuff we need to work harder to explain, and to think about making easier in the API.  One thing that trips up a lot of people (ahem, including moi as I was working on this post!) concerns the XDocument and XElement classes.  To illustrate this, consider the MSDN blogs RSS feed as an example.  Its top-level format (stripped down a bit for illustrative purposes) is:

<?xml version="1.0" encoding="UTF-8" ?>
<?xml-stylesheet type="text/xsl" href="rss.xsl" mce_href="rss.xsl" >
<rss version="2.0" xmlns:dc="http://purl.org/dc/elements/1.1/">
<channel><title>MSDN Blogs</title>
<description>The Blogs of MSDN</description>
<item><title>Develop Mental: Game Camp</title>

In general, use XElement when you just need to load the tree of elements, and XDocument when you are interested in any doctype information, PIs, etc. between the top of the file and the first element.  For example, if you want to get the xml-stylesheet processing instruction, load the XDocument:

var feed = XDocument.Load(@"http://blogs.msdn.com/MainFeed.aspx");
Console.WriteLine(feed.FirstNode);

This will display the value of the xml-stylesheet PI.  On the other hand, ...

var feed = XElement.Load(@"http://blogs.msdn.com/MainFeed.aspx");
Console.WriteLine(feed.FirstNode);

... will load only the <rss> element and display the contents of the first node beneath it, i.e. the <channel> element, but any markup before the <rss> element is thrown away.  That has a somewhat counterintuitive side effect:

feed.Element("rss")

returns an XElement with the name "rss" when feed is populated via the XDocument object's Load() method, but null when it is populated via XElement.Load().   

Second, it was not always obvious how to apply the LINQ approach based on IEnumerable<T> to XML trees. The answer is that queries work over the axes of a tree very much like XPath uses axes.  [1]  Thus, a big part of writing a clean and correct LINQ to XML query is in choosing the best axis to query over.  For example, if you are querying over the <item> elements in an RSS feed, you could do it by brute force by "dotting into" the element hierarchy (in this example, we loaded the XDocument, so you have to include the top level <rss> element) and then enumerating over all the <item> elements:

var feed = XDocument.Load(@"http://blogs.msdn.com/MainFeed.aspx");
var items = from i in feed.Element("rss").Element("channel").Elements("item")
          select i;
foreach (var i in items)
          Console.WriteLine(i.Element("title"));

 Or you could more conveniently use the Descendants() axis (and this will work whether we loaded the XElement or the XDocument):

var feed = XDocument.Load(@"http://blogs.msdn.com/MainFeed.aspx");
var items = from i in feed.Descendants("item") 
           select i;
foreach (var i in items)
          Console.WriteLine(i.Element("title"));

The former of these examples illustrates another point of confusion, the distinction between Element() vs Elements() and Attribute() vs Attributes().  The current naming scheme is that the singular form returns the first matching XElement or the only matching XAttribute object; the plural form returns an IEnumerable over all the elements or attributes.  This is, to be frank, creates a dilemma: The naming scheme is quite logical and aligned with English semantics, but can be confusing to real humans who don't necessarily see that they've typed "Element" when they meant to type "Elements".  Intellisense can lead people astray here as well; people tend to  grab the first option that Intellisense presents, which may not be the one they really need.  We could of course make the method names more verbose and less easily confused, e.g. "Element" vs "AllElements" or "ElementsAxis".  Some of us guess is that this verbosity will help people during the learning process, but annoy them for the rest of their careers ... like the much-hated DOM method GetElementsByTagName() does.  We would definitely appreciate some real world feedback here! 

Next time I'll dig more into that challenges that people run into when using functional construction to create or reshape XML, and the power that can be achieved by combining the LINQ operators and functional construction. 

[1]  OK, maybe it would be better to say that the more you know about imperative XML APIs, the faster you'll learn LINQ to XML .... but knowledge of the more declarative tools such as XPath/XSLT/XQuery is probably a benefit.

Declarative vs Imperative Streaming Input in LINQ to XML

Oleg Tkachenko has a nice post comparing the StAX (java) and XmlReader (.NET and XmlLite) approaches to streaming over a potentially large XML data source and filtering out unwanted elements.  He concludes:

if you work with StAX you can readily work with .NET XmlReader and the other way. Great unification saves hours learning for developers. I wonder if streaming XML processing API should be standardized?

We've been discussing how to add streaming capabilities onto LINQ to XML for some time now.  The value proposition is something like: Our target audience will sometimes encounter large documents or arbitrary streams of XML; they want the ease of use that LINQ to XML offers, but they don't want to have to load an entire data source into an in-memory tree before starting to work with it.  They could use XmlReader, of course, but that is a considerably lower-level API that requires attention to all sorts of details of XML syntax that we know mainstream developers don't want to worry about. Let's offer some easy to use methods to allow LINQ to XML users to load a well-structured XML data source in definable chunks that can be worked on one at a time.

The obvious way to do this is imperatively , much like StaX or XOM does: the user writes a filter function / subclass and the XML API uses that to determine which elements in the XML source to pass through to the calling application.  We think, however, that the better way is to do it more declaratively -- specifying what to do rather than how to do it.  We're not ready to publicize a specific streaming input API, but let's talk about why it's worthwhile to avoid the easy (and arbitrarily powerful!) imperative filtering approach. 

Consider, for example, Eric White's post on using the querying style that LINQ supports rather than the traditional imperative approach to process large text files. He follows up with another post explaining why he thinks this is so cool. Taking Eric's points and elaborating / extending them a bit, here are some concrete reasons for using the declarative / functional style rather than the imperative style: 

• Non-imperative application code is likely to be more easy to test and debug because the execution path depends on explicit inputs rather than on some funky internal state-driven logic.
• The less imperative, the more likely code is to be modularizable / refactorable as requirements change and code evolves, again because there is less need to pass that funky internal state around.
• The more purely declarative / functional, the more easily code can be mapped into some other more declarative language such as XQuery, XSLT, SQL, etc.  This might allow someone (maybe you, maybe us, maybe a third party) in the future to leverage LINQ expression trees to pass queries around and have them evaluated efficiently on some combination of the client, midtier, and server.  This is the design philosophy of LINQ to SQL, and there's no intrinsic reason it could be applied to "LINQ to XQuery" or "LINQ to XSLT" ... *if* the logic isn't tainted by a bunch of imperative statefulness.
• In the future, this analysis can be automated: PLINQ is (probably?) coming in a subsequent version of the .NET framework. As Joe Duffy puts it, this will mean that code written to the initial release of LINQ that uses "filters, projections, reductions, sorts, and joins can be evaluated in parallel...