The world has had a chance to look at the LINQ technologies a bit, and the overall response has been quite enthusiastic.  But now we're getting some harder questions about how various LINQ components relate to one another and to existing Microsoft technologies.  Let's compare XLinq --- oops sorry, LINQ to XML -- with some alternatives to tell the story of why we are building it and what that means for those who know and love the current generation. 

One basic question keeps coming up, something like: "We have SAX, DOM, XmlReader/Writer APIs (and the Java people have a bunch more), we have XSLT, we have XQuery ... why do you think we need Yet Another XML API?"

First, XML is just too hard  -- or at least too tedious -- to use for mainstream developers today.  Non-specialists using XML today tell stories that remind me of various home improvement projects gone awry.   As a weekend home improver, I want to focus on making things look nice and do useful things, not on the slimy details of connecting tubes of different materials, diameters, and states of corrosion together in a way that doesn't leak.  Likewise, people using XML to improve their software's interoperability generally want to focus on the areas they know and care about, and not on learning the subltle points of one API for efficient large document reading, another API for convenient small document tweaking, another technology for structural transformation, and yet another for database access. 

That's not to say that we are designing LINQ to XML (I'm just going to abbreviate that "XLinq" below until there is a PR-approved shorthand) to replace XmlReader, DOM, XSLT, and XQuery.  These will all live on, somewhere in the Microsoft stack, for the forseeable future:

• XmlReader / XmlWriter can't go away because XLinq uses them to parse and serialize between XLinq objects and XML text. Also, while we are making XLinq as streaming-friendly as possible (see the XStreamingElement class in the CTP release for a taste of where we are going), we're only aiming at hitting the 80/20 point.  We expect that only those with straightforward formats and common use cases will use XLinq to meet their streaming needs, and the "20%" will continue to use XmlReader / XmlWriter.
• DOM can't go away because there are important use cases for API-level interoperability, most notably in the browser.  Also, we have real customer scenarios where the (at least theoretical) ability to port code back and forth to Java is important.  DOM doesn't make code truly interoperable across implementations (especially on other languages), but there is enough conceptual similarity that porting is generally not terribly difficult.  Finally, XLinq is not trying to be all things to all people who use XML, but focuses on making mainstream use cases easy and efficient to implement. Although the XLinq spec is not final, but there will probably be scenarios where XML text fidelity is important (e.g, character entities are round-tripped) and DTD features (e.g. external entities) are used in which DOM is just more suitable than XLinq.  
• XSLT definitely won't go away. The Microsoft XML team was promoting XQuery as a "better XSLT than XSLT 2.0" a few years ago (before I came, don't hurt me!), and got set straight by the large and vocal XSLT user community on why this is not going to fly. While it may be true in some abstract way that XQuery or XLinq might logically be able to do everything that XSLT does, as a practical matter it won't.  Most obviously, XSLT a "real" standard and supported on most platforms; if you need to write XML processing code with a high confidence that it can be made to work on the client and the server, on Windows, OS X, Linux, etc., and easily co-exist with application logic written in C++, C#, Java, or a scripting languge, then XSLT is probably just the ticket.  Even if you are committed to the .NET platform and could take an XLinq dependency, there are jobs that are simply easier to do in XSLT than XLinq, especially when the XML data being processed is loosely structured and deeply nested. XSLT's recursive template matching paradigm is very well suited for that kind of data, and XLinq has not been designed with that type of data in mind
• XQuery won't go away, at least for its original use case as a database query language.  Microsoft supports a draft of XQuery in SQL Server 2005, contributes to the work of the XQuery working group at W3C, and will continue to invest in finalizing the XQuery Recommendation and implementing it in our DBMS. We're more skeptical about XQuery as a data integration or XML programming language, but remain open minded since some partners (e.g. DataDirect) and competitors (e.g. Oracle) seem quite enthusiastic about this direction for XQuery.  We'll be looking to our customers for guidance as to how to allocate resources between LINQ/XLinq and XQuery for these client/mid-tier scenarios

Second, there have been quite a number of developments in programming languages since the SAX and DOM APIs were designed almost a decade ago.  Back then, cross-platform APIs based on CORBA were "hot", interfaces/factories often preferred to concrete classes, and few widely-supported and understood collection mechanisms other than the array. Now RPCs are in disfavor, concrete classes generally preferred, and generics introduced to make it much easier to process collections of objects in fairly standardized ways.  Furthermore, additional features such as nullable types and functional constructors are in widespread use and APIs are expected to support them.  We believe that an XML API designed with with these "modern"  tools and conventions in mind will be more familiar and usable to today's programmers than APIs designed with 1990's conventions in mind.

Finally, we believe that the overall LINQ story is going to have a pretty profound impact on data programmability, and we want to make sure that LINQ has a good story for XML.  Serious thought was given to the challenge of LINQ-enabling the existing APIs, but no satisfactory designs emerged.  LINQ adds features to C# and VB that were quite openly borrowed from more "declarative" languages such as SQL and "functional" languages such as Haskell.  This may give purists some heartburn, but we believe it will give mainstream developers lots of new power in a familiar package. For XML users, I see a few really powerful implications:

• The ability to query data by declaraing the characterics of the result set rather than imperatively navigating through and filtering out all the data.  This approach was pioneered by relational DBMS 30 years ago, and now is available for all types of data -- hierarchical XML, object graphs, and relational tables.
• The ability to join across diverse data sources, be they XML documents, objects, or DBMS queries
• The ability to "functionally" reshape data within the same language as the application is written.  XSLT pioneered the functional transformation approach to XML processing, but it is difficult for many developers to learn and requires a processing pipeline architecture to combine XSLT transforms with conventional application logic.  XLinq's functional contructors bring the power of this approach to the mainstream; as noted above, there are definitely scenarios where XSLT experts will be able to exploit the power of this approach more easily than can be done with XLinq, but we hope to allow non-experts to exploit the power functional transformation for common, regularly structured data.

We get a bit of rude Q&A on these points:

Why not just fix the existing APIs? One hard lesson I've learned as PM for the DOM APIs is that it is almost impossible to accept the suggestions we get to improve the conformance, performance, usability, etc. of the existing APIs without breaking existing applications. Just adding new features without breaking the behavior of old ones creates even more confusion and complexity

Why have you re-invented XQuery? It's the standard, just support it as your XML programmability story!It's true that XLinq has ended up being (at least from a high level view) a superset of XQuery, and we have applied quite a bit from the XQuery spec (and the experience of working on it) to XLinq. Simply adopting XQuery, however, has several drawbacks:

• It's still a work in progress not a real standard; it's close to a W3C Recommendation, but real-world acceptance, understanding of best practices, and validation of its design decisions is some years in the future at best
• XQuery has no standard insert / update / delete sub-language. It's good for creating new documents that can be thought of as updates of their inputs, but cannot (in the Candidate Recommendation) update documents in place. While we like the functional transformation design pattern, there are many use cases that require in-place data manipulation of the sort that people use DOM more than XSLT for today.
• XQuery itself creates yet another XML data model and type system, thus creating even more impedance mismatches. Many see XQuery as a solution to the XML - SQL integration challenge, but it does so by providing a superset model to which existing data can be transformed. XLinq builds on the LINQ approach of building on the generic IEnumerable<T>, which can be implemented for most data without transformation or mapping. Likewise LINQ leverages the .NET type system, semantics, functions and operators, etc. whereas XQuery has had to invent their own equivalents. 

Why will XLinq succeed in the .NET world when JDOM, dom4j, XOM -- which offer cleaned-up, modernized, native platform-friendly XML APIs -- have not displaced the DOM in the Java world I raised this on the xml-dev mailing list and got some very interesting responses. The essence of matter seems to be that a) the sophisticated Java developers who know about them do use one or more of these in preference to DOM; b) Java platform vendors, however, have not embraced or evangelized any of them; c) mainstream users often don't know they exist, or d) their managers discourage use of them because they are not considered "safe" alternatives. We don't think Xlinq will have these problems, simply because it will come from Microsoft and have the full weight of the company, its education / evangelism, and its support policy / staff behind it. We are confident that developers who are exposed to Xlinq will frequently choose it for their XML programming needs, and their managers can feel confident that this is a safe, supported choice.

Are you just smoking your own [insert mind-altering substance here] or do you have evidence that real people will find XLinq powerful yet easy to learn and use I can't go into much detail, but we have quite a bit of evidence. Indeed, we have a bit of a problem because so many Microsoft developers who are exposed to Xlinq want to use it before it is really ready to take a dependency on. One project taking heroic measures to work with the XLinq API; another discovered that its Xlinq-based prototype was about 1/3 the amount of code as the DOM prototype; and most agree that it makes XML almost a pleasure to work with.

Let us know what confuses or angers you about this story. I'll do my best to clarify what I can, and try to fix the rest.