Microsoft XML Team's WebLog : XML Futures

LINQ to XSD Alpha 0.2

XmlTeam — Thu, 21 Feb 2008 19:37:00 GMT

I’m happy to announce that today we are re-releasing the LINQ to XSD Preview Alpha 0.2 for Visual Studio 2008. The previous preview release of LINQ to XSD targeted Beta 1 of Visual Studio 2008 but did not work on later builds. Many of the people who originally downloaded and tried the LINQ to XSD Preview Alpha requested an update for the final release of Visual Studio 2008 – this release is it. You can download it from here.

The LINQ to XSD preview illustrates our initial thinking on a strongly-typed programming experience over LINQ to XML. Instead of working with untyped XML trees, LINQ to XSD allows you to program in terms of strongly-typed classes, generated based on an XSD schema. The LINQ to XSD project is still incubation, but we are looking at ways to incorporate its functionality into future Visual Studio and .NET Framework releases. We are very much interested in gathering your feedback on the product, so please download it, give it a try, and send us some feedback.

I know it’s taken us a while to get this release out and some of you have been wondering what’s happening with this project. The team has only recently resurfaced from the Visual Studio 2008 release push and has begun looking at how we can take LINQ to XSD forward again. We’re looking forward to getting your feedback on this preview and on our future work on strongly-typed XML programming.

Please send us your comments, questions, and suggestions via the LINQ forum on MSDN: http://forums.microsoft.com/MSDN/ShowForum.aspx?ForumID=123&SiteID=1

Shyam Pather

Dev Manager, Data Developer Experience Team

Chris Lovett Interview

XmlTeam — Fri, 16 Nov 2007 11:34:00 GMT

Chris Lovett was interviewed by book author Michael van Otegem recently and he asked some very interesting questions:

Please tell us who you are and what you do.
I’m an architect on the Data Programmability Tools team in SQL Server, and I work on XML tools that ship in Visual Studio. As an architect I do lots of different things including: product development on the XML editor in Visual Studio 2008; cross-group collaboration to make sure all our tools work together; playing with other concepts like XML Notepad; and thinking about and communicating our strategy and future directions for our technologies.

How long have you worked for Microsoft and what did you do before your current position?
I have worked on pretty much every XML core technology from Microsoft starting with MSXML in 1997, then to System.Xml in the .NET frameworks v1, and v2. and for the last few years I’ve been focusing on building XML tools in Visual Studio – for example, I was the primary developer on the XML editor in Visual Studio 2005. Before Microsoft I worked at IBM on OS/2 applications, then I joined the IBM Apple/IBM joint venture called Taligent, then I started my own company in Sillicon Valley with friends from Taligent during the height of the .com boom and that’s what led me to Microsoft.

A lot of the .NET Framework works with XML in some form or another. Why is XML such a key component and what were the challenges you faced because of that?
For one thing the .NET frameworks were designed during the peak of the XML hype curve, but more seriously, many folks at Microsoft were waking up to the fact that not everything had to be buried in code. Some things could be very neatly described at a higher “declarative” level and Tim Berners-Lee showed the world that “markup” is a great way to do that. So XML became the way to specify configuration information (.config files), and build information (MSBuild files), and setup information (WIX), object remoting with SOAP, and security permissions and so on. All of these domain specific uses of XML were then supported by our core System.Xml classes. It’s interesting to note that even those teams that didn’t adopt XML back in .NET v1.0 are fixing that. For example, we have the new AJAX work from the ASP.NET team and we have XAML in the Windows Presentation Framework (I’m a huge fan of WPF by the way. I’ve done a lot of UI development in the last 20 years and I have to say WPF totally rocks).   So XML is touching everything from database, management, communications, content publishing and now even into the user interface layer. I was amazed at the last PDC where just about every talk showed some snippet of XML somewhere during their talk.

I remember when we started XML at Microsoft most people thought we were crazy. The biggest challenge was convincing people the cost of parsing and storing verbose XML tags was worth it and our team has been working on performance, and scalability ever since. But the real reason XML become a key component of .NET, (and Windows, and SQL Server and Office) is because it achieved true cross-platform interoperability and because it was good enough for that job.

What do you (personally) like the most about XML and its associated standards?
Simplicity, cross-platform interoperability and huge adoption. The great thing about XML is that it is humble. It’s not trying to solve world hunger. Just invent your own tags, group them into structures that make sense for your domain and voilà. It is very simple and it is this same simplicity that helped HTML take the world by storm.   XML then improved on HTML by providing a clean separation between data and UI. It brought MVC to the masses so to speak.

The other day my 12 year old son was all excited and just had to show me what he discovered. He was editing Age of Empires XML files to tweak the behavior of the game using Notepad.    I asked him how he knew that he could do that and how he knew the XML syntax. He didn’t know what “syntax” meant, but he knew how to edit XML! Then the same week my doctor was all excited when he heard I worked on XML because he was involved in a software purchasing decision at our local hospital and it came down to their level of XML support. I couldn’t believe my ears.

The funny thing is that most programmers don’t really like XML. Probably because it doesn’t use curly brackets, so most programmers treat XML a bit like the ugly duckling. But the reality is that the whole world gets markup – to them markup makes our programming world more approachable. There are still way too many programmers that don’t get this.
What would be your #1 tip to people learning XML, XML Schema, XSLT and XQuery?
First of all I would say that XML 1.0 is the foundation. A must have. Can’t go wrong there, learn it, and learn how XML encoding works – that’s the number one issue people have with XML 1.0 – people don’t take the time to understand how UTF-8 encoding works which is a pretty important foundation to XML. Don’t worry too much about DTD, because we now have XML schema.

XML Schema (XSD) is a layer on top of XML 1.0. When you need a way to describe your XML structure in “standard contract” XSD is handy and most importantly – it’s there and it’s a standard. So don’t re-invent the wheel, but I’m not going to say that XSD is the be all and end all of data modeling, because it isn’t. A lot of things are missing, which is why people had to invent things like Schematron, and why Microsoft is working on EDM and SML and so on. Model driven development is now on the peak of the hype curve so I expect that modeling will be a battle ground for a long time to come. So take a pragmatic approach to XSD - use it if it fits your purpose. Some folks use other modeling approaches then have a tool that spits out the XSD – and that’s fine too.

Same goes for XQuery and XSLT – I think of these as being yet another layer above XSD. We did XSLT and XQuery because we figured that XML is data therefore people will want ways to query and transform that data. Makes sense, and I’ve done a lot of XSLT development, I still use it for specific tasks, but some things are a bit tedious. I find myself escaping to script a lot. XSLT 2.0 is a good improvement, but again, these things are not going to be the be all and end all of query and transformation languages.

For example, I’m a huge fan of the work they are doing in Visual Basic 9 with XML literals connected to our new Language Integrated Query (LINQ). It makes a lot of sense, because instead of having to “escape” to script, you just write the code you need right there in place – you have a complete general purpose programming language at your finger tips.   VB-XML integration is big leap forward for VB programmers and allows those developers, who may not be as familiar with the standard XML technologies like XSLT, to easily process XML data inside their programs. It’s a huge advantage to the VB programmer and we think it helps make VB an extremely compelling language for XML programming – it makes me want to write VB again, and I’ve heard many others say the same thing. However, it is VB-specific so development teams that need cross-platform interoperability at the query/transform layer are likely to stick to the standard technologies like XSLT and XQuery.

There’s a very interesting tension going on here where general purpose languages like VB and C# are moving up into declarative space with LINQ but not going all the way into declarative, versus SQL, XQuery and XSLT which are fully declarative with no side effects, which are therefore more optimizable, but sometimes rather incomplete as programming languages go and rather hard to author in some cases. I really don’t know how it’s going to end up. I think we should continue innovating on both approaches and see what happens. It should be very interesting.

As for all the myriad other XML standards out there, there’s a lot of hype that you have to sort though. To me it’s a funny thing to see programmers going to town making XML more complicated with layer upon layer of new concepts. I remember going to a conference and people were telling me “stop! – we can’t take any more”. There is genius in simplicity. I’m glad to see the renewed focus on simple REST-ful XML based services for this reason. If simple works, why complicate it. Conversely, if it isn’t simple, chances are people just won’t use it.

What can we expect from Microsoft in the future in the XML arena? Will support for XQuery 1.0 and XSLT 2.0 become part of Microsoft’s offering?
Microsoft is a pretty big place, so it’s hard for me to know all that is going on with XML across the company. But I do know about .NET, Visual Studio and SQL server. As I’ve mentioned before we are shipping the XML support in Visual Basic 9.0 with XML literals, XML axis properties and integration with LINQ to XML. LINQ to XML is our API which we are adding to the XML runtime in .NET 3.5, it is a new XML object model that is designed to work well with the Language Integrate Query capabilities of C# and VB.    We are also shipping some cool new features in the XML tools in Visual Studio 2008, including an incremental parser with extensibility API based on LINQ to XML that 3rd party XML designers can build on. We are also adding data breakpoints in the XSLT debugger and we have a new command line tool named “xsltc.exe” which takes XSLT and generates a .NET assembly which you can then deploy with your app instead of the XSLT source so you don’t have to compile XSLT on the server. Anton Lapounov has a great blog that talks about that.   There is not much else new in the System.Xml runtime because Visual Studio 2008 is essentially a service pack release of the .NET 2.0 runtime, so we’ve fixed some bugs there. We are also working on some XML features in Silverlight and we put up a preview of our LINQ to XSD work on MSDN. We are working on a new XSD designer and you will see more CTP's on as this takes shape.

As for XQuery, you probably know we have a subset of XQuery already supported inside SQL Server. We currently have no official plans that we can announce on a client side XQuery engine but we are definitely interested in expanding client side query processing. LINQ offers a path to this (for both relational as well as XML). ESQL provides another client-side investment. We are open to customer feedback on the relative importance of client side XQuery compared to all these other possibilities. Meanwhile we are doing some XQuery improvements in SQL Server 2008, adding LET, better datetime support, and lax validation.

As for XSLT 2.0 - we’ve heard from customers and understand the improvements in XSLT 2.0 over XSLT 1.0, but right now we’re in the middle of a big strategic investment in LINQ and EDM for the future of the data programming platform which we think will create major improvements in programming against all types of data. But we are always re-evaluating our technology investments so if your readers want to ramp up their volume on XSLT 2.0 please ask them to drop us a line with their comments.

Meanwhile I was rather surprised by the positive feedback to my little XML Notepad 2007 tool. It now has over 1 million downloads and is still going strong. Not bad for a few weeks work and no marketing. So something about this tool hit the sweet spot. The interesting thing there is it reaches out to the non-programmer community and I think that is the key and it has the right balance of simplicity and usability. A Swedish customer said it is “logam” – just enough. I think you should expect to see more from Microsoft in the future that helps to make XML something that everyone on the planet can deal with easily and in a way that integrates deeply with everything else Microsoft provides.

As the XML hype is wearing off, folks are realizing that not everything that made it through the standards process needs to be implemented.   So I think you will see Microsoft continue to innovate on new XML technologies and tools like LINQ to XML and VB 9.0 XML and you’ll see Microsoft taking a more pragmatic customer-demand-driven approach to standards.    Microsoft will probably never implement every standard that comes out but I’m confident you will see Microsoft continue to be committed to the really important XML standards, like XML 1.0 and XSD, and any other standard that is essential to achieving cross-platform interoperability, including Open XML. There is enormous power in the cross-platform reach of XML and the huge industry adoption it has and I’m happy to see that Microsoft is continuing to do some really innovative work with XML.

XML Tools Screencasts

XmlTeam — Mon, 04 Jun 2007 20:11:00 GMT

These videos provide some great information and a preview of the new XML Editor, XSLT Debugger, and EDM Wizard coming in Orcas, as well as a sneak preview of a new XSD Designer and EDM Designer that we can expect to see released after the upcoming Orcas release.

PDC WebData XML Wrapup - XML Everywhere

XmlTeam — Sun, 18 Sep 2005 06:46:00 GMT

XML was everywhere at PDC, even if you didn't notice it. Bill Gates said in his keynote something like “In .Net V1, we put in XML at the surface, with 2.0, Windows Vista and Office 12, we are putting XML at the core of all our products, not only for interop but also in our file formats”. By one count, he used the term "XML" 37 times in his 1 hour presentation.

The WebData XML team showed several different technologies at PDC:

Tools in Visual Studio 2005 and .NET 2.0

Denise Draper gave a talk, and Stan Kitsis had a lab, on the new tools for working with XML in .NET 2.0, especially the new XML editor. Like XML itself, the Visual Studio 2005 XML Editor was used in lots of demos, even if few people made a big deal out of that. It was shown as a configuration file editor, as a Sparkle and Quark designer, in the Web Services and LINQ demo, and played a small role in most of the keynote presentations. We've previously seen how the years of work that have gone into building rock solid platform components such as MSXML and System.Xml have paid off for Microsoft and the customers. Now we are leveraging our investments to build XML tools, and it's nice to see those investments pay off already.

Those producing and using XML directly face a number of challenges, since XML is basically simple but is quite unforgiving if its numerous details are not handled exactly right. Likewise, XSLT is an extremely powerful language but is difficult for most new users to work with. Visual Studio 2005 includes some great new tools for working with XML, especially the new XML editor and the XSLT debugger.

The new XML editor in Visual Studio 2005 validates XML documents with smart IntelliSense and error messaging in the source with tool tips, color coding syntax, and more. XML document validation is supported for any XML document in the XML editor based on the W3C rules of XML as well as any referenced XSD schema or DTD. The XML editor can be customized to create domain-specific editors for different XML vocabularies, and can be "hooked in" to provide advanced editing functionality in custom designers with the Visual Studio SDK.

The new XML tools in Visual Studio includes full debugging of XSLT documents with graphical preview and source output integrated within the Visual Studio 2005 IDE. The XSLT debugger gives developers a window into the dynamic behavior of their XSLT transformations, helping them both understand their stylesheet and learn to grok the XSLT approach.

Language Integrated Query and XML

Dave Remy presented, and Erik Saltwell wrote a lab, on XLinq, the XML support in the forthcoming Language Integrated Query feature of .NET. Dave, Soumitra, and Mike have already blogged about this in some detail.

One of the key challenges to working with XML data has been the impedance mismatch between XML and programming languages. A future release of Visual Studio will provide programming languages and frameworks to help integrate XML and database queries with C# and Visual Basic. XLinq offers a straightforward, easy to use programming model that is lighter from a memory and performance perspective. It is distinguished from APIs based on the W3C DOM in several ways:
o XLinq supports an approach to building XML trees in a top-down manner using functional construction. DOM-like APIs use a bottom-up approach that has been confusing to many developers.
o There is no need to create an XML Document object to hold the XML tree. The XLinq object model does provide an XML Document but you only need to use it when it is necessary, for example if you need to add a comment or processing instruction at the top of the document
o XLinq goes out of its way to make XML names as straightforward as possible to deal with, whereas most developers find the DOM-based APIs quite confusing in their treatment of namespaces and related concepts.
o XLinq has no text nodes, instead developers can simply cast elements as well as attributes to the type of data they represent.

XML support in Visual Basic 9

Erik Meijer and Peter Drayton presented, and Avner Aharoni had a lab, on the XML support in Visual Basic 9. VB9 will offer radical improvements in its ability to work with data in all its forms: as objects, as XML, as relational data. It has features such as query comprehensions, object initializers and anonymous types that enable querying data in a more flexible, natural way than ever before. Also, VB becomes a state-of-the-art dynamic programming environment, with coverage of new features intended to radically simplify working with dynamically typed data on the .NET platform.

Visual Basic will be extended to cover a wide variety of data programming features, taking full advantage of the entire LINQ project, including XLINQ. The two main Xml features of VB will be XML literals and late-binding over XML. XML literals allow programmers to construct XLinq objects such as XDocument and XElement directly using Xml syntax. Values within these objects can be created with expression evaluation and variable substitution. Late-binding over XML allows programmers to access XML nodes directly by name, rather than indirectly using method calls.

Unifying Theme

How do all these PDC activities relate to one another? The short answer is making XML more usable by mainstream developers. XML now shows up just about everywhere, and it is being used as a file format for everything from configuration files to Office documents to UI specifications. This means that XML is coming to developers who may not be ready or willing to deal with it using the low-level XML tools prevalent today, which force the user to understand the complexities and subtleties of XML in some detail. Furthermore XML builds on a different view of data than do the SQL database tools that most organizations use to manage their information, and it is also distinct from the object-oriented programming style used by most developers. The technologies we showed at PDC aim to improve XML’s usability for users as well as its programmability for programmers.

For More Information

The conference website has links to presentation slides, video of some of the keynotes, instructions for ordering the 30GB of information distributed to attendees, and much more. I trust that team members who participated will update the team blog and their own blogs (linked off the team blog) with more details, their thoughts on what they learned from the experience, and point to the material they put together that may not be easy to find on the conference site.

One more plug - several of us will be at XML 2005 and will dive deeper into some of this material for a more XML-specific audience. We hope to see you there, and hear your response in person or via the contact or comment links here.

Mike Champion (stealing liberally from information supplied by several other team members)

XML, Dynamic Languages, and VB

XmlTeam — Fri, 16 Sep 2005 17:28:00 GMT

Thursday at PDC saw lots of details being put out about another big project our team has been working on -- the deep support for XML in Visual Basic 9. This is actually part of an even larger effort that our team member Erik Meijer has been involved in, to support fully dynamic languages such as Python in the .NET Common Language Runtime.

On the VB9 front, the big news is that two major features beyond and on top of LINQ will be supported in VB9:

"XML Literals" is the ability to embed XML syntax directly into VB code. For example,

Dim ele as XElement = <Customer/>

Is translated by the compiler to

Dim ele as XElement = new XElement("Customer")

The syntax further allows "expression holes" much like those in ASP.NET where computed values can be inserted.

"Late Bound XML" is the ability to reference XML elements and attributes directly in VB syntax rather than having to call navigation functions. For example

Dim books as IEnumerable(Of XElement) = bib.book

Is translated by the compiler to

Dim books as IEnumerable(Of XElement) = bib.Elements("book")

We believe that these features will make XML even more accessible to Visual Basic's core audience. Erik Meijer, a hard core languages geek who helped devise the Haskell functional programming language and the experimental XML processing languages X#, Xen, and C-Omega, now touts VB9 as his favorite. He gave a talk yesterday outlining the spectrum from statically typed languages such as C# and Java at one end, and fully dynamic languages such as Python at the other, but with VB right in the middle. His mantra is "Static typing where possible, dynamic typing where necessary." This fits in with what I quoted Anders saying about C# 3.0 having the ease of use of script while fully supporting strong typing, but VB takes it even further.

The proof that .NET is a platform for all languages, not just static ones, might be seein in one bit of information that came out in a presentation on Iron Python (the Python implementation on top of .NET) yesterday. Iron Python is actually faster on the Python benchmarks than the core Python written in C.

I'll post links to the supporting material later in a PDC wrapup post, but for now the message I want to get across is that our group is involved in some fascinating innovations to make XML easier for mainstream developers to work with, and we embrace both the static and dynamic languages.

Mike Champion

Changing the Game

XmlTeam — Tue, 13 Sep 2005 23:26:00 GMT

by Soumitra Sengupta

I am writing from PDC 2005, at Los Angeles. Anders Hejlsberg just showed a demonstration of the Language Integrated Query Framework (codename Project LINQ) at Jim Allchin's keynote address. If you are not at PDC and want to see what this is all about, please visit http://msdn.microsoft.com/netframework/future/linq/. The site will be up and running in the next few days. If you program against SQL Server and XML it will be worth your while. It is going to change the way you will query, transform and manipulate data from relational databases, XML and in-memory objects. The result I believe is dramatic increase in productivity.

This is as much hyperbole as I am going to use here. So let us get to the facts. There are millions of developers who program with in-memory objects, query and fetch data from relational databases and now want to query and transform XML. Till now their option was to learn a general purpose programming language like C#, VB, C++, learn SQL to write queries against SQL Server and XSLT, XPath, SAX, DOM, XmlReader and possibly XQuery to program against XML. I do not know about you, but it is quite a chore for me as at least in the realm of XML, because all use different data models. So we started asking the question - how can we let developers do this work more efficiently by using C#/VB and Visual Studio. The answer we came up with is Project LINQ. The project is led by Anders Hejlsberg, Distinguished Engineer here at Microsoft and spans several product units and divisions at Microsoft, including the product unit I work for.

Project LINQ is the Language Integrated Query framework is a set of language extensions to C# and VB and a unified programming model that extends the .NET Framework to offer integrated querying for objects, databases and XML.

DLinq: for relational data access
XLinq: for XML data accessL

I will only talk about XLinq from a very high level. XLinq has two facets.

a. One facet is an in-memory cache that is faster, lighter and I believe easier and more intuitive to use than XmlDocument (W3C DOM). Let us face it, DOM has been around for a while now and does not take advantage of modern language features like generics. I think we can all agree that reading a document into XmlDocument, making changes to it and save it back is not for the faint of heart and the code written is not very readable. The in-memory model is simple and elegant and makes good use of the new features in the CLR. Dave Remy who is driving XLinq from our team will be writing about it in more detail so I will let him explain the technical details of XLinq in-memory model.

b. The second facet is language integrated query support. We add familiar XPath axes (like Descendants, Child, Parent) to the query operators in the LINQ framework, functional construction of XML to make it composable with the query operators and a host of other features to provide a rich programming, query and transformation experience. Whether you program in C# or VB, you get all the benefits of this model plus compile time checks, intellisense support and auto completion of statements in Visual Studio. On top of that we add support in VB for XML literals and take advantage of late binding to simplify the developer's job even further.

I really want to stress one thing - this does not mean Microsoft is walking away from supporting XPath, XSLT, DOM and SAX. On the contrary we will make improvements to our MSXML and System.Xml components and add tools support to improve performance and ease of use for our developers. If you have invested in these technologies, you can bet that you will be able to leverage your investments over multiple releases of Visual Studio and Windows. There are scenarios like Docbook, localization and personalization where it makes sense to learn a language like XSLT. Our customers who depend on XSLT for these scenarios will continue to get value from the Microsoft platform. But for millions of developers who live and breathe C# or VB and do not want to go through the learning curve of DOM, XSLT and XPath, XLinq will provide a very productive alternative. XLinq will sharply lower their learning curve and increase their productivity.

How many of you routinely create a connection to a database, query the database using SQL, get a DataReader back, construct XML over the reader and send it over the wire? Once you play with the bits you will realize how easy it will be to do tasks like this using the technologies in Project LINQ.

So you ask, when is this going to be available? The bits were distributed at PDC and is also available for download, including whitepapers and sample code from http://msdn.microsoft.com/netframework/future/linq/ . I strongly encourage you to play with the bits and give us feedback. Your feedback is a critical part of shipping this technology in future releases of Visual Studio. If you like it, tell us what you like about it. If you do not like it, again tell us what you do not like about There will be Project LINQ related forums setup at http://forums.microsoft.com/ for you to provide feedback. The Project LINQ team is eagerly awaiting your feedback.

Finally stay tuned to the XML Team Blog if you want to get more technical details on XLinq in both C# and VB in the coming days.

If you are at PDC, do look us up. I look forward to meeting with you and sharing our thoughts on this exciting project. If you are not, provide your feedback in the comments section.