Arpan Desai's WebLog

Performance of System.Xml in the .NET Framework V2

MSDNArchive — Tue, 07 Jun 2005 23:29:00 GMT

Vindication tastes so sweet: http://msdn.microsoft.com/vstudio/java/compare/xmlperf/default.aspx

XQuery in the .NET Framework - Revisited

MSDNArchive — Fri, 29 Apr 2005 09:50:00 GMT

As I was catching up on email this afternoon, I noticed that Stylus Studio had sent out an XQuery-related newsletter which contained an interesting campaign to “save XQuery on Microsoft .NET 2.0”. As I read through the email, a few issues and questions arose for me which I felt would be best addressed by one of my famously, non-frequent blog posts.

Issue: The basic premise of this petition is to convince Microsoft to ship XQuery as part of.NET Framework 2.0.
We have been in feature freeze for the .NET Framework quite some time now. Heck, we recently released Visual Studio 2005 Beta 2, which includes .NET Framework 2.0 Beta 2, and are well on our way to RTM. There’s no throwing features like XQuery in at this point in time, sorry. If you’d like to know why we made the decision to cut this functionality, we posted our reasoning online several months ago.

Issue (this one's a minor nitpick): The email states that the “.NET Frameworks ship only every 3 or so years”.
Hmm, let’s see:

Version 1.0 shipped as a part of Visual Studio .NET 2001
Version 1.1 shipped in Visual Studio .NET 2003 and Windows 2003
Version 2.0 will be shipping as a part of Visual Studio .NET 2005 and SQL Server 2005

The .NET Framework will also ship as a free, standalone SDK and redistributable, just as it has in the past.
<marketing_plug>In addition, we will be releasing free “Express” versions of Visual Studio 2005 and SQL Server 2005. See previous product links for details.</marketing_plug>

Question: What’s so important about having XQuery in the .NET Framework?
One of the fundamental design points of System.Xml was extensibility. We decided well before we wrote our first line of C# that we needed to lay out an infrastructure that was extensible by everybody. There’s no reason why somebody else can’t provide an implementation of XQuery which sits outside of the .NET Framework; and do it just as well as we can. (hint, some enterprising people are already working on this)

At the end of the day, we sat down and talked in depth to a lot of people about XQuery before we came to this decision. On the .NET Framework, developers felt that XQuery added an unnecessary complication to an area which we should be focusing on the basics such as simplification and performance. We took this feedback seriously and redoubled our efforts regarding general usability and performance (more on this in a future post or MSDN article). On SQL Server, on the other hand, users felt that XQuery was an excellent fit for performing operations against the new native XML datatype available in SQL Server 2005.

I believe our current positioning shows that Microsoft is still commited to XQuery as a technology overall, but we’re not going to push it in areas where it doesn't provide a significant value-add over what's already available. That said, we are obviously keeping an eye on people adopting XQuery both inside and outside SQL Server in order to help us plan for the future.

As always, feel free to post comments or email me directly: arpande at microsoft.com

Pobre Mark

MSDNArchive — Sun, 26 Dec 2004 06:01:00 GMT

The Grinch (the watch, not Mark)

Apparently Mark's ongoing saga with this "SmartWatch" is just that: ongoing. I remember when he walked into my office earlier this year and showed me his nice, new, shiny watch and the fact that with the press of a button (or two), he could figure out when/where his next meeting was. This single feature made me an instant fan. Forget the news, the weather, my horoscope?, and whatever useless pieces of information they think is interesting on a 1-inch display. All I wanted to know was the time and location of my next meeting.

Initally, I balked at the price of the watch in combination with subscription requirement. Then Mark's watch went out. After the first watch self-destructed, I was still torn whether to get it or not. After the second watch died, I became suspicious...very suspicious. In fact, I remember the moment the smartwatch fell out of favor:

While we were at TechEd 2003, a few of us decided to go out to dinner. After a long meal, somebody asked Mark what time it was and he replied, "five past twelve". Amazed at how much time we had spent just at dinner, we decided to walk back to the hotel. On the way back, a random person on the street asked what time it was. This time Mark replied, "quarter past twelve". As we walked away, the person started to shout that we must have had the wrong time. We laughed it off, after all, who was the one with the watch? Once we got back to the lobby, we noticed that there were still quite a few people milling around. Seemed a bit strange for midnight. We decided to have a drink before we went to sleep and on our way to the bar, somebody noticed that it was, in reality, just after ten o'clock. Mark looked at his watch again. The watch claimed it was 12:20...January 1st. Apparently it reset just as we were finishing dinner. Who had the watch, indeed.

That did it, no watch for me. Man began keeping track of time approximately 5,000 years ago and we continue to make progress today. I'm sure that over five millenia, there have been many badly designed timekeeping devices. This is pretty high on the list in my opinion.

Salvation, refound

A little known fact is I used to be a developer on the Smartphone team way back when (circa 2000). Once they were released state-side, I always planned on getting one. Initially, I had to wait for the contract on my phone ran out. Once that happened, the only model that seemed interesting was the Motorola MPx200 which was lacking two features which I wanted: bluetooth & camera. Acutally, I never thought I'd want the camera, but who gets a phone thesedays without it? So I figured I'd wait it out until the MPx220 was released. So I waited...and waited...and waited.

Then I went to XML 2004. On the flight to Washington, DC, Dare took out his new toy: A new, shiny...Audiovox 5600. After playing with it for five minutes, I was sold. So I took the plunge and when I got back to my house a week later, there was a package from FedEx waiting at my front door. Within ten minutes, no joke, it ported my old number and I was synching with the Exchange server. I was especially proud of that second part since I had worked on the synchronization framework and written the first version of ActiveSync on the smartphone OS. (Although Andy likes to think that they threw all my code away once I joined the Dark Side aka became a Program Manager)

One month later, I can say that I'm thrilled with the Smartphone..and I'm not the only one. The smartphone, and the Audiovox in particular, is spreading like wildfire inside the company. The viral effect is beyond anything else I've seen, even puts the iPod effect to shame. As for the smartwatch, I can only hope that Mark has better luck in the new year. BTW, I'm sure that he's an EXTREMELY exceptional case. I know other people with smartwatches and they've never had a problem with it.

Timeshifting not always a good thing?

MSDNArchive — Wed, 15 Dec 2004 23:47:00 GMT

I love PVRs. They let me timeshift television shows so that I can watch whenever I want. One good thing about PVRs is that I KNOW what I'm watching isn't stale. I know that the ten episodes of "The Daily Show" on my Media Center aren't all current, but heck, they're still hilarious.

Timeshifting on MSDN TV is a different story. As Jeff noticed, MSDN decided to post an episode that mfussell taped earlier this year. As with just about everything that Microsoft develops, there has been a significant amount of thinking and churn that's happened since then....making portions of the episode completely incorrect. Lesson learned: Timeshifting on MSDN TV is not a good thing.

Oleg picked up on Jeff's post and expanded on some of the disappointments he sees with what we're shipping in Whidbey now compared to what the episode said. Although I agree with some of his gripes (some of this stuff should have made it into V1.1), I think the main problem is we haven't done a good job of explaining why we made some of these changes. I'm partially to blame for that, so let me try and rectify.

XmlDocument is (once again) the preferred store for XML
Today, V1.0/1/1, there's the XmlDocument which virtually everybody uses as their XML store. There's also the XPathDocument which has better performance in some query scenarios, but it's not editable.

At the beginning of Whidbey, we took a look at the XmlDocument and "decided" that the performance can't be fixed. So what's the answer? Bolt all sorts of interesting features onto XPathDocument! Editable XPathNavigator? Add it to XPathDocument! Type support? Add it to XPathDocument! Later in the cycle, we decided to reinvestigate the performance issues of the XmlDocument. Thanks to some work done by the CLR team in Whidbey and some fresh eyes, we realized that maybe the performance issues weren't insurmountable. So what's our new story for XML stores? Keep using the XmlDocument. You now have typing/validation support and editable XPathNavigator support (Jeff should be happy). We're also doing a lot of performance work so that we can effectively obsolete the XPathDocument in the long run. BTW, addressing performance and usability of the XmlDocument is going to be one of my main objectives going into the future.

XQuery/XSLT
This should be an entire post, possibly even a book, in itself. As the Program Manager for XSLT and XQuery in Whidbey, this is a subject near and dear to my heart. I'll try and cut to a few fundamental facts on this one...of course I'm sure there will be some lingering questions/disagreements from many of you:

- Neither XQuery 1.0, nor XSLT 2.0, will be finalized until probably the end of next year (2005). <opinion type="personal">Based off how many times both of these specs have already slipped, literally, years; I'm not confident even on that timeline.</opinion>

- Whidbey is scheduled to release before this time. Regardless of whether Whidbey slips beyond this, we can't plan our release on this fact. <opinion type="personal">I don't think Whidbey will slip. Recently, the organization has done an excellent job of locking down this release.</opinion>

- When we say that we're not going to support something, this goes for pretty much EVERY feature, it simply means we're not shipping in our stack at that point in time. We may implement it down the road. It does NOT mean we've written it off completely and forever.

- When we do decide support something, it should have a pretty long lifetime. XQuery 1.0 and XSLT 2.0 are huge undertakings. Writing an implementation is, surprisingly enough, is one of the simpler factors. Making sure it's scalable makes things a little harder. Getting the quality high enough so it's enterprise-ready (btw, this is much, much more than passing XYZ test suite) is pretty difficult. Getting tooling support like query generators, debuggers, and profilers is a substantial amount of work on top of everything else. Investing this much into a couple of unfinished standards? Maybe not the best idea.

So was XQuery a black hole? Maybe, but some good came out of it. For example, we came up with a new query architecture to support both XSLT and XQuery. Our Whidbey XSLT implementation is built on this architecture and the performance of this implementation, well, blows the socks off XslTransform. Another major benefit was our ability to provide feedback to the XQuery/XPath working groups in finding issues in the standard. This aspect of early implementation is often trivialized, but it's an important step in validating a standard.

The remaining 95%
There are literally hundreds of enhancements that were made to the rest of System.Xml in Whidbey from simple things like implementing typed accessors on XmlReader/XmlWriter/XPathNavigator for usability to rewriting the XmlReader to double its performance...Not to mention the new XML tools support in Visual Studio (should be another book in itself). Sometimes our team doesn't do the best job of conveying this information because we become too enamored with some specific problem-set we're trying to solve.

As you can tell from my blogging history, I don't have the best record in terms of regular posts; maybe that will be my New Year's resolution. However, I'm always accessible via email. My alias is arpande. If you have any questions or want to start a friendly debate, feel free :)

-Arpan

XPathNavigator and the DataSet

MSDNArchive — Wed, 25 Aug 2004 06:21:00 GMT

An article I recently wrote about an XPathNavigator over DataSets is currently running on MSDN (http://msdn.microsoft.com/xml/default.aspx?pull=/library/en-us/dnxmlnet/html/datasetnav.asp). Might be an interesting read if you ever perform XPath queries or XSLT transformations on top of DataSets.

Obligatory cheesy conference post

MSDNArchive — Wed, 26 May 2004 22:01:00 GMT

It seems that most members of the the blogging community succumb to posting a “Yesterday at the -Insert Name- Conference...”, so who am I to rock the boat.

Yesterday at TechEd 2004, Mark gave his talk on “XML Today and Tommorrow”...basically a rundown on how people use our bits in the current version of the .NET Framework and some of the improvements we're making in Whidbey. Unfortunately I wasn't able to make it because I was meeting with some customers, but from feedback I got from talking with several customers afterwards, it went quite well. Rumor is there was applause when he demoed some of the XQuery functionality we're introducing. That's good for my XQuery talk on Friday.

In the afternoon, I went an saw Dave Campbell's talk on SQL Server 2005 “Yukon”. He went through all the new features of Yukon, several of which I had no absolutely no clue we were doing. Sometimes we suffer from a sort of product myopia because we're too busy to understand what everybody else is working on. Dave did a great job of showcasing everything and tying it together. One of the interesting features I actually have been keeping an eye on is data partitioning. I think our story there is going to be killer.

Later, I met up with Andy and Luca, of ObjectSpaces fame, and we attended a Birds of a Feather talk on Object/Relational Persistence Frameworks (and really anything else O/R related). The delay of ObjectSpaces seemed to be a disappointment to several of the attendees. We discussed several other available frameworks and the design/coding pains that everybody has experienced with them and ObjectSpaces. An interesting point that one of the attendees brought up was that he'd like to see a general (pluggable) persistence framework which would enable him to persist to relational sources, XML, etc. This is one of those ideas we've been throwing around internally for several years.

After the BOF, we met up with Mark and the four of us attempted to go a really good Szechuan restaurant called Spicy City. Unfortunately, it was closed when we showed up at 9pm, so we decided to go back to the Gaslamp District and find something else. I suggested Trattoria La Strada, an Italian favorite from my college days. We had a nice, relaxing dinner which included a few courses of amazing food, a nice bottle of chianti, and some grappa to finish. After dinner, we attempted to find some sort of a dive bar to hang out in, but Gaslamp thwarted us by only providing fine upstanding establishments. We decided beggers can't be choosers :)

That concludes my cheesy post.

Promises, promises

MSDNArchive — Thu, 13 May 2004 20:27:00 GMT

So much for for blogging more regularly. But if there was ever a reason to start up again, Mark's recent post on XSLT 2.0 and Dare's response are as good a reason as you can get. As the XML Query (XPath, XSLT, XQuery, anything else) Program Manager for the .NET Framework, it's time for me to chime in.

First, we're not abandoning XSLT 1.0. Far from it, we've completely rewritten our XSLT implementation from scratch in Whidbey (XsltCommand) in order to solve the major performance issues we had with the current implementation. Our original intention was to meet MSXML 4.0's incredible performance. I believe we'll surpass this goal by the time we release.

XSLT 1.0 will still be a (the?) core scenario for the query team, even in the Whidbey timerame. Transformation is a vital piece of functionality in most XML processing pipelines, this will not change in the forseeable future. The question then becomes, what is XSLT 2.0 bringing to the table that makes it compelling. We're not interested in implementing a standard just so we can tick off a checkbox on some feature list. We're here to solve customer problems.

Several years ago we sat down and evaluated the intentions of both XQuery 1.0 and XSLT 2.0 while they were both in their infancy. We looked at the pains our customers were telling us they were having. We looked at where we anticipated their pains would be in 5 or 10 years. This yielded some interested conclusions:

People wanted a more powerful XPath...and all they had was XSLT. We saw (and still see) case after case of people who need some advanced querying functionality and shoehorning XSLT to solve the problem. XPath 2.0 doesn't solve the problem either because the scenarios required some XML construction in the results.
XSLT syntax and programming paradigm is not user friendly. This was Mark's main point and is something we've heard from developers over and over again. Syntax is an easy problem to solve, just make a human-friendly alternative. However, the programming paradigm is quite hostile to anybody who isn't a fan of LISP. This is a fundamental issue many people have with the language.
ASP.NET 2.0 is a phenomenal product. Many of us believe that one of our key XSLT scenarios, HTML generation, will be greatly diminished with the ease in which an ASP.NET solution can be developed. There will still be cases for XSLT on a web server, however this will be reduced in time.

These data points left us two real conclusions:

Implement both XSLT 2.0 and XQuery in the .NET Framework
Implement XQuery in the .NET Framework

There was much deliberation on this topic and we concluded that we were satisfying the transformation scenarios our customers had, but were failing on the core query scenarios. Given that we don't have infinite resources, we felt that having a strong XPath 1.0, XSLT 1.0, and XQuery 1.0 implementation was more important than having a mediocre XPath 1.0/2.0, XSLT 1.0/2.0, XQuery 1.0 story.

This is a bet that Microsoft is making. We're putting a stake in the ground and saying we want XQuery to be defacto XML query language. SQL Server 2005 will have a native XML datatype which will utilize XQuery. We're also talking with other product teams which use XML extensively to get them on the wagon. Again, this is to ensure that long term we have a strong, consistent message rather than a weak, fragmented one.

For those of you attending TechEd 2004 in San Diego this year, feel free to check out DAT327. Michael Rys and myself will be presenting a session on XQuery in SQL Server and the .NET Framework. You'll be able to grill us in person :)

Tim Bray / RSS Bandit strikes again

MSDNArchive — Wed, 24 Sep 2003 03:29:00 GMT

Tim Bray, of XML co-creator fame, was just interviewed by News.Com. In the article, he gives a good overview of why XML was created and the possible solutions that were available at the time. Few other things of note:

Positive things to say about Microsoft and XML. He likes Visual Studio .NET's easy integration with web services. I appreciate the fact that he mentions he's seen the Office 2003 XML schemas. There's a lot of FUD out there regarding the openness of the XML with regard to Office 2003. For those of you non-believers, start believing.
Apparently feels the XmlReader API in System.Xml was a good step forward. Wait till he sees what we're doing in Whidbey. :)
He mentions he doesn't believe that XML could be done at the W3C today. I completely agree. The various W3C WGs are bloated, monolithic beasts that can rarely agree that true == true . If you don't understand how more representation can be a bad thing, go watch Star Wars - Episodes 1 and 2. The Republic Senate == W3C.

Our very own dareo has written an article on revamping RSS Bandit. As always, a good read. I really like the ability to apply stylesheets to the feeds for a custom look and feel.

PDC, XQuery, and mfussell

MSDNArchive — Tue, 23 Sep 2003 09:30:00 GMT

That special time of year has finally come: Microsoft is hosting PDC (Professional Developers Conference) at the end of October. For those of you not completely in the know, PDC is where we get to reveal all the cool technologies we've been working on. There are several sessions of particular interst this year this year:

.NET Framework: Exploring What's New in the Base Class Library for "Whidbey"
The BCL guys have done a great job of adding a lot of needing functionality to the framework.
.NET Framework: What's New in System.Xml for "Whidbey" - mfussell
My boss is giving a talk on what our team has been working on for the past few years.
Programming "Indigo": Building Services and Connected Applications
Should be a good overview of what to expect in the web services world
Introducing "Avalon," the Next Generation of Windows Presentation Services: A Technology Overview
These guys are doing some cool stuff. 'nuff said
Visual C# "Whidbey": Language Enhancements
Some new cool features being added to C#, won't spoil the fun here
Developing Data Applications for Devices with Microsoft SQL Server"! CE 3.0
SQL Server running on a smartphone. How much techier can you get :)
Building Database Applications with SQL Server "Yukon": XQuery, XML Datatype - arpande
What can I say, shameless plug for a talk I'm giving. The current plan is to go over the XML support in Yukon and how it integrates with client technologies.
Programming SQL Server "Yukon" Using Managed Code: Building Store Procedures, Functions, and User-Defined Types
More Yukon goodness/essentials.
ASP.NET: Overview of ASP.NET "Whidbey"
Another bread and butter talk. Can't miss this.

Apart from the talk mentioned above, the INETA has agreed to have an XQuery BOF (Birds of a Feather) talk. This will be a great opportunity to have a round table discussion with members of the product team at Microsoft implementing XQuery and with other conference attendees who have an interest in XQuery.

And last, but not least, mfussell (Mark Fussell) has joined the fray of Microsoft bloggers. Mark is responsible for keeping Dare, Joshua, and myself in line, especially when we start coming up with crazy features which would take years to implement. Although wrangling us is already a full time job, he still managed to find time to write a book on the new features of System.Xml. Check it out.

Lazing Around

MSDNArchive — Tue, 16 Sep 2003 00:16:00 GMT

This past weekend I had two tasks I wanted to accomplish:

Buy a new watch
See michaelb's baby

What I actually did:

Saturday
Went to Fry's, bought some component cable
Went to see Once Upon a Time in Mexico. It was interesting to note that the teenagers didn't seem to enjoy the movie very much, probably because they've never seen El Mariachi or Desperado.
Sunday
Went to Home Depot & Staples to pick up random tools and things
Helped my friend, Jason, (bi)wire up his new speakers

Which brings me to the real reason for this post. Jason's new speakers are quite impressive, to say the least. He had broken them in over the past month with some generic Radio Shack cable. Yesterday we broke out the real cable and we had enough left over to bi-wire the front left & rights (note: they actually support tri-wiring!). Anyway, after a bit of fine tuning, we started throwing every acoustically impressive CD and DVD at it and it sounded quite good.

I think an SACD player may be next in line.