Microsoft XML Team's WebLog : xml

XML and Languages

XmlTeam — Fri, 15 May 2009 05:16:00 GMT

I've written in the past about XML and languages, and why you might be interested in being aware of the language associated with text.

Text with no language is just not quite there
Impact of text language on WPF
Text, language and sorting

For dealing with languages, xml:lang is your friend, as you can tell from these older posts.

Something that is a bit special about xml:lang is that xml is a reserved namespace. From http://www.w3.org/TR/REC-xml-names/#xmlReserved

The prefix xml is by definition bound to the namespace name http://www.w3.org/XML/1998/namespace. It MAY, but need not, be declared, and MUST NOT be bound to any other namespace name. Other prefixes MUST NOT be bound to this namespace name, and it MUST NOT be declared as the default namespace.

Here is the code you can use to write an xml:lang attribute using an XmlWriter.

XmlWriterSettings settings = new XmlWriterSettings();

settings.Indent = true;

using (StringWriter textWriter = new StringWriter())

using (XmlWriter writer = XmlWriter.Create(textWriter, settings))

{

writer.WriteStartElement("e");

writer.WriteStartElement("t1");

writer.WriteAttributeString("xml", "lang", null, "en-US");

writer.WriteString("Hello, world!");

writer.WriteEndElement();

writer.WriteStartElement("t2");

writer.WriteAttributeString("xml", "lang", null, "es-AR");

writer.WriteString("¡Hola, mundo!");

writer.WriteEndElement();

writer.Flush();

Trace.WriteLine(textWriter.ToString());

}

Here is the traced output.

<?xml version="1.0" encoding="utf-16"?>
<e>
<t1 xml:lang="en-US">Hello, world!</t1>
<t2 xml:lang="es-AR">¡Hola, mundo!</t2>
</e>

If you are not using XmlWriter, but instead prefer to use LINQ to XML, it is even easier. This is because LINQ to XML has support for the xml namespace built in. Here is the code you could use to set the language on elements by adding the xml:lang attribute after creating the XDocument. Notice the lack of the xml:lang attribute in the source.

XDocument doc = XDocument.Parse(@"

<e>

<t1>Hello, world!</t1>

<t2>¡Hola, mundo!</t2>

</e>");

XNamespace xmlNs = XNamespace.Xml;

foreach (var element in doc.Root.Descendants())

{

if (element.Name.LocalName == "t1")

element.SetAttributeValue(xmlNs + "lang", "en-US");

if (element.Name.LocalName == "t2")

element.SetAttributeValue(xmlNs + "lang", "es-AR");

}

Trace.WriteLine(doc.ToString());

Here is the traced output.

<e>
<t1 xml:lang="en-US">Hello, world!</t1>
<t2 xml:lang="es-AR">¡Hola, mundo!</t2>
</e>

Enjoy!

Marcelo Lopez Ruiz

http://blogs.msdn.com/marcelolr/

XmlSchemaSet Thread Safety

XmlTeam — Tue, 28 Apr 2009 00:59:00 GMT

Here's a good word of warning: even if an object "feels" read-only because you're not calling code to modify it, if it's not documented as safe for use from multiple threads, then you shouldn't risk it.

As an example, let’s look at XmlSchema and XmlSchemaSet. Initializing these has a cost associated with it, and so it's nice to be able to build them once and then reuse them. But you have to be very careful in doing this. The docs say that all instance methods are not safe for multiple thread usage, but you don't really use them directly during validation, so it's hard to tell from the outside what's safe and what's not.

In a nutshell, the only thing you can do that is safe for concurrent usage is to use a validating reader. Here's the sample code to try this out (for some reason, this "breaks" more on 64-bit machines, but it's unsafe on all architectures).

First, a little helper to create an XmlSchema.

private XmlSchema CreateSchema()
{
string schemaText = @"<?xml version='1.0'?>
<xs:schema id='play' targetNamespace='http://tempuri.org/play.xsd'
elementFormDefault='qualified' xmlns='http://tempuri.org/play.xsd'
xmlns:xs='http://www.w3.org/2001/XMLSchema'>
<xs:element name='myShoeSize'>
<xs:complexType>
   <xs:simpleContent>
    <xs:extension base='xs:decimal'>
     <xs:attribute name='sizing' type='xs:string' />
    </xs:extension>
   </xs:simpleContent>
</xs:complexType>
</xs:element>
</xs:schema>";

using (StringReader reader = new StringReader(schemaText))
{
    return XmlSchema.Read(reader, null);
}
}

Next, a simple XmlSchemaSet.

private XmlSchemaSet CreateSchemaSet(XmlSchema schema)
{
XmlSchemaSet set = new XmlSchemaSet();
set.Add(schema);
set.Compile();
return set;
}

Finally, some validation:

private void ValidateDocument(XmlSchemaSet set)
{
string doc = @"<myShoeSize xmlns='http://tempuri.org/play.xsd' sizing='123' />";
XmlReaderSettings settings = new XmlReaderSettings();
settings.Schemas = set;
settings.ValidationEventHandler += new ValidationEventHandler(settings_ValidationEventHandler);
using (StringReader reader = new StringReader(doc))
using (XmlReader x = XmlReader.Create(reader, settings))
{
while (x.Read()) { }
}
}

private int failCount;
void settings_ValidationEventHandler(object sender, ValidationEventArgs e)
{
System.Threading.Interlocked.Increment(ref failCount);
}

Now, armed with these, I will show you some code that is thread-safe, but that a single line reorder would cause to break.

XmlSchema schema = CreateSchema();
Thread[] threads = new Thread[10];
XmlSchemaSet set = CreateSchemaSet(schema);
for (int i = 0; i < threads.Length; i++)
{
threads[i] = new Thread((x) =>
    {
      for (int j = 0; j < 1000; j++)
      {
        // If the CreateSchemaSet were here
        // instead of outside this would break!
        //
        // Don't add the schema to the
        // XmlSchemaSet from multiple threads!
        //
        // XmlSchemaSet set = CreateSchemaSet(schema);
        //
        ValidateDocument(set);
      }
    });
}

Array.ForEach(threads, (t) => t.Start());
Array.ForEach(threads, (t) => t.Join());
this.Text = "Failure count: " + failCount;

The part before the thread creation runs on a single thread, and so there are no multi-thread concerns; the stuff inside the callback is happening on multiple threads at the same time. You can only use the set for validation here!

Enjoy!

Marcelo Lopez Ruiz

http://blogs.msdn.com/marcelolr/

Frequently Asked Questions on XML in .NET - Part 1

XmlTeam — Thu, 11 Sep 2008 00:51:00 GMT

As part of a separate task, the XML team came up with a list of frequently encountered issues in System.XML; mainly points that we felt were interesting because they were the source of a lot of difficulty for our users. These questions ranged from rarely used (or misused) methods to difficult XML constructs. We focused specifically on scenarios that were particularly difficult to debug. When we had completed the exercise, it occurred to the team that we should publish the list.

So what follows is the first in a multi-part series in which we will outline each of the questions we defined and provide an explanation of the correct usage, along with some sample code. However, our motives are not all selfless in this exercise; we are hoping to hear back from you in the comments section if there are any good questions we have missed. Let us know!

Q1: Invalid Literals

There are a number of reserved characters in XML that cannot be included as literals in an XML string – characters such as “&” and “<”. These characters must be escaped and the XML standard provides three methods for doing this: characters references (&#38), entity references (&) and CDATA sections.

This idea will seem like basic XML101 information for experienced XML users but it can be a difficult pain point for those new to XML. The problem is compounded because the error messages returned by the XML processor can be confusing. The error is also a simple one for a new user to stumble upon because the characters are often contained in regular text that may be used as the source for the XML data. The invalid literals are:

· & (&)

· < (<)

· > (>)

· ‘ (')

· “ (")

Here is a sample list of incorrect literal strings, along with the correct usage and the exception message that is given for the incorrect usage. The messages all produce a line and position number which greatly assist the debugging process.

Character String	Correct Usage	Exception Message
A & B	A & B	An error occurred while parsing EntityName. Line X, position Y.
A &c B	A &c B	' ' is an unexpected token. The expected token is ';'. Line X, position Y.
A &# B	A &# B	Invalid syntax for a decimal numeric entity reference. Line X, position Y.
A < B	A < B	Name cannot begin with the ' ' character, hexadecimal value 0x20. Line X, position Y.

Refer to section 2.4 of the XML Standard for more information on invalid literals.

Note: There will be a later post that deals with invalid XML characters in general and the correct way to deal with them.

Some future FAQ post topics include but are not limited to:

- Conformance Levels

- Reporting Validation Warnings

- Correct Encodings

- ProhibitDTD setting

- XLinq Bridge Classes

Shayne Burgess

Program Manager | Data Programmability

Introducing the XmlPreloadedResolver

XmlTeam — Fri, 15 Aug 2008 02:05:00 GMT

The XmlPreloadedResolver is a new type that we’ve been working on in SilverLight that provides the ability to load a DTD without a call to the network. The new type can act as a manually maintainable cache of DTD’s and XML Streams.

Let’s look at two examples of how the XmlPreloadedResolver can be used.

Example 1:

The XmlPreloadedResolver can be used to pre-load an external DTD. In the following example, a user-defined DTD is pre-loaded into the resolver. The resolver is then used when loading the XML file.

String strDtd = "<!ENTITY entity 'Replacement text'>";

String strXml = @"<!DOCTYPE doc SYSTEM 'myDtd.dtd'>

<doc>&entity;</doc>";

System.Xml.Resolvers.XmlPreloadedResolver resolver = new XmlPreloadedResolver();

resolver.Add(resolver.ResolveUri(null, "myDtd.dtd"), strDtd);

XmlReaderSettings rs = new XmlReaderSettings();

rs.XmlResolver = resolver;

rs.DtdProcessing = DtdProcessing.Parse;

XmlReader reader = XmlReader.Create(new StringReader(strXml), rs);

For the complete example, refer to: http://msdn.microsoft.com/en-us/library/cc189063(VS.95).aspx

Example 2:

The XmlPreloadedResolver also contains two well-known DTD sets: XHTML 1.0 and RSS 0.91. These two well known DTD sets can be enabled via the XmlKnownDtd enumeration. The following is an example of using the XmlPreloadedResolver to preload the XHTML 1.0 DTD into an XmlReaderSettings object. In this example, no network calls are needed to parse an XHTML file.

XmlReaderSettings settings = new XmlReaderSettings();

settings.DtdProcessing = DtdProcessing.Parse;

settings.XmlResolver =

new XmlPreloadedResolver(XmlKnownDtds.Xhtml10);

XmlReader reader = XmlReader.Create("HTMLPage.html", settings);

XDocument document = XDocument.Load(reader);

For the complete example, refer to: http://msdn.microsoft.com/en-us/library/cc189059(VS.95).aspx

Look for this new class in System.Xml.Resolvers namespace in a new DLL: System.Xml.Utils.dll that will be included as part of the upcoming SilverLight 2.0 SDK.

Shayne Burgess
Program Manager | Data Programmability

- and -

Helena Kotas
Senior Software Development Engineer | Data Programmability

Happy 10th anniversary, XML!

XmlTeam — Wed, 13 Feb 2008 10:26:00 GMT

The W3C Recommendation Extensible Markup Language (XML) 1.0 was initially published on the 10th of February 1998. Since then, XML has proliferated at a rapid pace, becoming the predominant method for electronic data interchange between information systems, and we can safely describe it as universally applicable, fast, scalable and durable integration code. XML and its related standards and tools are continuing to enable data interoperability, content manipulation, content sharing and reuse, document assembly, document security and access control, document filtering, and document formatting for all types of devices and applications. As we celebrate its tenth anniversary, we have a great opportunity to look back and reflect on the role XML has played in the technical ecosystem in which it functions, and to acknowledge the fact that it will take its place among the technical standards having the greatest impact on the world in which we live.

The Microsoft’s XML Team believes that XML's potential has only begun to be realized and that there are still new XML tools to be created, steps to be made, issues to be resolved and risks to be avoided, in order for XML to fulfill its promise. The XML team is committed to continue to assist developers everywhere to quickly build implementations that are scalable, robust, and flexible enough to work across all types of applications. Systems that implement the tooling required to process and query XML must address performance problems without destroying the requirement for ubiquity and the Microsoft’s XML Team is here to help enable implementers to rapidly prototype ideas.

Congratulations to all who made XML such a milestone 10 years ago and congratulations to all that today are releasing and innovating products and technologies that will direct the future of XML, helping it realize its full potential for increasing the value of information technology!

Irinel Crivat
Program Manager | Data Programmability

Chris Lovett Interview

XmlTeam — Fri, 16 Nov 2007 11:34:00 GMT

Chris Lovett was interviewed by book author Michael van Otegem recently and he asked some very interesting questions:

Please tell us who you are and what you do.
I’m an architect on the Data Programmability Tools team in SQL Server, and I work on XML tools that ship in Visual Studio. As an architect I do lots of different things including: product development on the XML editor in Visual Studio 2008; cross-group collaboration to make sure all our tools work together; playing with other concepts like XML Notepad; and thinking about and communicating our strategy and future directions for our technologies.

How long have you worked for Microsoft and what did you do before your current position?
I have worked on pretty much every XML core technology from Microsoft starting with MSXML in 1997, then to System.Xml in the .NET frameworks v1, and v2. and for the last few years I’ve been focusing on building XML tools in Visual Studio – for example, I was the primary developer on the XML editor in Visual Studio 2005. Before Microsoft I worked at IBM on OS/2 applications, then I joined the IBM Apple/IBM joint venture called Taligent, then I started my own company in Sillicon Valley with friends from Taligent during the height of the .com boom and that’s what led me to Microsoft.

A lot of the .NET Framework works with XML in some form or another. Why is XML such a key component and what were the challenges you faced because of that?
For one thing the .NET frameworks were designed during the peak of the XML hype curve, but more seriously, many folks at Microsoft were waking up to the fact that not everything had to be buried in code. Some things could be very neatly described at a higher “declarative” level and Tim Berners-Lee showed the world that “markup” is a great way to do that. So XML became the way to specify configuration information (.config files), and build information (MSBuild files), and setup information (WIX), object remoting with SOAP, and security permissions and so on. All of these domain specific uses of XML were then supported by our core System.Xml classes. It’s interesting to note that even those teams that didn’t adopt XML back in .NET v1.0 are fixing that. For example, we have the new AJAX work from the ASP.NET team and we have XAML in the Windows Presentation Framework (I’m a huge fan of WPF by the way. I’ve done a lot of UI development in the last 20 years and I have to say WPF totally rocks).   So XML is touching everything from database, management, communications, content publishing and now even into the user interface layer. I was amazed at the last PDC where just about every talk showed some snippet of XML somewhere during their talk.

I remember when we started XML at Microsoft most people thought we were crazy. The biggest challenge was convincing people the cost of parsing and storing verbose XML tags was worth it and our team has been working on performance, and scalability ever since. But the real reason XML become a key component of .NET, (and Windows, and SQL Server and Office) is because it achieved true cross-platform interoperability and because it was good enough for that job.

What do you (personally) like the most about XML and its associated standards?
Simplicity, cross-platform interoperability and huge adoption. The great thing about XML is that it is humble. It’s not trying to solve world hunger. Just invent your own tags, group them into structures that make sense for your domain and voilà. It is very simple and it is this same simplicity that helped HTML take the world by storm.   XML then improved on HTML by providing a clean separation between data and UI. It brought MVC to the masses so to speak.

The other day my 12 year old son was all excited and just had to show me what he discovered. He was editing Age of Empires XML files to tweak the behavior of the game using Notepad.    I asked him how he knew that he could do that and how he knew the XML syntax. He didn’t know what “syntax” meant, but he knew how to edit XML! Then the same week my doctor was all excited when he heard I worked on XML because he was involved in a software purchasing decision at our local hospital and it came down to their level of XML support. I couldn’t believe my ears.

The funny thing is that most programmers don’t really like XML. Probably because it doesn’t use curly brackets, so most programmers treat XML a bit like the ugly duckling. But the reality is that the whole world gets markup – to them markup makes our programming world more approachable. There are still way too many programmers that don’t get this.
What would be your #1 tip to people learning XML, XML Schema, XSLT and XQuery?
First of all I would say that XML 1.0 is the foundation. A must have. Can’t go wrong there, learn it, and learn how XML encoding works – that’s the number one issue people have with XML 1.0 – people don’t take the time to understand how UTF-8 encoding works which is a pretty important foundation to XML. Don’t worry too much about DTD, because we now have XML schema.

XML Schema (XSD) is a layer on top of XML 1.0. When you need a way to describe your XML structure in “standard contract” XSD is handy and most importantly – it’s there and it’s a standard. So don’t re-invent the wheel, but I’m not going to say that XSD is the be all and end all of data modeling, because it isn’t. A lot of things are missing, which is why people had to invent things like Schematron, and why Microsoft is working on EDM and SML and so on. Model driven development is now on the peak of the hype curve so I expect that modeling will be a battle ground for a long time to come. So take a pragmatic approach to XSD - use it if it fits your purpose. Some folks use other modeling approaches then have a tool that spits out the XSD – and that’s fine too.

Same goes for XQuery and XSLT – I think of these as being yet another layer above XSD. We did XSLT and XQuery because we figured that XML is data therefore people will want ways to query and transform that data. Makes sense, and I’ve done a lot of XSLT development, I still use it for specific tasks, but some things are a bit tedious. I find myself escaping to script a lot. XSLT 2.0 is a good improvement, but again, these things are not going to be the be all and end all of query and transformation languages.

For example, I’m a huge fan of the work they are doing in Visual Basic 9 with XML literals connected to our new Language Integrated Query (LINQ). It makes a lot of sense, because instead of having to “escape” to script, you just write the code you need right there in place – you have a complete general purpose programming language at your finger tips.   VB-XML integration is big leap forward for VB programmers and allows those developers, who may not be as familiar with the standard XML technologies like XSLT, to easily process XML data inside their programs. It’s a huge advantage to the VB programmer and we think it helps make VB an extremely compelling language for XML programming – it makes me want to write VB again, and I’ve heard many others say the same thing. However, it is VB-specific so development teams that need cross-platform interoperability at the query/transform layer are likely to stick to the standard technologies like XSLT and XQuery.

There’s a very interesting tension going on here where general purpose languages like VB and C# are moving up into declarative space with LINQ but not going all the way into declarative, versus SQL, XQuery and XSLT which are fully declarative with no side effects, which are therefore more optimizable, but sometimes rather incomplete as programming languages go and rather hard to author in some cases. I really don’t know how it’s going to end up. I think we should continue innovating on both approaches and see what happens. It should be very interesting.

As for all the myriad other XML standards out there, there’s a lot of hype that you have to sort though. To me it’s a funny thing to see programmers going to town making XML more complicated with layer upon layer of new concepts. I remember going to a conference and people were telling me “stop! – we can’t take any more”. There is genius in simplicity. I’m glad to see the renewed focus on simple REST-ful XML based services for this reason. If simple works, why complicate it. Conversely, if it isn’t simple, chances are people just won’t use it.

What can we expect from Microsoft in the future in the XML arena? Will support for XQuery 1.0 and XSLT 2.0 become part of Microsoft’s offering?
Microsoft is a pretty big place, so it’s hard for me to know all that is going on with XML across the company. But I do know about .NET, Visual Studio and SQL server. As I’ve mentioned before we are shipping the XML support in Visual Basic 9.0 with XML literals, XML axis properties and integration with LINQ to XML. LINQ to XML is our API which we are adding to the XML runtime in .NET 3.5, it is a new XML object model that is designed to work well with the Language Integrate Query capabilities of C# and VB.    We are also shipping some cool new features in the XML tools in Visual Studio 2008, including an incremental parser with extensibility API based on LINQ to XML that 3rd party XML designers can build on. We are also adding data breakpoints in the XSLT debugger and we have a new command line tool named “xsltc.exe” which takes XSLT and generates a .NET assembly which you can then deploy with your app instead of the XSLT source so you don’t have to compile XSLT on the server. Anton Lapounov has a great blog that talks about that.   There is not much else new in the System.Xml runtime because Visual Studio 2008 is essentially a service pack release of the .NET 2.0 runtime, so we’ve fixed some bugs there. We are also working on some XML features in Silverlight and we put up a preview of our LINQ to XSD work on MSDN. We are working on a new XSD designer and you will see more CTP's on as this takes shape.

As for XQuery, you probably know we have a subset of XQuery already supported inside SQL Server. We currently have no official plans that we can announce on a client side XQuery engine but we are definitely interested in expanding client side query processing. LINQ offers a path to this (for both relational as well as XML). ESQL provides another client-side investment. We are open to customer feedback on the relative importance of client side XQuery compared to all these other possibilities. Meanwhile we are doing some XQuery improvements in SQL Server 2008, adding LET, better datetime support, and lax validation.

As for XSLT 2.0 - we’ve heard from customers and understand the improvements in XSLT 2.0 over XSLT 1.0, but right now we’re in the middle of a big strategic investment in LINQ and EDM for the future of the data programming platform which we think will create major improvements in programming against all types of data. But we are always re-evaluating our technology investments so if your readers want to ramp up their volume on XSLT 2.0 please ask them to drop us a line with their comments.

Meanwhile I was rather surprised by the positive feedback to my little XML Notepad 2007 tool. It now has over 1 million downloads and is still going strong. Not bad for a few weeks work and no marketing. So something about this tool hit the sweet spot. The interesting thing there is it reaches out to the non-programmer community and I think that is the key and it has the right balance of simplicity and usability. A Swedish customer said it is “logam” – just enough. I think you should expect to see more from Microsoft in the future that helps to make XML something that everyone on the planet can deal with easily and in a way that integrates deeply with everything else Microsoft provides.

As the XML hype is wearing off, folks are realizing that not everything that made it through the standards process needs to be implemented.   So I think you will see Microsoft continue to innovate on new XML technologies and tools like LINQ to XML and VB 9.0 XML and you’ll see Microsoft taking a more pragmatic customer-demand-driven approach to standards.    Microsoft will probably never implement every standard that comes out but I’m confident you will see Microsoft continue to be committed to the really important XML standards, like XML 1.0 and XSD, and any other standard that is essential to achieving cross-platform interoperability, including Open XML. There is enormous power in the cross-platform reach of XML and the huge industry adoption it has and I’m happy to see that Microsoft is continuing to do some really innovative work with XML.