SQL Programmability & API Development Team Blog : XML

XML: Typed and Untyped

brandon.berg — Wed, 27 Sep 2006 03:09:00 GMT

Occasionally we hear from people who are surprised to find that their XML data uses more space when typed than when untyped. In general, this is to be expected. Typed XML has some advantages over untyped XML, namely smarter query plans and the ability to constrain user input, but size usually isn't one of them. In addition to storing all the same markup information needed for untyped XML, typed XML also has type annotations. Storing values in their binary forms rather than as Unicode strings does offer some opportunities for savings, but these savings usually aren't enough to make up for the extra space needed for type annotations. The space requirements for typed values in bytes (excluding markup) are as follows:

xs:boolean - 1 byte
xs:decimal - 17 bytes
xs:float - 4 bytes
xs:double - 8 bytes
xs:date/xs:time/xs:dateTime - 8 bytes
xs:hexBinary - 1 byte per 2 characters of input
xs:base64Binary - 3 bytes per 4 characters of input

Note that all types derived from xs:decimal (xs:integer, xs:long, xs:byte, etc.) take 17 bytes. All other types, as well as all text nodes in untyped XML, are stored as Unicode strings, and use two bytes per character. The biggest opportunities for saving are with binary types (xs:hexBinary values takes up four times as much space when untyped), and to a lesser extent xs:dateTime.

The size of the type annotation varies. For attributes and elements with complex content, the annotation is seven bytes: One for the extension token (indicating an extension of the standard binary XML format), one for the number of bytes in the extension, one for a set of flags, and four for the type ID.

For elements with simple content, there are two type annotations: one for the element itself, and one for the value. The annotation for the value is the same as the annotation for an attribute: seven bytes. The annotation for the element itself is 11 bytes, the the first seven being the same as for attributes and values, and the remaining four being a pointer to the value. The pointer is necessary because an arbitrary number of attributes may come between the element and its value. This gives a total of 18 bytes for elements with simple content.

So an element typed as xs:decimal will require 35 bytes between the value and type annotation. Only if the value is at least 18 digits long does this give any savings over the untyped representation. On the other hand, an attribute of type xs:dateTime needs only 15 bytes for its value and annotation if typed, whereas its string representation would require 40-58 bytes. While it's possible in principle for a document to be smaller as typed XML than as untyped XML, space savings should not generally be expected without a specific reason, such as use of large binary values or many dateTime values.

XML Data Type Limitations

brandon.berg — Wed, 24 May 2006 00:33:00 GMT

I'd like to take some time today to explain some of the seemingly arbitrary limits placed on the XML data type, specifically those related to ID/IDREF validation, complex XML Schema types, the depth limit for XML data, and the enigmatic "XSD schema too complex" error.

ID/IDREF Validation
If your typed XML document has attributes of type xs:ID and/or xs:IDREF, SQL Server will enforce referential integrity on these attributes: Within a given document, no two attributes of type xs:ID may have the same value, and all attributes of type xs:IDREF must have the same value as some attribute of type xs:ID.

This requires the validator to remember which ID values it's seen. Since the validator does only one pass, it must also remember any IDREF values for which it has not yet encountered a corresponding ID. It is thus possible to construct an XML document which requires an arbitrary amount of memory to validate correctly. In order to prevent denial of service attacks, we capped the amount of memory available for this purpose at one megabyte. If you try to validate a document which exceeds this limit, validation will fail with error 6969:

ID/IDREF validation consumed too much memory. Try reducing the number of ID and IDREF attributes. Rearranging the file so that elements with IDREF attributes appear after the elements which they reference may also be helpful.

There's no simple way to describe the precise conditions necessary to produce this error, but the relevant factors are the number of ID and forward-referencing IDREF attributes and the lengths of their values. The cap is the same for the 32-bit and 64-bit versions of SQL Server, so there are some documents which will validate on the 32-bit version but fail to validate on the 64-bit version due to the larger pointer size.

Complex XML Schema Types
When submitting a schema to be added to an XML Schema Collection, you may see message 6998:

Type or content model '[TypeName]' is too complicated. It may be necessary to reduce the number of enumerations or the size of the content model.

When a type is needed for validation, the validator loads its definition from metadata and compiles it into a format suitable for quick validation. In order to prevent any one type from using too much memory, SQL Server caps the size of a compiled type at one megabyte. SQL Server compiles all types and performs this check when the schema is imported in order to avoid accepting types which exceed the limit.

As with the ID/IDREF limit, there's no simple way to describe precisely the conditions necessary to exceed this limit. Having a large number of attributes, a content model with many particles (xs:sequence, xs:choice, xs:all, xs:element, or xs:any), or many enumeration facets are the most likely causes. Note that the properties inherited from the base type or imported via xs:group or xs:attributeGroup references are expanded in the compiled type definition, so it's possible for a type to exceed the limit just by adding a few attributes to its base type, if the base type is near the limit.

The types of child elements, however, do not contribute to the limit. For example, you should have no problem defining a type whose content model contains several child elements, each of which has a different type whose compiled representation is 500K. If you find yourself running up against this limit, it may be helpful to split the type's properties between two or more sub-types.

"Schema Too Complex"
When adding a schema to an XML Schema Collection, you may occasionally run into error 2362:

XSD schema too complex.

This is somewhat misleading; what it actually means is that SQL Server is running low on stack space. We rely heavily on recursion for parsing and semantic validation of XML Schema documents, and in rare (and usually intentionally pathological) cases, this presents a danger of stack overflow, which would kill the process and crash the server. To prevent this, we check the remaining stack space at recursion points and abort the transaction if it's low enough to cause concern.

If you encounter this error and your schema is not intentionally pathological, you may be able to make some semantically insignificant changes that will allow SQL Server to process your schema. The most common causes of recursion are nesting and forward references. If you have several anonymous types nested in the Russian-doll style, it may help to unnest them and move the local element or type definitions up to the global level. Additionally, it may help to rearrange schema components to eliminate forward references--that is, try to make sure that component definitions precede their references in document order.

XML Depth Limit
Finally, SQL Server limits the depth of any XML instance, typed or untyped, to 128 levels. Conversion of a string with deeper nesting to XML will fail with error 6335:

XML datatype instance has too many levels of nested nodes. Maximum allowed depth is 128 levels.

We impose this limit in order to guarantee that we will be able to create an XML index for any XML column. In SQL Server, The primary key of an XML index consists of the primary key of the base table and the ordpath. The maximum length for an index key in SQL Server is 900 bytes, so the combined length of the base table's primary key and the ordpath must be 900 bytes or less. We decided to impose a limit of 128 bytes on the primary key of the base table, leaving 772 bytes for the ordpath. Based on the properties of ordpath, we decided that 128 levels would be a good upper limit to ensure that the ordpath never exceeds the maximum size.

We'd like to know if any of these limitations are causing problems, so please leave a comment if you've run into any of these. That the depth limit causes problems for SHOWPLAN_XML is a known issue, but we're not aware of practical problems with any of the other limitations.

Check Out Denis Ruckebusch's Blog

brandon.berg — Tue, 25 Apr 2006 00:30:00 GMT

I'm adding a link in the sidebar to Denis Ruckebusch's blog. Denis is one of our testers for the XML data type, and he has two recent posts worth reading if you use XML in SQL Server 2005. The first explains how to modify a schema to work around some of the limitations in our support for XML Schema. The second highlights some of the improvements we've made in Service Pack 1.

Dealing with the Limitations of XSD Content Models

brandon.berg — Tue, 25 Apr 2006 00:09:00 GMT

There was recently a question on an internal mailing list about how to model RSS 2.0 with XML Schema. The short answer is that it can’t be done. The problem lies with the content model of the <channel> element, which consists of three elements which must appear exactly once, several elements which may optionally appear once, and one element, <item>, which may appear any number of times. Additionally, the items may appear in any order.

You can’t model this with <xs:all>, because its children may not have maxOccurs > 1. And you can’t model it with <xs:sequence>, because that requires that items appear in a particular order. Ideally, the best solution would have been to take the limitations of XML Schema into account when designing the RSS specification. For example, we might require the children to appear in a specific order. However, this does have the effect of making authoring more difficult (on the other hand, since RSS files are usually not authored manually, this may be acceptable). Another alternative might be to place all the <item> elements under a wrapper element, perhaps called <items>.

<xs:all>

<xs:element name=”required1”/>

  <xs:element name=”required2”/>

  <xs:element name=”optional1” minOccurs=”0”/>

  <xs:element name=”optional2” minOccurs=”0”/>

  <xs:element name=”items”>

    <xs:complexType>

      <xs:sequence>

        <xs:element name=”item” minOccurs=”0” maxOccurs=”unbounded”/>

      </xs:sequence>

    </xs:complexType>

</xs:element>

</xs:all>

Unfortunately, when dealing with an existing specification like RSS, we usually don’t have the luxury of making breaking changes, so we’ll have to find a way to work around these issues problems. Arguably the best solution is to underspecify the content model in the schema by allowing more liberal occurrence constraints, and then to perform additional checks at the application level. For example:

<xs:choice maxOccurs=”unbounded”>

<xs:element name=”required”/>

<xs:element name=”optional”/>

<xs:element name=”item”/>

</xs:choice>

The content model above will allow one or more of any of the above elements in any order. You would then have to write additional application code to verify that exactly one of each required element and no more than one of each optional element appears.

But if for some reason you absolutely must validate RSS or some schema with a similar issue using only XML Schema, there’s a way to do it. The basic idea is to put everything in an unbounded <xs:choice> (as in the example above), and then use identity constraints (<xs:unique> and <xs:key>) to enforce cardinality limits. Use <xs:key> for required elements and <xs:unique> for optional elements.

Identity constraints won’t work without a field, so you also need to add a fixed attribute to each element whose cardinality you want to constrain. This attribute need not be present in the actual instance documents, since the schema processor will assume the existence of fixed attributes if they’re not explicitly specified.

There are some drawbacks to this approach:

SQL Server doesn’t support identity constraints in XML Schema Collections, so you’d have to validate it using UDFs or with client-side code (System.Xml does support them).
The error messages may be a bit confusing, since you’re using identity constraints in a non-standard way.
You can get 0-1, 1-1, and 0-unbounded cardinalities this way. I’m not aware of a way to get 1-unbounded.
The post-schema validation infoset will contain the fixed attribute.

The XSD below illustrates this technique:

<xs:schema xmlns:x="http://ns" targetNamespace="http://ns"xmlns:xs="http://www.w3.org/2001/XMLSchema" elementFormDefault="qualified">

<xs:element name="root">

<xs:complexType>

    <xs:choice maxOccurs="unbounded">

      <xs:element name="required">

        <xs:complexType>

          <xs:simpleContent>

            <xs:extension base="xs:string">

              <xs:attribute name="required" fixed="required"/>

            </xs:extension>

          </xs:simpleContent>

        </xs:complexType>

      </xs:element>

      <xs:element name="optional">

        <xs:complexType>

          <xs:simpleContent>

            <xs:extension base="xs:string">

              <xs:attribute name="optional" fixed="optional"/>

            </xs:extension>

          </xs:simpleContent>

        </xs:complexType>

      </xs:element>

      <xs:element name="many"/>



    </xs:choice>

</xs:complexType>

<xs:key name="reqkey">

    <xs:selector xpath="."/>

    <xs:field xpath="x:required/@required"/>

</xs:key>

<xs:unique name="optunique">

    <xs:selector xpath="."/>

    <xs:field xpath="x:optional/@optional"/>

</xs:unique>

</xs:element>

</xs:schema>

I mention this technique more as a curiosity than as a way of recommending it. In most cases, I think the solutions mentioned above (better schema design or additional checks in the application code) would be preferable. Nevertheless, the possibility is there for those who feel a need to take advantage of it.

An article on FOR XML in Microsoft SQL Server 2000 and 2005

ekogan — Fri, 14 Apr 2006 01:04:00 GMT

I just posted an article What does server side FOR XML return? (http://blogs.msdn.com/sqlprogrammability/articles/576095.aspx) which gives some details of the design and performance characteristics of various FOR XML flavors.

I also added links to XML whitepapers and publications on the design of XML features in Microsoft SQL Server 2005 to the links section of the main SQL Programmability & API Development Team Blog page (http://blogs.msdn.com/sqlprogrammability/default.aspx).

Best regards,

Eugene Kogan

Technical Lead

Microsoft SQL Server Relational Engine

Name resolution, default schema, implicit schema Part II

andrewz — Tue, 04 Apr 2006 00:39:00 GMT

Last section clarified some background, let us continue with Name Resolution Algorithm for Types and XML Schema Collections

[Algorithm]

As implied by general syntax for referencing entities, types or XML schema collections cannot be referenced across database. If schema name is unspecified, SQL server uses “sys first” algorithm as outlined below.

1. First, Look in the “sys” schema if not DDL access, then

2. Look in the active default schema, then

3. Look in “dbo” schema.

If schema name is specified, SQL server simply looks in the designated schema.

Note:

· Database collation is used to match schema, type and XML schema collection names.

· DDL access includes CREATE/DROP/ALTER on types or XML schema collections.

[Backward Compatibility]

During upgrade, system types are put in “sys” schema, user defined types are put in “dbo” schema. In SQL 2000, user defined types can only be created thru sp_addtype. This procedure is modified in SQL2005 to always put newly created types in “dbo” schema. Since types can only be referenced using single part name in SQL 2000, above algorithm clearly satisfies backward compatibility requirement.

Note that new type DDL can be used to create types in schemas other than “dbo” and “sys”. It is recommended to qualify with schema when referencing user defined types in applications, if active default schema cannot be dependent on.

XML schema collections are newly introduced in SQL2005, so there is no backward compatibility issue with it.

Next time will look at name resolution algorithm for objects.