Typed XML in SQL Server 2005 : Schema Collections

Silent XQuery failures

denisruc — Fri, 20 Apr 2007 03:10:00 GMT

A newsgroup post I read earlier today reminded me that there are cases where failure to get the expected result when using XQuery can sometimes be difficult to diagnose.

Here’s what the poster was doing

declare @xml xml

set @xml = '<root><test></test></root>'

set @xml.modify('replace value of (/root/test/text())[1] with "test

new value"')

select @xml

This user expected to get something like <root><test>test new value</test></root> but instead, the resulting instance looked like <root><test /></root>.

The server never gave any error or warning, yet the XML instance appeared to be untouched.

The problem, as you might have guessed already, is that XPath expression (/root/test/text())[1] returns the empty sequence. In the original instance, ‘test’ is an empty element. It has no children at all. Therefore the expression (/root/test/text())[1] doesn’t point to any existing node. The solution to this problem is to insert a new test node inside the ‘test’ element, like this

set @xml.modify('insert text{"test new value"} as first into (/root/test)[1] ')

In general, when you use the modify() method and the target XPath expression returns the empty sequence, nothing happens. The only exception to this rule is when the server can statically determine that the result will be empty. That’s when XML schemas come in handy.

For example, let’s look at the following example

DECLARE @x XML

SET @x = '<root><x/><y/><z/></root>'

SET @x.modify('delete /root[1]/a[1]')

Let’s suppose I made a typo and instead of typing ‘z’ in my XPath expression I typed ‘a’. The query doesn’t do anything because with this particular XML instance the XPath expression /root[1]/a[1] returns the empty sequence (it doesn’t point to any existing node). There is no way to know that before executing the query. Statically, the type of this XPath expression is element(a, xdt:untyped)?

Now, let’s try something similar when a schema collection is associated with the XML variable.

CREATE XML SCHEMA COLLECTION SC AS '

</sequence>

</complexType>

</element>

</schema>'

DECLARE @x XML(SC)

SET @x = '<root><x/><y/><z/></root>'

SET @x.modify(' delete /root[1]/a[1]')

This time the query fails with the following error message:

XQuery [modify()]: There is no element named 'a' in the type 'element(root,#anonymous) ?'.

The server was able to use the schema to determine that my XPath expression will never point to an existing node and sent me a helpful error message. Even though typing your data doesn’t mean you’ll catch all those mistakes, it still helps when debugging XML queries.

-
Disclaimer:
This posting is provided “AS IS” with no warranties, and confers no rights.
Use of included script samples are subject to the terms specified at http://www.microsoft.com/info/cpyright.htm.

Mixed content (part 3)

denisruc — Wed, 14 Mar 2007 02:22:00 GMT

It’s been a few weeks since my last post about mixed content so let’s pick up right where we left off.

This time we will look at DML operations on mixed content. We’ll keep using the XML schema as well as the table I created in the first post of this series.

Let’s insert an instance to work with.

INSERT INTO T VALUES (3, CONVERT(XML(SC_Letter), '<l:letter xmlns:l="urn:letter" xmlns:t="urn:letter:tags">

<l:date>2006-10-15-08:00</l:date>

<l:sender>

<l:FirstName>James</l:FirstName>

<l:LastName>Kirk</l:LastName>

<l:Line1>1234 Elm Street</l:Line1>

<l:City>Redmond</l:City>

<l:ZIPCode>98052</l:ZIPCode>

<l:State>WA</l:State>

</l:sender>

<l:recipient>

<l:FirstName>Jean-Luc</l:FirstName>

<l:LastName>Picard</l:LastName>

<l:Line1>13 rue des Lilas</l:Line1>

<l:City>Ronchin</l:City>

<l:ZIPCode>59555</l:ZIPCode>

<l:Country>France</l:Country>

</l:recipient>

Dear Sir,

I received your letter dated <t:date>2006-10-10+00:00</t:date> this morning. It would be my pleasure to attend the annual symposium <t:title>XML database systems in class-B Federation Spacecrafts</t:title>.

Please contact my assistant <t:person>Mr. Spock</t:person> at <t:email>spock@space-federation.org</t:email> to make all necessary arrangements.

Sincerely,

James T. Kirk

</l:letter>', 1))

DML operations on elements

There are no behavior differences between element-only content and mixed content. You can insert, delete, and replace the value of elements provided the resulting instance validates against the schema.

For example, let’s update the name of the assistant mentioned in the letter.

UPDATE T SET xmlCol.modify(' declare namespace l = "urn:letter";

declare namespace t = "urn:letter:tags";

replace value of (/l:letter//t:person[data(.) = "Mr. Spock"])[1] with "Capt. Spock"')

WHERE iCol = 3

You can query the instance back and see for yourself that the value of the element has been updated.

We could also decide not to mention the assistant by name and delete the element.

UPDATE T SET xmlCol.modify(' declare namespace l = "urn:letter";

declare namespace t = "urn:letter:tags";

delete (/l:letter//t:person[data(.) = "Capt. Spock"])[1]')

WHERE iCol = 3

If you query the instance back you can see that the letter now says

Please contact my assistant at <t:email>spock@space-federation.org</t:email> to make all necessary arrangements.

Something very interesting happened here that is not immediately visible. If you remember, in our previous posts, we used the following query to examine the nodes in our instance

SELECT x.query('.'), x.query('if (. instance of text()) then "text" else

if (. instance of element()) then "element" else ""')

FROM T

CROSS APPLY xmlCol.nodes('/*:letter[1]/node()') AS Tref(x)

WHERE iCol = 3

Had we run this query prior to deleting the <person> element, we would’ve seen this as part of our results (partial results printed to save space)

. Please contact my assistant	text
<p1:person xmlns:p1="urn:letter:tags">Capt. Spock</p1:person>	element
at	text
<p1:email xmlns:p1="urn:letter:tags">spock@space-federation.org</p1:email>	element
to make all necessary arrangements. Sincerely, James T. Kirk	text

Now that we’ve deleted the element, the results look like this (partial results again)

. Please contact my assistant at	text
<p1:email xmlns:p1="urn:letter:tags">spock@space-federation.org</p1:email>	element
to make all necessary arrangements. Sincerely, James T. Kirk	text

Since we delete an element that was between two text nodes, those nodes merged to form only one.

DML operations on text nodes

Text nodes can be updated, inserted or deleted.

First, let’s replace the value of a text node.

UPDATE T SET xmlCol.modify(' declare namespace l = "urn:letter";

declare namespace t = "urn:letter:tags";

replace value of /l:letter[1]/text()[6] with ". I suggest that you talk to my asisstant, who can be contacted at "')

WHERE iCol = 3

You can query the instance to see that the text of the letter was modified appropriately.

Now let’s insert a text node after an existing one

UPDATE T SET xmlCol.modify(' declare namespace l = "urn:letter";

declare namespace t = "urn:letter:tags";

insert text{"		PS: Live long and prosper."} as last into /l:letter[1]')

WHERE iCol = 3

If you look at the nodes, you’ll see that the new text node we inserted merged with the one preceding it to form one text node so that all text between two elements tags counts as only one text node (partial results printed below)

. I suggest that you talk to my asisstant, who can be contacted at

text

<p1:email xmlns:p1="urn:letter:tags">spock@space-federation.org</p1:email>

element

to make all necessary arrangements.

Sincerely,

James T. Kirk

PS: Live long and prosper.

text

Complementing XSD with CHECK constraints

denisruc — Wed, 23 Aug 2006 02:43:00 GMT

I was recently asked to look at a customer’s question. This person wanted to write a schema that would validate instances like the following one.

<a>

<an_element att1="data 1" att2="data 2"/>

</a>

</TopElement>

The customer had two requirements for the children of element ‘a’.

1. They should be allowed to have any valid unqualified name.

2. They are required to contain both attributes att1 and att2.

The first requirement can easily be taken care of using element wildcards but I couldn’t come up with a way to define the second constraint in an XML schema.

There are constraints like this one that are not expressible (or not easily expressible) using XSD but that are easy to check using a simple XQuery. The usual workaround is to use a T-SQL CHECK constraint.

First, let’s create a schema collection

CREATE XML SCHEMA COLLECTION SC AS '

</sequence>

</complexType>

</element>

</sequence>

</complexType>

</element>

</schema>

If we type an XML column with this collection, we will accept all unqualified elements as children of ‘a’, whether they contain the required attributes or not. In order to implement this second requirement, we are going to use a CHECK constraint.

Since CHECK constraints don’t allow calls to XML data type methods, we need to create a user-defined function.

CREATE FUNCTION dbo.checkAttrs(@x XML(SC))

RETURNS bit

BEGIN

RETURN ~(@x.exist('/TopElement/a/*[not( ./@att1 and ./@att2 )]'))

END

The function defined above checks for the presence of a child element of ‘a’ that doesn’t contain both attributes att1 and att2. If such an element exists, the function returns 0. If none is found, the function returns 1.

With this function, we can now create a table with an XML column that behaves according to the user’s specifications.

CREATE TABLE T(xmlCol XML(SC)

CHECK (1 = dbo.checkAttrs(xmlCol)))

When an XML instance is inserted into or modified inside table T it is first validated against the schemas present in the ‘SC’ schema collection. Then the server verifies that CHECK constraint holds true. If the instance doesn’t validate against the schema, or if it violates the check constraint, it is rejected.

In conclusion this short example shows how you can harness the power of XQuery through the T-SQL CHECK syntax in order to implement constraints that cannot be defined with XSD alone.

Explaining the reasons behind some of XML datatype's limitations

denisruc — Thu, 29 Jun 2006 07:24:00 GMT

If you use XML in SQL Server 2005, and especially if you use it in association with schemas, I suggest you read Brandon's recent blog post about the limitations that were placed on the XML datatype. Brandon is one of our developers and he's got first-hand information about the design decisions that lie behind those limitations.

On a different note, I know my contributions to this blog have been few and far between recently. There's no shortage of topics I'd like to address, just a shortage of time.

Restricting the contents of a typed XML column using triggers

denisruc — Sat, 13 May 2006 02:26:00 GMT

A typed XML column/variable is constrained by a schema collection. That means only instances that can be validated against the schemas in the collection will be accepted. You can tighten the rules a little bit with the use of the DOCUMENT facet, which adds the extra requirement that only well formed XML documents be accepted (no fragments).

For some users, this isn’t enough. They would also like to be able to constrain their XML columns or variables to a single namespace or even to a single element.

Unfortunately the DDL syntax doesn’t allow for such requirements. There are workarounds though, and we’ll try to look at a simple example.

First, let’s create an Xml Schema Collection and a table.

CREATE XML SCHEMA COLLECTION myCollection AS '

<xsd:schema xmlns:xsd="http://www.w3.org/2001/XMLSchema"

targetNamespace="http://a">

<xsd:element name="a">

<xsd:complexType>

<xsd:sequence>

<xsd:element name="strElement" type="xsd:string"/>

<xsd:element name="bElement" type="xsd:boolean"/>

</xsd:sequence>

</xsd:complexType>

</xsd:element>

</xsd:schema>

<xsd:schema xmlns:xsd="http://www.w3.org/2001/XMLSchema"

targetNamespace="http://b">

<xsd:element name="b">

<xsd:complexType>

<xsd:sequence>

<xsd:element name="strElement" type="xsd:string"/>

<xsd:element name="floatElement" type="xsd:float" minOccurs="0"/>

</xsd:sequence>

</xsd:complexType>

</xsd:element>

</xsd:schema>

<xsd:schema xmlns:xsd="http://www.w3.org/2001/XMLSchema"

targetNamespace="http://main"

xmlns:ns="http://main"

xmlns:b="http://b"

xmlns:a="http://a">

<xsd:import namespace="http://a"/>

<xsd:import namespace="http://b"/>

<xsd:element name="root" type="ns:rootType"/>

<xsd:complexType name="rootType">

<xsd:choice minOccurs="0" maxOccurs="unbounded">

<xsd:element ref="a:a"/>

<xsd:element ref="b:b"/>

</xsd:choice>

</xsd:complexType>

</xsd:schema>

CREATE TABLE myTable (

date Datetime,

xmlCol XML(myCollection)

)

At this point any valid instance of elements “root” (namespace “http://main”), “a” (namespace “http://a”) and “b” (namespace “http://b”) can be inserted into the table.

Let’s decide that we should only accept instances of elements from namespace “http://main”.

Let’s create the following trigger

CREATE TRIGGER CheckNS_OnInsert_Trigger on myTable INSTEAD OF INSERT

BEGIN

INSERT INTO myTable

SELECT date, xmlCol.query('for $i in /* return (if (namespace-uri($i) = "http://main") then $i else ())')

FROM inserted

END

Now, let’s try the following insertion

INSERT INTO myTable VALUES ('1/1/2001 15:00:00', '

<x:a xmlns:x="http://a"><strElement>Data a</strElement><bElement>false</bElement></x:a>

<x:root xmlns:x="http://main"/>

<x:b xmlns:x="http://b"><strElement> Data b</strElement></x:b>

<x:root xmlns:x="http://main">

<x:a xmlns:x="http://a"><strElement>Data a</strElement><bElement>true</bElement></x:a>

<x:b xmlns:x="http://b"><strElement> Data b</strElement></x:b>

</x:root>

<x:b xmlns:x="http://b"><strElement/></x:b>'

)

Apparently it succeeds (no error message back) but let’s pull the data back out of the table.

select xmlCol FROM myTable WHERE date = '1/1/2001 15:00:00'

The data looks like this (I pretty-printed it for better legibility)

<x:root xmlns:x="http://main" />

<x:root xmlns:x="http://main">

<x:a xmlns:x="http://a">

</x:a>

<x:b xmlns:x="http://b">

</x:b>

</x:root>

All instances of top level elements that were not qualified with namespace “http://main” were eliminated by the trigger.

The problem is, we can still perform a DML operation that will insert a top level element from any namespace at the root, like this

UPDATE myTable

SET xmlCol.modify('insert <x:b xmlns:x="http://b"><strElement/></x:b> as last into / ')

WHERE date = '1/1/2001 15:00:00'

When we pull the data out of the table again we get

<x:root xmlns:x="http://main" />

<x:root xmlns:x="http://main">

<x:a xmlns:x="http://a">

</x:a>

<x:b xmlns:x="http://b">

</x:b>

</x:root>

<x:b xmlns:x="http://b"><strElement></strElement></x:b>

To remedy this, we need to create another trigger that will catch any “illegal” element(s) inserted during updates.

CREATE TRIGGER CheckNS_OnUpdate_Trigger on myTable AFTER UPDATE

BEGIN

IF UPDATE (xmlCol)

UPDATE myTable

SET xmlCol.modify('delete /*[namespace-uri(.)!="http://main"] ')

END

Now let’s try that update again

UPDATE myTable

SET xmlCol.modify('insert <x:b xmlns:x="http://b"><strElement/></x:b> as last into / ')

WHERE date = '1/1/2001 15:00:00'

The query seems to succeed but when we query the data out we get this:

<x:root xmlns:x="http://main" />

<x:root xmlns:x="http://main">

<x:a xmlns:x="http://a">

</x:a>

<x:b xmlns:x="http://b">

</x:b>

</x:root>

All top level elements from namespaces other than “http://main”, including the one present before we created the trigger are gone.

This trigger deletes all top level element whose namespace uri isn’t “http://main”, in all rows. Performance-wise this might become a problem once your table contains more than a few rows. It can probably be made more effective with a primary key and a join on the inserted or delete table. I’ll leave the optimization to you.

One last thing is left to check. We must ensure that an element from “http://a” or “http://b” can still be inserted inside an element from “http://main”.

UPDATE myTable

SET xmlCol.modify('declare namespace main="http://main";

insert <x:b xmlns:x="http://b"><strElement/></x:b> as first into /main:root[1] ')

WHERE date = '1/1/2001 15:00:00'

When we query the data out we get this:

<x:root xmlns:x="http://main">

<x:b xmlns:x="http://b">

</x:b>

</x:root>

<x:root xmlns:x="http://main">

<x:a xmlns:x="http://a">

</x:a>

<x:b xmlns:x="http://b">

</x:b>

</x:root>

The first “root” element has now a child “b” qualified with namespace “http://b”.

Importing the Microsoft Office 2003 schemas into the database

denisruc — Wed, 03 May 2006 00:35:00 GMT

As promised earlier I'm going to give you an example using a real-world schema. I've chosen to work with the Office 2003 schemas but I'm going to limit myself to Word and Excel.

First, you need to install the schemas from here.

In the T-SQL samples below I'll always assume that the installation was performed at the default location, under "C:\Program Files\Microsoft Office 2003 Developer Resources".

First, we're going to create a collection for all the schemas necessary in order to validate a Word 2003 document. This is fairly easy. For each schema file in folder "WordprocessingML Schemas" we do the following

Load the contents into an XML variable
If necessary remove the xsd:unique, xsd:key, xsd:keyref nodes
Remove any top level processing instruction

When this is done we concatenate all those schema documents by converting them to nvarchar(MAX), using the string concatenation, and converting the result back to XML

Finally, for all lax wildcards we replace the value of the processContents attribute to "skip" and the resulting XML can be used to create the xml schema collection.

DECLARE @aml XML

SELECT @aml = xmlCol FROM OPENROWSET(Bulk 'C:\Program Files\Microsoft Office 2003 Developer Resources\Microsoft Office 2003 XML Reference Schemas\WordprocessingML Schemas\aml.xsd', SINGLE_NCLOB) as results(xmlCol)

SET @aml.modify('delete //xs:unique')

SET @aml.modify('delete /processing-instruction()')

DECLARE @w10 XML

SELECT @w10 = xmlCol FROM OPENROWSET(Bulk 'C:\Program Files\Microsoft Office 2003 Developer Resources\Microsoft Office 2003 XML Reference Schemas\WordprocessingML Schemas\w10.xsd', SINGLE_NCLOB) as results(xmlCol)

SET @w10.modify('delete /processing-instruction()')

DECLARE @wordnetaux XML

SELECT @wordnetaux = xmlCol FROM OPENROWSET(Bulk 'C:\Program Files\Microsoft Office 2003 Developer Resources\Microsoft Office 2003 XML Reference Schemas\WordprocessingML Schemas\wordnetaux.xsd', SINGLE_NCLOB) as results(xmlCol)

SET @wordnetaux.modify('delete /processing-instruction()')

DECLARE @xsdlib XML

SELECT @xsdlib = xmlCol FROM OPENROWSET(Bulk 'C:\Program Files\Microsoft Office 2003 Developer Resources\Microsoft Office 2003 XML Reference Schemas\WordprocessingML Schemas\xsdlib.xsd', SINGLE_NCLOB) as results(xmlCol)

SET @xsdlib.modify('delete /processing-instruction()')

DECLARE @office XML

SELECT @office = xmlCol FROM OPENROWSET(Bulk 'C:\Program Files\Microsoft Office 2003 Developer Resources\Microsoft Office 2003 XML Reference Schemas\WordprocessingML Schemas\office.xsd', SINGLE_NCLOB) as results(xmlCol)

SET @office.modify('delete /processing-instruction()')

DECLARE @vml XML

SELECT @vml = xmlCol FROM OPENROWSET(Bulk 'C:\Program Files\Microsoft Office 2003 Developer Resources\Microsoft Office 2003 XML Reference Schemas\WordprocessingML Schemas\vml.xsd', SINGLE_NCLOB) as results(xmlCol)

SET @vml.modify('delete /processing-instruction()')

DECLARE @wordnet XML

SELECT @wordnet = xmlCol FROM OPENROWSET(Bulk 'C:\Program Files\Microsoft Office 2003 Developer Resources\Microsoft Office 2003 XML Reference Schemas\WordprocessingML Schemas\wordnet.xsd', SINGLE_NCLOB) as results(xmlCol)

SET @wordnet.modify('delete /processing-instruction()')

DECLARE @xmlSchemas XML

SET @xmlSchemas = CONVERT(XML,

CONVERT(nvarchar(MAX), @aml) +

CONVERT(nvarchar(MAX), @w10) +

CONVERT(nvarchar(MAX), @wordnetaux) +

CONVERT(nvarchar(MAX), @xsdlib) +

CONVERT(nvarchar(MAX), @office) +

CONVERT(nvarchar(MAX), @vml) +

CONVERT(nvarchar(MAX), @wordnet))

DECLARE @iCount int

DECLARE @i int

SET @iCount = @xmlSchemas.value('count(//@processContents[. = "lax"])','int')

SET @i = 0

WHILE (@i < @iCount)

BEGIN

SET @xmlSchemas.modify('replace value of (//@processContents[. = "lax"])[1] with "skip"')

SET @i = @i + 1

END

CREATE XML SCHEMA COLLECTION Word2003 AS @xmlSchemas

Importing the Excel schemas is a similar process, with two differences.

First, not all the schema files in folder "SpreadsheetML Schemas" are unicode. For those who are not, they must be loaded using the OPENROWSET parameter SINGLE_CLOB (as opposed to SINGLE_NCLOB).

The second difference is that one of the schemas imports another schema that's not on disk. File udc.xsd makes references to components from the Soap Envelope schema and relies on the schemaLocation attribute to provide the schema processor with the location of that schema. Unfortunately SQL Server 2005, even though it doesn't reject the schemaLocation attribute, cannot use it to retrieve a missing schema. The solution is very low-tech. It consisted in manually retrieving the schema from the web and fixing it by hand (doubling-up the single quotes for example) so it can be copy-pasted into our T-SQL script. There are probably better solutions, such as making use of the CLR to retrieve the schema directly from the web site, but for a single missing schema, it was easier to deal with it manually.

DECLARE @c XML

SELECT @c = xmlCol FROM OPENROWSET(Bulk 'C:\Program Files\Microsoft Office 2003 Developer Resources\Microsoft Office 2003 XML Reference Schemas\SpreadsheetML Schemas\c.xsd', SINGLE_CLOB) as results(xmlCol)

SET @c.modify('delete /processing-instruction()')

DECLARE @dt XML

SELECT @dt = xmlCol FROM OPENROWSET(Bulk 'C:\Program Files\Microsoft Office 2003 Developer Resources\Microsoft Office 2003 XML Reference Schemas\SpreadsheetML Schemas\dt.xsd', SINGLE_CLOB) as results(xmlCol)

SET @dt.modify('delete /processing-instruction()')

DECLARE @excel XML

SELECT @excel = xmlCol FROM OPENROWSET(Bulk 'C:\Program Files\Microsoft Office 2003 Developer Resources\Microsoft Office 2003 XML Reference Schemas\SpreadsheetML Schemas\excel.xsd', SINGLE_CLOB) as results(xmlCol)

SET @excel.modify('delete /processing-instruction()')

DECLARE @excel2003xml XML

SELECT @excel2003xml = xmlCol FROM OPENROWSET(Bulk 'C:\Program Files\Microsoft Office 2003 Developer Resources\Microsoft Office 2003 XML Reference Schemas\SpreadsheetML Schemas\excel2003xml.xsd', SINGLE_CLOB) as results(xmlCol)

SET @excel2003xml.modify('delete /processing-instruction()')

DECLARE @excelss XML

SELECT @excelss = xmlCol FROM OPENROWSET(Bulk 'C:\Program Files\Microsoft Office 2003 Developer Resources\Microsoft Office 2003 XML Reference Schemas\SpreadsheetML Schemas\excelss.xsd', SINGLE_CLOB) as results(xmlCol)

SET @excelss.modify('delete /processing-instruction()')

DECLARE @office XML

SELECT @office = xmlCol FROM OPENROWSET(Bulk 'C:\Program Files\Microsoft Office 2003 Developer Resources\Microsoft Office 2003 XML Reference Schemas\SpreadsheetML Schemas\office.xsd', SINGLE_NCLOB) as results(xmlCol)

SET @office.modify('delete /processing-instruction()')

DECLARE @rowsetschema XML

SELECT @rowsetschema = xmlCol FROM OPENROWSET(Bulk 'C:\Program Files\Microsoft Office 2003 Developer Resources\Microsoft Office 2003 XML Reference Schemas\SpreadsheetML Schemas\rowsetschema.xsd', SINGLE_CLOB) as results(xmlCol)

SET @rowsetschema.modify('delete /processing-instruction()')

DECLARE @rowsset XML

SELECT @rowsset = xmlCol FROM OPENROWSET(Bulk 'C:\Program Files\Microsoft Office 2003 Developer Resources\Microsoft Office 2003 XML Reference Schemas\SpreadsheetML Schemas\rowsset.xsd', SINGLE_CLOB) as results(xmlCol)

SET @rowsset.modify('delete /processing-instruction()')

DECLARE @schema XML

SELECT @schema = xmlCol FROM OPENROWSET(Bulk 'C:\Program Files\Microsoft Office 2003 Developer Resources\Microsoft Office 2003 XML Reference Schemas\SpreadsheetML Schemas\schema.xsd', SINGLE_CLOB) as results(xmlCol)

SET @schema.modify('delete /processing-instruction()')

DECLARE @udc XML

SELECT @udc = xmlCol FROM OPENROWSET(Bulk 'C:\Program Files\Microsoft Office 2003 Developer Resources\Microsoft Office 2003 XML Reference Schemas\SpreadsheetML Schemas\udc.xsd', SINGLE_CLOB) as results(xmlCol)

SET @udc.modify('delete /processing-instruction()')

DECLARE @udcsoap XML

SELECT @udcsoap = xmlCol FROM OPENROWSET(Bulk 'C:\Program Files\Microsoft Office 2003 Developer Resources\Microsoft Office 2003 XML Reference Schemas\SpreadsheetML Schemas\udcsoap.xsd', SINGLE_CLOB) as results(xmlCol)

SET @udcsoap.modify('delete /processing-instruction()')

DECLARE @udcxmlfile XML

SELECT @udcxmlfile = xmlCol FROM OPENROWSET(Bulk 'C:\Program Files\Microsoft Office 2003 Developer Resources\Microsoft Office 2003 XML Reference Schemas\SpreadsheetML Schemas\udcxmlfile.xsd', SINGLE_CLOB) as results(xmlCol)

SET @udcxmlfile.modify('delete /processing-instruction()')

DECLARE @vml XML

SELECT @vml = xmlCol FROM OPENROWSET(Bulk 'C:\Program Files\Microsoft Office 2003 Developer Resources\Microsoft Office 2003 XML Reference Schemas\SpreadsheetML Schemas\vml.xsd', SINGLE_NCLOB) as results(xmlCol)

SET @vml.modify('delete /processing-instruction()')

-- udc.xsd has the following import

-- <xsd:import namespace="http://schemas.xmlsoap.org/soap/envelope/" schemaLocation="http://schemas.xmlsoap.org/soap/envelope/" />

-- I retrieved the schema from the web and manually doubled up the single quotes; I also removed the big comment at the beginning.

DECLARE @soapenvelope XML

SET @soapenvelope = ' <xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema"

xmlns:tns="http://schemas.xmlsoap.org/soap/envelope/"

targetNamespace="http://schemas.xmlsoap.org/soap/envelope/" >

<xs:element name="Envelope" type="tns:Envelope" />

<xs:complexType name="Envelope" >

<xs:sequence>

<xs:element ref="tns:Header" minOccurs="0" />

<xs:element ref="tns:Body" minOccurs="1" />

<xs:any namespace="##other" minOccurs="0" maxOccurs="unbounded" processContents="lax" />

</xs:sequence>

<xs:anyAttribute namespace="##other" processContents="lax" />

</xs:complexType>

<xs:element name="Header" type="tns:Header" />

<xs:complexType name="Header" >

<xs:sequence>

<xs:any namespace="##other" minOccurs="0" maxOccurs="unbounded" processContents="lax" />

</xs:sequence>

<xs:anyAttribute namespace="##other" processContents="lax" />

</xs:complexType>

<xs:element name="Body" type="tns:Body" />

<xs:complexType name="Body" >

<xs:sequence>

<xs:any namespace="##any" minOccurs="0" maxOccurs="unbounded" processContents="lax" />

</xs:sequence>

<xs:anyAttribute namespace="##any" processContents="lax" >

<xs:annotation>

<xs:documentation>

Prose in the spec does not specify that attributes are allowed on the Body element

</xs:documentation>

</xs:annotation>

</xs:anyAttribute>

</xs:complexType>

<xs:attribute name="mustUnderstand" >

<xs:simpleType>

<xs:restriction base=''xs:boolean''>

<xs:pattern value=''0|1'' />

</xs:restriction>

</xs:simpleType>

</xs:attribute>

<xs:attribute name="actor" type="xs:anyURI" />

<xs:simpleType name="encodingStyle" >

<xs:annotation>

<xs:documentation>

''encodingStyle'' indicates any canonicalization conventions followed in the contents of the containing element. For example, the value ''http://schemas.xmlsoap.org/soap/encoding/'' indicates the pattern described in SOAP specification

</xs:documentation>

</xs:annotation>

<xs:list itemType="xs:anyURI" />

</xs:simpleType>

<xs:attribute name="encodingStyle" type="tns:encodingStyle" />

<xs:attributeGroup name="encodingStyle" >

<xs:attribute ref="tns:encodingStyle" />

</xs:attributeGroup>

<xs:element name="Fault" type="tns:Fault" />

<xs:complexType name="Fault" final="extension" >

<xs:annotation>

<xs:documentation>

Fault reporting structure

</xs:documentation>

</xs:annotation>

<xs:sequence>

<xs:element name="faultcode" type="xs:QName" />

<xs:element name="faultstring" type="xs:string" />

<xs:element name="faultactor" type="xs:anyURI" minOccurs="0" />

<xs:element name="detail" type="tns:detail" minOccurs="0" />

</xs:sequence>

</xs:complexType>

<xs:complexType name="detail">

<xs:sequence>

<xs:any namespace="##any" minOccurs="0" maxOccurs="unbounded" processContents="lax" />

</xs:sequence>

<xs:anyAttribute namespace="##any" processContents="lax" />

</xs:complexType>

</xs:schema>

SET @soapenvelope.modify('delete /processing-instruction()')

DECLARE @xmlSchemas XML

SET @xmlSchemas = CONVERT(XML,

CONVERT(nvarchar(MAX), @c) +

CONVERT(nvarchar(MAX), @dt) +

CONVERT(nvarchar(MAX), @excel) +

CONVERT(nvarchar(MAX), @excel2003xml) +

CONVERT(nvarchar(MAX), @excelss) +

CONVERT(nvarchar(MAX), @office) +

CONVERT(nvarchar(MAX), @rowsetschema) +

CONVERT(nvarchar(MAX), @rowsset) +

CONVERT(nvarchar(MAX), @schema) +

CONVERT(nvarchar(MAX), @udc) +

CONVERT(nvarchar(MAX), @udcsoap) +

CONVERT(nvarchar(MAX), @udcxmlfile) +

CONVERT(nvarchar(MAX), @vml) +

CONVERT(nvarchar(MAX), @soapenvelope))

DECLARE @iCount int

DECLARE @i int

SET @iCount = @xmlSchemas.value('count(//@processContents[. = "lax"])','int')

SET @i = 0

WHILE (@i < @iCount)

BEGIN

SET @xmlSchemas.modify('replace value of (//@processContents[. = "lax"])[1] with "skip"')

SET @i = @i + 1

END

CREATE XML SCHEMA COLLECTION Excel2003 AS @xmlSchemas

Download SP1 today!

denisruc — Fri, 21 Apr 2006 04:23:00 GMT

Yesterday we released our first Service Pack for SQL Server 2005. You might remember a while back I had written a small post about being hard at work on SP1. Back then I avoided giving any details for fear of running afoul of corporate policy but since the bits are officially out, I can now talk about the changes that were made to the XML datatype functionality.

The very first thing we did was increase the number of schemas we can import by accepting some non-deterministic schemas. In short, non-deterministic content models are now acceptable, providing that the occurrence constraints are “0”,”1” or “unbounded”. For other values, we have to keep rejecting the schemas. The reason is that for “0”, “1” and “unbounded” we can build a basic automaton to handle validation of instances. When you use integer values strictly greater than one in your constraints, you get into cases where in order to validate properly you might sometimes have to backtrack and implementation becomes more complex. So far I haven’t seen a single industry schema that contains non-deterministic content models where occurrence constraints aren’t all “0”, “1” or “unbounded” (if you know of any please let me know)
With this improvement we’re now able to import the complete set of Office 2003 schemas into SQL Server 2005 (providing you remove all unique, key and keyref nodes and change all lax wildcards to skip but as we saw in this week’s other post, that’s not very hard to do).
The developer in charge of this change was Brandon and he contributes to the SQL Programmability & API Development Team Blog. If you were tired of seeing the “XML instances of the content model of type or model group '...' can be validated in multiple ways and are not supported.” error message and this fix changed your life he’s the man to thank.

Our second improvement has to do with query performance. Static typing makes things slower when querying typed XML vs. untyped XML. In particular, when dealing with very large schema collections and complex XPath expressions static typing was a big consumer of both memory and CPU cycles. Our static typing dev AdrianB came up with implementation changes that speed up the computation of XQuery static types a great deal.
For example, we had imported all the fpML schemas (available from http://www.fpml.org) into a schema collection and compiled the following query:
select convert(xml(fpML), '').query('//*:party')
On my office server this used to take 99.952 seconds with the RTM bits. With SP1 installed, it goes down to 0.328 seconds. That's over 300 times faster! As for memory consumption the number of pages is divided by almost 140. You shouldn’t expect such drastic improvements on all queries but if you use the descendant or descendant-or-self axis in your XPath expressions you're likely to notice the difference.
The code changes were significant and required something in the neighborhood of 175 new hand-crafted tests. Fortunately Adrian is a great guy to work with and in the end everything came together fine. He doesn’t have a blog yet but if you experience significant performance increases with your queries you should know he’s the one who made it happen.

I hope this post will make you want to try SP1 for yourself. Don’t hesitate to drop me a comment to share your experiences, good or bad.

-
Disclaimer:
This posting is provided “AS IS” with no warranties, and confers no rights.
Use of included script samples are subject to the terms specified at http://www.microsoft.com/info/cpyright.htm.

Importing an existing XML schema into SQL Server 2005

denisruc — Wed, 19 Apr 2006 04:33:00 GMT

In my last post I addressed the issue of non deterministic content models, which can be valid with respect to the W3C’s standards, but which are always rejected by SQL Server 2005. There are a few other XSD features that SQL Server 2005 doesn’t support. There are also challenges inherent to the nature of SQL Server 2005. As a result it is sometimes impossible to import an existing schema as-is if it wasn’t designed with SQL Server in mind. Fixes must be used, and workarounds must be applied, before the schema can be successfully imported.

Today I’m going to walk you through the import of a schema that contains multiple features that conflict with SQL Server 2005’s implementation of XSD. To maximize the value of this example I will use a made-up schema. In a future post I will show how the techniques described here can apply to a real-world schema.

Let’s assume that there is a directory C:\Example on my SQL Server 2005 machine and that it contains the following three schema files

order.xsd

<xsd:schema xmlns:xsd="http://www.w3.org/2001/XMLSchema"
targetNamespace="http://www.example.com/denisruc"
xmlns="http://www.example.com/denisruc"
xmlns:t="http://www.example.com/denisruc/types">

<xsd:import namespace="http://www.example.com/denisruc/types" schemaLocation="types.xsd"/>

<xsd:complexType name="itemType">
 <xsd:sequence>
 <xsd:element name="name" type="t:nameType"/>
 <xsd:element name="description" type="xsd:string"/>
 </xsd:sequence>
 <xsd:attribute name="referenceId" type="t:referenceType"/>
 <xsd:attribute name="price" type="t:priceType"/>
</xsd:complexType>

<xsd:complexType name="itemOrderType">
 <xsd:complexContent>
 <xsd:extension base="itemType">
 <xsd:sequence>
 <xsd:any processContents="lax" namespace="##other" minOccurs="0" maxOccurs="unbounded"/>
 </xsd:sequence>
 <xsd:attribute name="quantity" type="xsd:integer"/>
 </xsd:extension>
 </xsd:complexContent>
</xsd:complexType>

<xsd:complexType name="orderType">
 <xsd:sequence minOccurs="0" maxOccurs="unbounded">
 <xsd:element name="Item" type="itemOrderType"/>
 </xsd:sequence>
</xsd:complexType>

<xsd:element name="Order" type="orderType">
 <xsd:unique name="refAndname">
 <xsd:selector xpath="Item"/>
 <xsd:field xpath="@referenceId"/>
 <xsd:field xpath="name"/>
 </xsd:unique>
</xsd:element>

</xsd:schema>

types.xsd

<xsd:schema xmlns:xsd="http://www.w3.org/2001/XMLSchema"
targetNamespace="http://www.example.com/denisruc/types"
xmlns="http://www.example.com/denisruc/types">

<xsd:include schemaLocation="nameType.xsd"/>

<xsd:simpleType name="referenceType">
 <xsd:restriction base="xsd:string">
 <xsd:pattern value="[A-Z](\-[0-9]{4}){3}"/>
 </xsd:restriction>
</xsd:simpleType>

<xsd:simpleType name="priceType">
 <xsd:restriction base="xsd:decimal">
 <xsd:fractionDigits value="2"/>
 </xsd:restriction>
</xsd:simpleType>

</xsd:schema>

nameType.xsd

<xsd:schema xmlns:xsd="http://www.w3.org/2001/XMLSchema"
targetNamespace="http://www.example.com/denisruc/types"
xmlns="http://www.example.com/denisruc/types">

<xsd:simpleType name="nameType">
 <xsd:restriction base="xsd:string">
 <xsd:maxLength value="20"/>
 </xsd:restriction>
</xsd:simpleType>

</xsd:schema>

Incompatibilities with SQL Server 2005

The first problem to solve is that of schema dependencies.

First, you have to look at the use of <xsd:import> in the schema files. You cannot import the files in any order because you might leave unresolved references.

For example, if you load up the contents of order.xsd and try to create a schema collection, like this

DECLARE @order varchar(MAX)
SELECT @order = xmlCol FROM OPENROWSET(Bulk 'C:\\example\\order.xsd', SINGLE_CLOB) as results(xmlCol)
create xml schema collection SC AS @order

you will get the following error
Reference to an undefined name 'nameType' within namespace 'http://www.example.com/denisruc/types'
The contents of the schema with namespace ‘http://www.example.com/denisruc/types’ have to already be present in the collection when you import the schema from order.xsd.

The other dependency is introduced by <xsd:include>. To resolve this, you can use an external tool. Books Online contains such a tool at http://msdn2.microsoft.com/en-us/library/ms190756(SQL.90).aspx. In this case, there is really no need to pre-process the schema files. We can import the contents of both types.xsd and nameType.xsd in one DDL operation since you can use multiple schema fragments with CREATE XML SCHEMA COLLECTION or ALTER XML SCHEMA COLLECTION, even if those schema fragments have the same target namespace.

In general, if possible, it is better to import all the schemas at once, since this also solves the problem of circular dependencies (when A imports B and B imports A).

The second problem to solve is that of the “lax” wildcard. SQL Server 2005 supports “strict” and “skip” but not “lax”. The workaround in this case is to use “skip”. The same wildcards instances will be allowed by the server but no type information will be available.

Finally we need to remove the <xsd:unique> node since this feature is not supported

“Fixing” and importing the schema

We could perform all the changes described above directly on the schema files but this might not be desirable and for big schemas, with lots of incompatibilities, this would be time-consuming.
Instead, we can take advantage of the XML datatype itself by loading the files into an XML variable and performing the changes as XML DML operations.

-- First we import all three files into varchar(MAX) variables
DECLARE @order varchar(MAX)
SELECT @order = xmlCol FROM OPENROWSET(Bulk 'C:\\example\\order.xsd', SINGLE_CLOB) as results(xmlCol)

DECLARE @types varchar(MAX)
SELECT @types = xmlCol FROM OPENROWSET(Bulk 'C:\\example\\types.xsd', SINGLE_CLOB) as results(xmlCol)

DECLARE @nametype varchar(MAX)
SELECT @nametype = xmlCol FROM OPENROWSET(Bulk 'C:\\example\\nametype.xsd', SINGLE_CLOB) as results(xmlCol)

-- concatenate all three files and convert to XML datatype
DECLARE @schemas XML
SET @schemas = CONVERT (XML, @order + @types + @nametype)

-- first remove the <xsd:include> tag
SET @schemas.modify('delete /xs:schema/xs:include')

-- then change all the 'lax' wildcards to 'skip'

-- first we count how many of those wildcards there are
DECLARE @iCount int
SELECT @iCount = @schemas.value('count(//xs:any[@processContents="lax"])','int')
-- then we run the DML operation as many times as necessary
DECLARE @i int
SET @i = @iCount
WHILE (@i > 0)
BEGIN
SET @i = @i - 1
SET @schemas.modify('replace value of (//xs:any[@processContents="lax"]/@processContents)[1] with "skip"')
END

-- finally we remove xsd:unique
SET @schemas.modify('delete //xs:unique')

-- the schema is now compatible with SQL Server 2005
-- We can create the schema collection
CREATE XML SCHEMA COLLECTION myCollection AS @schemas
go

The schema collection is successfully created and you can retrieve its content with SELECT XML_SCHEMA_NAMESPACE(N'dbo', N'myCollection')

If you look at this statement's results you’ll notice that only two <xsd:schema> elements are serialized out, even though you imported three files. That is because nameType.xsd was included in types.xsd. When the schema components are stored into metadata, there is no way to tell that they might have come from two different physical files. When serializing out the schema with target namespace ‘http://www.example.com/denisruc/types’ the server puts all the components under one unique schema element.

In a future post I will do this again with a real (and much more complex) industry schema.

Fixing non deterministic schemas

denisruc — Tue, 07 Feb 2006 04:23:00 GMT

If you’ve been importing XML schemas into SQL Server 2005 you might have encountered this error message

XML Validation: XML instances of the content model of type or model group '…' can be validated in multiple ways and are not supported.

The reason for this message is that the schema you tried to import into an XML schema collection contains a non deterministic content model. Non determinism in schemas is allowed by the W3C’s XSD specs (as long as the Unique Particle Attribution rule isn’t broken) but not by SQL Server 2005’s implementation of the standard.

Let’s look at a simple example

<xsd:choice minOccurs="0" maxOccurs="unbounded">

<xsd:element name="a" minOccurs="0" maxOccurs="unbounded"/>

<xsd:element name="b" minOccurs="0" maxOccurs="unbounded"/>

</xsd:choice>

If the XML document you’re validating contains five elements ‘a’ in a row, how should that be interpreted? Did we go five times through the choice particle, picking one time ‘a’ each time? Or did we go through the choice only once and pick five a’s at once? There are numerous ways to validate the instance.

Fortunately, this kind of content model is easy to fix. We can bring the occurrence constraints into the xsd:choice particle like this

<xsd:choice minOccurs="0" maxOccurs="unbounded">

<xsd:element name="a" />

<xsd:element name="b" />

</xsd:choice>

The set of valid instances will be exactly the same but this time the content model is deterministic.

There are however cases where it is impossible to fix the non-determinism issue while at the same time keeping the set of valid XML instances the same. The schema designer’s goal should be to come up with a content model that validates a superset of the XML instances valid according to the original schema.

For example, a content model like this one

<xsd:sequence minOccurs="1" maxOccurs="2">

<xsd:element name="a" minOccurs="2" maxOccurs="4"/>

<xsd:element name="b" minOccurs="0" maxOccurs="2"/>

</xsd:sequence>

could be replaced with this one

<xsd:choice minOccurs="2" maxOccurs="12">

<xsd:element name="a"/>

</xsd:element name="b"/>

</xsd:sequence>

All the instances valid against the original content model would be valid against this one also.

The problem is that a lot of instances that were invalid against the old content model would be successfully validated against this one (for example an instance that contains elements but no instance of <a/>). There are ways to reduce the number of such instances with CHECK constraints. For example, in this case we could reject all instances that contain no <a/>’s by checking that the return value of .value(‘count(/root[1]/a)’, int) is greater than zero.

In my own experience, content models such as this last one are few and far between. Most of the time, non-determinism issues are easily fixed.

How to preserve whitespaces

denisruc — Mon, 12 Sep 2005 23:30:00 GMT

For once I'm going to venture outside of the realm of typed XML and address an issue that might be of concern to users of untyped XML as well.

We've been asked a few times if we had a solution to preserve non-significant whitespaces in an XML instance. The short answer is "Yes".

First, let's sum up the problem with this simple example

declare @var XML
SET @var = '
<root>
 <a>
 Data b
 </a>
</root>'
SELECT @var
go

The query returns
<root><a>Data b</a></root>

We gave the server a nicely formatted XML instance, with all the indentations, but when we get it back, the formatting is gone. In short, the server, got rid of all the whitespace-only text nodes.

You can use the xml:space attribute and set its value to "preserve" but that is not always practical. In particular, if you use typed XML, the attribute will be rejected unless your schema specifically contains a reference to xml:space.

Fortunately, SQL Server 2005 contains a mechanism to achieve whitespace preservation. If a whitespace-only text node contains at least one entitized character then it is preserved.
For example, if we do this

DECLARE @var XML
SET @var = '<root>	<a>
 	Data b
 </a>
</root>'
SELECT @var
go

the query returns
<root>	<a>
	Data b
	</a>
</root>

The parser preserves whitespace-only text nodes and for each of them it entitizes the last character.

Of course SQL Server 2005 provides a way to do it for you in the form of an optional parameter for the CONVERT function, like this

DECLARE @var XML
SET @var = CONVERT (XML,'<root>
<a>
Data b
</a>
</root>',1)
SELECT @var
go

The query returns
<root>
	<a>
	Data b
	</a>
</root>

Things are very different for typed XML. Unless the whitespace-only text nodes are part of mixed content, they will not be preserved, no matter what option you use.
Here is an example

create xml schema collection SC as '<xsd:schema xmlns:xsd="http://www.w3.org/2001/XMLSchema">

<xsd:complexType name="T_mixed" mixed="true">
 <xsd:sequence>
 <xsd:element name="a" type="xsd:byte"/>
 <xsd:element name="b" type="xsd:string"/>
 </xsd:sequence>
 </xsd:complexType>

<xsd:complexType name="T" mixed="false">
 <xsd:sequence>
 <xsd:element name="a" type="xsd:byte"/>
 <xsd:element name="b" type="xsd:string"/>
 </xsd:sequence>
 </xsd:complexType>

<xsd:element name="E_mixed" type="T_mixed"/>

<xsd:element name="E" type="T"/>

</xsd:schema>
'
go

DECLARE @var XML(SC)
SET @var = CONVERT(XML(SC),
'<E>
<a>1</a>
Data b
</E>', 1)
SELECT @var
go

The query returns
<E><a>1</a>Data b</E>

DECLARE @var XML(SC)
SET @var = CONVERT(XML(SC),
'<E_mixed>
<a>1</a>
Data b
</E_mixed>', 1)
SELECT @var
go

This time the query returns
<E_mixed>
	<a>1</a>
	Data b
</E_mixed>

As you can see the non significant whitespaces are only preserved in the second case.

If we don't use the "1" option for the CONVERT option, whitespace only text nodes are discarded, even in the case of mixed content models as shown in the example below

DECLARE @var XML(SC)
SET @var = CONVERT(XML(SC),
'<E_mixed>
<a>1</a>
Data b
</E_mixed>')
SELECT @var
go

The query returns
<E_mixed><a>1</a>Data b</E_mixed>

Please be mindful that if in a same XML instance you have both mixed and unmixed content models, the behavior will vary.
Let's update our schema collection and run an example

alter xml schema collection SC add '<xsd:schema xmlns:xsd="http://www.w3.org/2001/XMLSchema">
<xsd:element name="root">
 <xsd:complexType>
 <xsd:choice minOccurs="0" maxOccurs="unbounded">
 <xsd:element ref="E"/>
 <xsd:element ref="E_mixed"/>
 </xsd:choice>
 </xsd:complexType>
</xsd:element>

</xsd:schema>'
go

Now let's run the following query

DECLARE @var XML(SC)
SET @var = CONVERT(XML(SC),
'<root>
<E>
 <a>1</a>
 non mixed
</E>
<E_mixed>
 <a>2</a>
 mixed
</E_mixed>
</root>', 1)
SELECT @var
go

The result is
<root><E><a>1</a>non mixed</E><E_mixed>
 	<a>2</a>
 	mixed
	</E_mixed></root>

As you can see whitespace-only text nodes were preserved under E_mixed but not under E.

Finally, I'll leave you with one small piece of advice: always be mindful of leading and trailing whitespaces. If you use CONVERT with the option that preserves whitespaces they will be seen as text nodes, which can be a problem.
For example

DECLARE @var XML(SC)
SET @var = CONVERT(XML(SC),
' <root/>', 1)
SELECT @var
go

will fail with the following error
XML Validation: Text node is not allowed at this location, the type was defined with element only content or with simple content. Location: /

The leading spaces were preserved and became a text node, which is not allowed at the top level for typed XML when at least one global element definition is available, hence the error.

Validating without typing

denisruc — Thu, 08 Sep 2005 03:01:00 GMT

You might encounter a situation where you want to validate incoming XML instances against the contents of an xml schema collection, but you still want to store your XML data as untyped.

We're going to look at a very simple way to do this. All you have to do is add a CHECK constraint to your table where your XML instance is converted into typed XML. Here is an example

CREATE XML SCHEMA COLLECTION SC_test AS '
<schema xmlns="http://www.w3.org/2001/XMLSchema">
<element name="e" type="byte"/>
</schema>'
go

CREATE TABLE T1 (xmlCol XML,
CHECK (datalength(CONVERT(XML(dbo.SC_test), xmlCol)) > -1))
go

Please note, I used datalength() to make up a dummy test that would always return true provided the call to CONVERT succeeds. I am not a T-SQL expert so if you know of a more elegant way to do this please let me know.

So given the schema collection SC_test and the table T1 defined above, we can try to insert various values into the table.

INSERT INTO T1 VALUES (null)
INSERT INTO T1 VALUES ('<e>1</e>')
INSERT INTO T1 VALUES ('')
go

All 3 INSERTs above will succeed.

The following however will fail because “a1” is not a valid byte value.
insert into T1 VALUES ('<e>a1</e>')

Another interesting thing to note is that you can also use the “document” facet which is normally unavailable for untyped XML (if you do not know about this feature please check this previous post)

CREATE TABLE T2 (xmlCol XML,
CHECK (datalength(CONVERT(XML(DOCUMENT dbo.SC_test), xmlCol)) > -1))
go

Now let’s try to insert into T2 the same values we tried to insert into T1.

The following 2 statements will succeed

INSERT INTO T2 VALUES (null)
INSERT INTO T2 VALUES ('<e>1</e>')
go

This time, the following statement will fail because the value is not a valid XML document

INSERT INTO T2 VALUES ('')
go

And finally the statement below will fail once again because of the invalid byte value

INSERT INTO T2 VALUES ('<e>a1</e>')
go

This way, you can make sure that all XML instances that aren't valid XML documents will be rejected, even though your column was declared as plain vanilla untyped XML.

Document and content

denisruc — Fri, 12 Aug 2005 21:41:00 GMT

Document and content are two options you can use in SQL Server 2005 when typing an XML column or variable with an XML schema collection.
To put it simply document means only XML documents should be accepted (exactly one top level element) while content means fragments should be accepted (0 or more top level elements). By default, if none of the 2 options is specified it is content that applies. The table below shows the syntax and the consequences for all 3 different possibilities

XML(myCollection)	Fragments (CONTENT applies by default)
XML(DOCUMENT myCollection)	Documents only
XML(CONTENT myCollection)	Fragments

This feature is in itself pretty easy to understand, and I wouldn’t have featured it on my blog if it didn’t have some interesting effects on… you guessed it… static typing.

First, we’re going to create an xml schema collection that contains one very simple schema.

CREATE XML SCHEMA COLLECTION test1 AS '
<schema xmlns="http://www.w3.org/2001/XMLSchema">
<element name="e" type="string"/>
</schema>
'
go

Now let’s see what the static type of expression /node() is when we use CONTENT and DOCUMENT. (If you don’t know how to obtain the static type of an expression please refer to this post by Mike Rorke)

DECLARE @var XML(CONTENT test1)
SET @var = '<e>data</e>'
SELECT @var.query('/node()')
go

static type is (element(e,xs:string) ? & comment ? & processing-instruction ?) *

DECLARE @var XML(DOCUMENT test1)
SET @var = '<e>data</e>'
SELECT @var.query('/node()')
go

static type is (comment ? & processing-instruction ?) * , element(e,xs:string) , (comment ? & processing-instruction ?) *

As you can see the results are very different. The main thing you should note is that in the second case, when DOCUMENT is used, the server expects one and only one instance of element ‘e’. It can be preceded and followed by zero or more intertwined comments and processing instructions, but there has to be one instance of element ‘e’, no more, no less. In the case when CONTENT is used there can be zero or more instances of ‘e’ intertwined with zero or more comments and processing instructions.

Now let’s try to run a simple DML query.

DECLARE @var XML(CONTENT test1)
SET @var = '<e>data</e>'
SET @var.modify('replace value of /* with "new data"')
go

DECLARE @var XML(DOCUMENT test1)
SET @var = '<e>data</e>'
SET @var.modify('replace value of /* with "new data"')
go

The results are once again very different.
The first query (where CONTENT is used) fails with the following error message
XQuery [modify()]: The target of 'replace' must be at most one node, found 'element(e,xs:string) *'
The second query (where DOCUMENT is used) on the other hand succeeds.

If you’ve read my post about singleton issues you’ve probably already guessed what the problem is. Success or failure in this case depends on the static type of XPath expression /*. When CONTENT is used, that static type is element(e,xs:string)*, which means that the expression could possibly return more than one node. When DOCUMENT is used, the static type is element(e,xs:string), which guarantees that the expression will return only one node.

Now, let’s modify our schema collection by adding a second element.

ALTER XML SCHEMA COLLECTION test1 ADD '
<schema xmlns="http://www.w3.org/2001/XMLSchema">
<element name="d" type="string"/>
</schema>
'
go

Let’s try to run our DML query again (still using DOCUMENT).

DECLARE @var XML(DOCUMENT test1)
SET @var = '<e>data</e>'
SET @var.modify('replace value of /* with "new data"')
select @var
go

This time the query fails and the server returns the following error message

XQuery [modify()]: The target of 'replace value of' cannot be a union type, found 'element(e,xs:string) | element(d,xs:string)'.

There are now two different elements that are eligible to be at the top level in the document. Therefore the addition of global element ‘d’ to our schema caused the static type of expression /* to change to element(e,xs:string) | element(d,xs:string)

To conclude, let’s look at what the static type of /node() has become in both cases now that the schema contains two global element declarations.

DECLARE @var XML(CONTENT test1)
SET @var = '<e>data</e>'
SELECT @var.query('/node()')
go

static type is (element(e,xs:string) ? & element(d,xs:string) ? & comment ? & processing-instruction ?) *

DECLARE @var XML(DOCUMENT test1)
SET @var = '<e>data</e>'
SELECT @var.query('/node()')
go

static type is (comment ? & processing-instruction ?) * , (element(e,xs:string) | element(d,xs:string)) , (comment ? & processing-instruction ?) *

When CONTENT is used, you can have as many instances of the two elements as you want. When DOCUMENT is used, you have to have exactly one instance of ‘e’ or exactly one instance of ‘d’, but you cannot have both. The static typing code takes into account which of the two options DOCUMENT or CONTENT is applied to the typed XML column or variable you're using.

Using the XPath provided in XML validation error messages

denisruc — Wed, 27 Jul 2005 23:31:00 GMT

Yesterday’s post regarding the addition of a location in validation error messages brought back a conversation I’d had with our Program Manager regarding the use of that XPath to automatically retrieve invalid XML fragments when validation fails

I’ve decided to try it and since it works, I’m going to share it with you.

The idea behind this is very simple. Through a TRY-CATCH construct, it is possible to retrieve the error message when validation fails. Once you have this error message, you can do some string manipulation on it to extract the XPath. Your XML instance might be invalid with respect to your schema but as long as it is valid XML you can store it in an untyped XML variable. Then you can use the XPath to build a query that is run against your instance to retrieve the faulty XML fragment. In this case, I’ve decided to retrieve the parent of the faulty node pointed to by the XPath.

First I am going to create a stored procedure that, given an error message @err and an XML instance @x will return the parent of the node the XPath points to

CREATE PROCEDURE sp_fragment @x XML, @err nvarchar(MAX)
AS
BEGIN

-- first find where the location starts in the error message
DECLARE @index int
SET @index = CHARINDEX('Location: ',@err)
-- if no location we cannot do anything
IF (@index <= 0)
GOTO NoLoc

-- move to the beginning of the XPath expression
SET @index = @index + 10
DECLARE @Xpath nvarchar(MAX)
-- capture the XPath and use the parent axis
SET @Xpath = SUBSTRING(@err, @index , LEN(@err) - @index + 1) + '/..'

-- build the rest of the query
DECLARE @query nvarchar(MAX)
SET @query = 'SET QUOTED_IDENTIFIER ON; DECLARE @var XML;'
SET @query = @query + ' SET @var = ''' + CONVERT(nvarchar(MAX), @x) + ''';'
SET @query = @query + 'SELECT @var.query('''+@Xpath + ''')'

-- print the error message for the benefit of the user
PRINT @err
-- Run the query to retrieve the location
EXEC(@query)
GOTO Done

NoLoc:
PRINT @err
PRINT 'Error Message does not specify an XPath location'

Done:
END
go

As you can see, this is fairly simple. Now, let’s see how we can use this stored procedure in conjunction with validation.

First we’re going to need a schema collection and a table with a typed XML column.

create xml schema collection SC_test AS '
<schema xmlns="http://www.w3.org/2001/XMLSchema"
xmlns:ns="myNS"
targetNamespace="myNS">

</schema>
'
go

create table T(xmlCol XML(SC_test))
go

Then we’re going to build a stored procedure that, given an XML instance @var will insert it into table T

CREATE PROCEDURE insert_into_T @var XML
AS
BEGIN

DECLARE @err nvarchar(MAX)

BEGIN TRY
  INSERT INTO T VALUES (@var);
END TRY
BEGIN CATCH
  SET @err = ERROR_MESSAGE()
  EXEC sp_fragment @var, @err;
END CATCH;

END
go

As you can see, I am making use of the TRY-CATCH construct. If an error occurs as we insert the value into T we retrieve the error message and pass it, as well as the XML instance, to the sp_fragment procedure.

Now let’s use procedure insert_into_T with an XML instance that is invalid with respect the the SC_test collection and see what happens.

EXEC insert_into_T '<x:root xmlns:x="myNS" order="1"><a/><a/><a/></x:root><x:root xmlns:x="myNS" order="2"><a/><a/><a/>12</x:root><x:root xmlns:x="myNS" order="3"><a/><a/><a/><a/><a/><a/><a/><a/>0</x:root><x:root xmlns:x="myNS" order="4"><a/><a/><a/></x:root><x:root xmlns:x="myNS" order="5"><a/><a/><a/></x:root><x:root xmlns:x="myNS" order="6"><a/><a/><a/></x:root>'
go

We get the following output (I simplified the formatting)

XML Validation: Invalid content. Expected element(s):b where element 'a' was specified. Location: /*:root[3]/*:a[7]

<x:root xmlns:x="myNS" order="3"><a/><a/><a/><a/><a/><a/><a/><a/>0</x:root>

Now I can use any clue I want (such as the value of the ‘order’ attribute on the ‘root’ element) to quickly find the problem spot in my instance.

I hope this quick example inspires you to use the XPath we’ve added to the validation error messages. Don’t hesitate to drop me a comment or an email if you have interesting solutions of your own that you are willing and able to share.

Substitution groups and their effects on type derivation

denisruc — Tue, 12 Jul 2005 03:54:00 GMT

If you’ve been playing with Xml schema collections in SQL Server 2005, you might have noticed that it is forbidden to expand substitution groups membership in ALTER XML SCHEMA COLLECTION statements.

What this mean practically is that already-persisted elements should not be used as substitution group heads when altering a collection

Fro example, this is valid

CREATE XML SCHEMA COLLECTION SC1 AS '
<schema xmlns="http://www.w3.org/2001/XMLSchema"
targetNamespace="myNS"
xmlns:ns="myNS">

</schema>
'
go

But the example below is invalid

CREATE XML SCHEMA COLLECTION SC2 AS '
<schema xmlns="http://www.w3.org/2001/XMLSchema"
targetNamespace="myNS"
xmlns:ns="myNS">

</schema>
'
go

ALTER XML SCHEMA COLLECTION SC2 ADD '
<schema xmlns="http://www.w3.org/2001/XMLSchema"
targetNamespace="myNS"
xmlns:ns="myNS">

</schema>
'
go

The last batch will fail with the following error message: “Invalid definition for element 'myNS:member1'. SQL Server does not currently permit additions to existing substitution groups via ALTER XML SCHEMA COLLECTION.”.
The reason for this is that adding members to a substitutionGroup might render previously defined type derivations invalid. Let’s look at a simple example.

create xml schema collection mySC AS '
<schema xmlns="http://www.w3.org/2001/XMLSchema" targetNamespace="http://ns" xmlns:ns="http://ns">

<element name="E" type="long"/>

</schema>'
go

This statement will fail with the following error: “Invalid restriction for type 'http://ns:myCt2'. Invalid model group restriction.”
It is the declaration of element E2 that causes the restriction from myCt1 to myCt2 to be invalid (because E2 could be substituted for E1 in any element of type myCt2)

Remove the element definition for E2 and the schema becomes valid, like this

create xml schema collection mySC AS '
<schema xmlns="http://www.w3.org/2001/XMLSchema" targetNamespace="http://ns" xmlns:ns="http://ns">

<element name="E" type="long"/>

</schema>'
go

The problem is, now we could very well try to add E2 to the schema collection, like this

alter xml schema collection mySC ADD '
<schema xmlns="http://www.w3.org/2001/XMLSchema" targetNamespace="http://ns" xmlns:ns="http://ns">

</schema>'
go

The statement above would render the derivation by restriction from myCt1 to myCt2 illegal, even though the definitions of the types themselves remain the same.
This is in short why we prohibit the use of already-persisted elements as substitution groups heads.

Welcome

denisruc — Mon, 13 Jun 2005 18:27:00 GMT

Welcome to my newly created blog. My name is Denis Ruckebusch. I've been part of the test organization at Microsoft for the past 6 years. I spent the last 4 working on the new XML datatype feature in SQL Server 2005.

My main area of focus is typed XML. I mostly deal with XML schemas, validation of XML instances against thoses schemas, and running XQueries over typed XML. Since there are already a number of Microsoft bloggers dealing with the new XML functionality I will try to focus on issues specific to schemas and typed XML.

If you are currently using SQL Server 2005 and already have comments or questions about the use of typed XML, don't hesitate to drop me a line!