Changing of the Guard: AXIS out, JAXWS in

DotNetInterop — Wed, 30 Jul 2008 19:03:00 GMT

I have come to a conclusion: I no longer want to deal with the hassle that has become Apache AXIS.

Back in the day, when interconnecting Java to .NET was still novel, I started using Apache SOAP to show how interop between .NET and JAva was possible. I was looking for something simple and easy, something that would allow interconnection without a ton of pain. Apache SOAP was that thing. But Apache SOAP was brittle. It had scads of problems with xsi:type and SOAP Section 4 encoding. I needed something more flexible, something easier.

Then came Glue from The Mind Electric. Glue was the easier, more flexible web services stack. It was simple. It worked. It was actively maintained. Graham Glass came regularly to Redmond to make sure the interop with .NET was good.

Then TME got bought by WebMethods, and Glue disappeared. I needed to pick a new Java stack. When AXIS came out, I switched, looking for better tools, and something that would be reliable, maintained longer term.

At that time, I looked at Sun's web services stuff, but it was always bundled with a Web server or Open Office or a Sun Ray or something else from Sun I didn't want. It was like Scott McNealy's proverbial furball. I just wanted a working web services stack, I didn't want to buy a Solaris Server. I didn't want a furball. So I went with AXIS. I could use it on Tomcat, on JBoss, or on Jetty. I could use it client or server side. I wrote a few pages on the AXIS wiki, specifically around how to get it to work with .NET. I sat on the AXIS users mail list, responded to interop questions.

AXIS (Version 1) worked great for a long time, still does in fact. It has connected to many different endpoints for me - Excel Services, WCF endpoints, .NET, SQL Server, and many other pieces. It was neat and clean; the tool options made sense. The code it generated was pretty clean.

But AXIS1 is no longer being maintained. Any new standards (like MTOM - heh, new since 2005) will not be supported in AXIS1. Also, it didn't work so well when it came to connecting to Exchange. AXIS2 likewise. And that allowed me to step back and re-evaluate. AXIS2 is current, but has become very large, the doc is poor, the support is invisible, the generated code smells, and the seams are everywhere. Fifty nine jar files? Really? Do I need this hassle? With JAX-WS, do I need AXIS any longer? I think not.

I'm switching to JAX-WS.

This is kinda backwards, because my goal is to show customers that they can in fact connect disparate systems together. Given that goal, my own interests should rank near the bottom in the list of criteria for selecting a web services library. It should be - what are customers using? What do customers want to see? But I honestly cannot believe customers will continue to put up with the furball that AXIS2 has become.

And if I am in a position to make a recommendation, I will recommend JAX-WS.

It works. There is a single runtime jar file. The tools are better. The interop is good. It's fast. I'm happy.

I'm changing my decision but I haven't changed my criteria. I still want something that just works, simply. Basically I want the closest philosophoical equivalent of .NET, but in Java. AXIS used to be that. Now, JAX-WS fits that bill.

I still love .NET and WCF. But if I need to connect to .NET from Java, I will use JAX-WS.

Connecting to Exchange using JAX-WS, part 1

DotNetInterop — Tue, 29 Jul 2008 21:52:00 GMT

Yes, it is possible to Connect to Exchange using JAX-WS

The fear and trepidation I had when considering the effort to connect a Java application to Exchange Server 2007 was well founded. It wasn't easy. I'm going to spoil the ending and tell you, yes, it is possible to connect Java apps to Exchange Server via Web services, and you can do it securely and with good performance and reliability. But it took some starts and stops to make it happen.

Warning: Old Guy Reminiscing Ahead

Back a few years, I was a technical marketing guy for the Microsoft platform, and my main role Was talking to customers. One of the things I talked with customers about often, was interop. Pretty much every customer I talked to already had a signficant investment in information technology, they had a strategy that was in place and working, they had systems in place, working. They had off-the-shelf apps, they had some custom-built apps, and they had skillz. They were looking to make adjustments, to evolve their systems, to improve. It reminded me of the principles of Agile Development.

Do you remember XP? Do you remember how trendy that was, like 3 or 4 years ago. Everyone was talking about XP, everyone was looking to apply XP principles into their approaches. That seems a long time ago. XP is so 2004.

The approach that companies took with their information systems architecture was something like what developers did in a different domain with XP. Lots of little iterations, constant improvement, and steadily you get closer to your goal.

I remember talking to customers about web services technology, about using standard XML-based protocols to enable better connections between disparate information systems. Yes, Virginia, this was actually a theme of my conversations back only a few years ago. Now it seems so obvious. Now that everything speaks XML and interconnection is fairly easy, the wisdom of using standard protocols to gain efficiencies in IT is apparent. But only a short time ago this issue was not so clearly understood.

To make my case that web services and XML based protocols were the future, I'd show customers the investments that Microsoft was making into them. I'd tell them about all the standards was Microsoft working on. But the thing that customers wanted was hard evidence - not standards or papers. They wanted proof of XML usage in Microsoft products. So I'd talk about all the XML-based protocols that were already in the existing Microsoft systems they had: the web services libraries in the .NET Framework, the tooling in Visual Studio (but these were fairly obvious). The XML shredding capability that was introduced in SQL 2000 and improved in SQL 2005. Actually with the FOR XML clause you can do some pretty nifty things with the data that is already in your database. The MSXML component - now sort of just accepted as part of the landscape, a key, if almost forgotten enabler of AJAX - but 6 or 7 years ago it was undergoing rapid development.

These are all pretty good examples of Microsoft's investment in XML and WS-*. But one of the things that really surprised people was the web services capabilities we were introducing into Office (with WordML and ExcelML and so on), and the Exchange Server.

Exchange Server 2007 and Web services

Exchange Server 2003 had some limited web services capabilities. With Exchange Server 2007, Microsoft delivered an email and collab server that exposed all of its features through standard web services interfaces. Today if you have an Exchange Server, you can point your browser to http://my.exchange.server.com/ews/Exchange.asmx and get a WSDL for the server. Wow. That is pretty cool.

Standards are standards, and complying to a standard doesn't guarantee interop. If you are designing a web services interface, you can get better interop by keeping your WSDL fairly simple, and constraining your interface definitions to avoid some of the more esoteric nooks and crannies of the XML Schema spec. This is advice we've all been following for years. It works.

The problem with that advice is that it does not work if the information systems at the end of the wire are not simple, or if the requirements are not simple. For example, if you want WS-Addressing or delegated security, things can get complicated. And, similarly, Exchange Server is not simple. It stores lots of different types of things (email messages, contacts, tasks, notes, calendar entries, requests, resent items, recurrences, mime content type, cancellations, suggestions for new times, and so on). And then it compounds that complexity by adding in an unstructured aspect - document attachments and other unstructured data. Then there is stuff like impersonation, delegation (your admin assistant can update your calendar), out-of-office messages, and so on. It's pretty rich.

With all that capability, necessarily, the interface to Exchange is not going to be simple. Look at the user interface in Outlook - it is pretty rich. And the same is true with the network interface on the wire, described in the WSDL.

As a result, the WSDL for Exchange is complicated. It's big. It's got lots of nested structures. There are unions and restrictions and headers and bodies. It breaks all the rules for designing schema with interop in mind.

Anyway, who needs web services, right? We can use WebDAV!

Interop and the Exchange Server web services interface

First decision: which web services library to use? My first thought was to use the tried-and-true Apache AXIS 1.4 library that I had been depending on since 2006. It wasn't THAT old, and it worked for all the other interop tasks I had tried it on. It connected to .NET systems using ASMX or WCF. It was simple. I knew the tools.

My first step was to grab the Exchange.wsdl, the types.xsd and the message.xsd. I knew that I needed to add a service definition to the end of the Exchange.wsdl file, because most WSDL compilers want it:

   <wsdl:service name="ExchangeServices">
     <wsdl:port name="ExchangeServicePort" binding="tns:ExchangeServiceBinding">
       <soap:address location="https://myexchange.company.com/EWS/Exchange.asmx"/>
     </wsdl:port>
   </wsdl:service>

</wsdl:definitions>

But very early the work with AXIS 1.4 was hitting roadblocks. That library generated source code that did not compile. It choked on quite a few things. Other people have walked this path but custom modifications to the generated code did not sound like fun to me. I quickly decided that I'd need to move to something else.

My next idea was to move to AXIS2, which, coincidentally, is also at version 1.4. Now why they chose to call it "AXIS2 1.4" instead of "AXIS 2.4" is beyond me. There is a significant breakage in back-compatability with the move to AXIS to AXIS2, but still the version numbers are very confusing.

AXIS2 introduces a number of changes over AXIS (1). The AXIS guys improved the XML schema support, for one. Also, they made the XML serialization engine pluggable, so you can use the built-in simple library (called ADB), or you can switch to use JiBX or XMLBeans.

As with AXIS1, I ran the Exchange WSDL through the AXIS2 tools to generate the client-side proxies. This time, the code actually compiled. I was pleased to see that there was just one giant Java file generated for the entire Exchange interface. In AXIS 1.4, there was a whole series of files, 447 of em. Too many, thought I. But AXIS2 v1.4 generated just one file. Cool!

And it was a big one. Opening the file in emacs - emacs asked me, "are you sure you want to open this file? it's really big!" Seriously. And compiling that single file with javac, I actually got a heap overflow. Wow! That's a big file. I just used the -X heap magic incantations on that file and I was able to compile it.

I wrote up a simple proof of concept Java class that used the generated AXIS stub classes to connect to Exchange Server. When I ran the code, I got a failure: "required attribute localTraversal is null" I had no idea where that was leading me. I searched in the WSDL and schema for Exchange but couldn't find any elements or attributes by that name. I considered ripping open the AXIS2 source code to try to troubleshoot it. Wasn't real excited about that prospect. I had no idea where that was leading me.

My next step was to try the XMLbeans serialization option in AXIS 2. With this, I was back to generating ~ 400 Java files. But I didn't care, as long as the interop worked. And I had to update my client application code, because switching from ADB to XMLBeans necessitates a change in the programming model. I didn't care, though. Interop! (The things I do for interop!)

The AXIS2 + XMLbeans combo was looking promising, but also failed, this time with

java.lang.ClassCastException: org.apache.xmlbeans.impl.values.XmlComplexContentImpl cannot be cast to com.microsoft.schemas.exchange.services._2006.types.Folder
IdType

This is apparently a common error, if there is a duplication in classes. I don't think I had that. I keep a clean workspace. Some of the XMLBeans people suggested on forums to upgrade to the later version of XMLBeans. There was no better suggestion I found. The AXIS2 library bundles XMLBeans 2.3.0, but XMLBeans standalone was at v2.4.0. So I downloaded 2.4.0, and tried to manually compile the schema with the scomp tool.

c:\apache\xmlbeans-2.3.0\bin\scomp -out GeneratedTypes.jar contract\ExchangeServices.wsdl  -compiler c:\sunjdk\bin\javac.exe

But, then, another runtime exception:

Exception in thread "main" java.lang.NoClassDefFoundError: com/microsoft/schemas/exchange/services/_2006/types/FolderIdType
        at ExchangeClient.Run(ExchangeClient.java:161)
        at ExchangeClient.main(ExchangeClient.java:217)
Caused by: java.lang.ClassNotFoundException: com.microsoft.schemas.exchange.services._2006.types.FolderIdType
        at java.net.URLClassLoader$1.run(Unknown Source)

It was one problem after another. Also, I had found the documentation and examples to be completely lacking for AXIS2. I couldn't find anyplace that gave me any hints. Why was it so hard to find information on this thing? And why did I have to use Class.Factory.Create(); instead of just calling new Class(). How dumb is this? And finally, there were 59 JAR files I needed to reference for AXIS2. That's what the doc told me. Commons-logging, xmlbeans, axis-api, axis-rt, corba, neethi, axiom, anogen, and on and on. I don't even know what those things are. The whole thing felt very... messy. I've already experienced JAR Hell and I don't want to go back there. I'm sure each one of those jars is required, but AXIS2 *could* be a little more seamless. As it is, it feels like a science project.

Ok, time to switch horses. The AXIS Stuff was not giving me joy. My next attempt would be to use JAXWS.

With JAXWS, there is one runtime JAR. One. The code-gen produces... a JAR. How refreshing! I don't need 400 class files. I can just reference a JAR. (Yes, I know there are 400 classes inside the jar. I don't care. It's a single jar). Hmm, someone ought to tell the AXIS project about this upstart JAXWS. The stub classes get generated and compile nicely. (small victory, cross fingers and hope for more). I am starting to like JAXWS.

The programming model exposed by JAXWS-generated client-side classes is a little different than what I was used to. Rather than using return values from a stub method for the response, JAXWS has this idea of using the javax.xml.ws.Holder class for return values. Kinda odd, but... whatever. Maybe there is a way to change the client-side programming model, but I wasn't going to worry about that right then.

There is some stuff in the Exchange interface (WSDL) having to do with impersonation and SerializedSecurityContext in each request on the EWS Service port, which is not useful. I'm using neither of those things, yet they are included in every Web services stub method. Passing null for these parameters does not satisfy, I get an error saying that I cannot set that to xsi:nil unless the schema has nillable='true'. I tried fiddling with the WSDL.. to set it to nillable in the right spots... That didn't work, so I just removed that stuff from the WSDL completely, from every wsdl:message in the WSDL file:

        <wsdl:message name="GetFolderSoapIn">
                <wsdl:part name="request" element="tns:GetFolder" />
<!-- dinoch - change for JAXWS
                <wsdl:part name="Impersonation" element="t:ExchangeImpersonation"/>
                <wsdl:part name="S2SAuth" element="t:SerializedSecurityContext"/>
                <wsdl:part name="MailboxCulture" element="t:MailboxCulture"/>
-->
                <wsdl:part name="RequestVersion" element="t:RequestServerVersion"/>
        </wsdl:message>

and also from every soap:operation:

                <wsdl:operation name="GetItem">
                        <soap:operation soapAction="http://schemas.microsoft.com/exchange/services/2006/messages/GetItem" />
                        <wsdl:input>
<!--
                                <soap:header message="tns:GetItemSoapIn" part="Impersonation" use="literal"/>
                                <soap:header message="tns:GetItemSoapIn" part="S2SAuth" use="literal"/>
                                <soap:header message="tns:GetItemSoapIn" part="MailboxCulture" use="literal"/>
-->
                                <soap:header message="tns:GetItemSoapIn" part="RequestVersion" use="literal"/>
                                <soap:body parts="request" use="literal" />
                        </wsdl:input>
                        <wsdl:output>
                                <soap:body parts="GetItemResult" use="literal" />
                                <soap:header message="tns:GetItemSoapOut" part="ServerVersion" use="literal"/>
                        </wsdl:output>
                </wsdl:operation>
                <wsdl:operation name="CreateItem">

Ok, I was getting somewhere. With this change I was able to send a FindItem message over the wire to Exchange, and get message IDs back. Very cool. The approach I was taking was to use findItem to get the item IDs, and then use GetItem on the Ids to get the full content. The FindItem() was working flawlessly, but the GetItem() was not. I was not getting the results I expected. No exceptions, but my data was not coming back.

To troubleshoot this, I used Fiddler2 to see the HTTP messages back and forth between Java and Exchange. Because the communication was using SSL for encryption, Fiddler2 does not work. Some people have used Charles, a similar web debugging proxy. But I already had Fiddler, so... With the fiddler tool I could see exactly why the GetItem message was not giving me joy. (By the way, if you use Fiddler to decrypt SSL traffic, it's necessary to specify to the Java app that it should accept any SSL traffic - basically disable cert validation in the application. Only do this for testing purposes!!)

Correct flow from a .NET app to Exchange:

<soap:Envelope xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/">
  <soap:Body>
    <GetItem xmlns="http://schemas.microsoft.com/exchange/services/2006/messages">
      <ItemShape>
        <BaseShape xmlns="http://schemas.microsoft.com/exchange/services/2006/types">Default</BaseShape>
        <AdditionalProperties xmlns="http://schemas.microsoft.com/exchange/services/2006/types">
          <FieldURI FieldURI="item:Subject" />
          <FieldURI FieldURI="item:Body" />
          <FieldURI FieldURI="item:Categories" />
          <ExtendedFieldURI PropertySetId="0006200E-0000-0000-C000-000000000046" PropertyId="35584" PropertyType="Integer" />
          <ExtendedFieldURI PropertyTag="0x10F3" PropertyType="String" />
          <ExtendedFieldURI PropertyTag="0x3008" PropertyType="SystemTime" />
        </AdditionalProperties>
      </ItemShape>
      <ItemIds>
        <ItemId Id="AAAZAERpbm8uQ2hpZXNhQG1pY3Jvc29mdC5jb20ARgAAAAAALBHixAbt0hGt6gCAX5/eGgcAd75VstDi0RGt3QCAX5/eGgAAAnopngAAEKx2OMNuVUuHEMvofVOxpwANFT4iewAA" ChangeKey="AAAAABQAAAAdMskaJAAzTpK1s6mN4vCnAAhyCw==" xmlns="http://schemas.microsoft.com/exchange/services/2006/types" />

Broken flow from a JAXWS app to Exchange:

<S:Envelope xmlns:S="http://schemas.xmlsoap.org/soap/envelope/">
  <S:Header>
    <ns2:RequestServerVersion xmlns="http://schemas.microsoft.com/exchange/services/2006/messages" xmlns:ns2="http://schemas.microsoft.com/exchange/services/2006/types" Version="Exchange2007_SP1" />
  </S:Header>
  <S:Body>
    <GetItem xmlns="http://schemas.microsoft.com/exchange/services/2006/messages" xmlns:ns2="http://schemas.microsoft.com/exchange/services/2006/types">
      <ItemShape>
        <ns2:BaseShape>Default</ns2:BaseShape>
        <ns2:AdditionalProperties>
          <ns2:Path xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:type="ns2:PathToUnindexedFieldType" FieldURI="item:Subject" />
          <ns2:Path xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:type="ns2:PathToUnindexedFieldType" FieldURI="item:Body" />
          <ns2:Path xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:type="ns2:PathToUnindexedFieldType" FieldURI="item:Categories" />
        </ns2:AdditionalProperties>
      </ItemShape>
      <ItemIds>
        <ns2:ItemId ChangeKey="AAAAABQAAAAdMskaJAAzTpK1s6mN4vCnAAhyCw==" Id="AAMkAGUwNmJhYWVjLTI1ZTgtMTFkMy05MDc5LTAwODA1ZjMxZjgyNgBGAAAAAAAsEeLEBu3SEa3qAIBfn94aBwB3vlWy0OLREa3dAIBfn94aAAACeimeAAAQrHY4w25VS4cQy+h9U7GnAA0VPiJ7AAA=" />

Exchange wasn't sending me the data I expected, because the JAXWS request message was not what Exchange expected it to be. With this understanding, I looked into the contract. For the element in question, the schema used a ref="type" structure, and this was the only use of the ref in the entire contract. Clearly causing a problem; JAXWS was formatting it strangely. I modified it to eliminate the ref, just as a test.

  <!-- Arrays of paths -->
  <xs:complexType name="NonEmptyArrayOfPathsToElementType">
    <!-- using choice here so that proxy generator will not flatten the array and thus lose the
                item element names -->
    <xs:choice maxOccurs ="unbounded">
<!-- dinoch - change for jaxws -->
  <xs:element name="Path" type="t:BasePathToElementType"/>
  <xs:element name="FieldURI" type="t:PathToUnindexedFieldType" />
  <xs:element name="IndexedFieldURI" type="t:PathToIndexedFieldType" />
  <xs:element name="ExtendedFieldURI" type="t:PathToExtendedFieldType"/>

<!--
      <xs:element ref="t:Path"/>
-->

    </xs:choice>
  </xs:complexType>

After re-generating the client-side stubs, I was able to get the GetItem message to work nicely. Here's the message from JAXWS after the change in schema:

<S:Envelope xmlns:S="http://schemas.xmlsoap.org/soap/envelope/">
  <S:Header>
    <ns2:RequestServerVersion xmlns="http://schemas.microsoft.com/exchange/services/2006/messages" xmlns:ns2="http://schemas.microsoft.com/exchange/services/2006/types" Version="Exchange2007_SP1" />
  </S:Header>
  <S:Body>
    <GetItem xmlns="http://schemas.microsoft.com/exchange/services/2006/messages" xmlns:ns2="http://schemas.microsoft.com/exchange/services/2006/types">
      <ItemShape>
        <ns2:BaseShape>Default</ns2:BaseShape>
        <ns2:AdditionalProperties>
          <ns2:FieldURI FieldURI="item:Subject" />
          <ns2:FieldURI FieldURI="item:Body" />
          <ns2:FieldURI FieldURI="item:Categories" />
        </ns2:AdditionalProperties>
      </ItemShape>
      <ItemIds>

After that, it was a matter of writing up the code to use the jaxws-generated stubs. It all works nicely now. It's pretty darn fast, too!

Let me summarize what I did to get Java to talk to Exchange:

Modified the exchange.wsdl to include a service definition, and to remove all occurrences of impersonation, S2SAuth, and MailboxCulture from each wsdl:message.
Modified the types.xsd to eliminate the ref="t:Path", and replace it with actual elements
Use Fiddler2 to trace and debug the SOAP messages.

The bottom line is that web services interop is possible even with complicated schema and interfaces, if you are willing to get your hands dirty on the schema, and if you choose the right tools.

In part 2 of this story, I'll show you some of the application code that uses the JAXWS-generated client classes, and I'll pack up the source code for everything too.

Great resources: