Well, June is here so it must be time to make a post.  Let's talk about...  Hmmm... What should we talk about?  Hmmm....  How about WinFS!

You know WinFS... It's one of the three main pillars of Longhorn Client (Avalon, Indigo, and WinFS).  Or maybe you don't know much about WinFS.  I haven't seen any WinFS bloggers on this site yet and maybe you missed PDC.  So perhaps a short WinFS tutorial is in order. 

Consider the world of today.  The files on our hard drives are basically binary blobs.  We open the file with a Win32 FileOpen() command and then index into the byte stream.  That works just fine because your code knows that the jpg image starts just past the header at byte 186.  That's pretty much all you get from the file system: A stream of bytes.  Sad really, when you think about it.

It’s unreasonable to think that people could really write meaningful, rich applications with only this “byte indexing” scheme that the file system provides.  So application developers have augmented this file system shortcoming for years by providing “APIs” to their file formats.  Basically, the application developer creates a DLL which has some method called “GetImage()” for jpgs.  Then an ISV who wants to get at the image in the jpg file just calls the “GetImage()” method in the DLL, and the DLL encapsulates the fact that you need to index 186 bytes into the file to find the image.  

Well, that's certainly an improvement over a stream of bytes.  But it is an imperfect solution at best.  Each ISV builds these “File Access APIs“ in a different way.  If your application wants to read Adobe Acrobat files, then you have to hand-code knowledge of the Adobe Acrobat file access API into your code.  But what if you wanted to build an application which worked with many different file types in a rich way?  For example, what if you wanted to build a shell application something like Explorer.exe that could do rich queries across the 100,000 heterogeneous files you've got stored on your new 250 gig hard drive?

To accomplish this, you would need a standard mechanism for discovering the content in your files.  What would that look like?  Well, let’s start by thinking about how it might work in the world of SQL.  Say I've got an ERP database with 2,000 different tables.  I have tables for Customers, Vendors, Orders, Invoices, etc... 

I could fairly easily build a general purpose application which allowed me to browse over any of the tables in a generic, yet rich way.  My application could discover the list of tables with an sp_ command.  (And to be fair, the file system would let me get a list of files by traversing the directory structure).  But my SQL application could go way beyond the file system application.  My SQL application could call another sp_ command and get the column list for each table.  That's where the file system gets left in the dust.  The file system sees the file as a big binary blob, while SQL sees its relational data as structured.   My SQL browser could then fetch the data from each column, along with the datatype information and display it in a generic, yet fairly rich way.

So the original WinFS vision (long, long ago) had its original goal to “move the Windows file system into SQL“.   But that (perhaps simplistic) approach didn't last long.  Sure, we could have moved the file system to SQL.  But a fundamental change to the OS file system is a big, big, traumatic deal.  We can only afford to make this kind of change perhaps once a decade.  Do we really want to inflict this pain, only to have applications using “SELECT Blah FROM filesystem“ SQL commands to directly access file data for the next decade?  Is there something better?

If we only cared about browsing file system data (read only), then “SELECT blah“ with a result set of primitive data returned might have been OK.  But this is our big chance; shouldn't we maximize our opportunity?  Raw data is Dumb data...  In the next decade, people want to act on their data.  They want their data to have behavior.  When I find a jpg picture on my hard drive, I want to edit it.  I want to email it.  All these things require logic.  And that's something “SELECT blah“ can't easily provide.

Hmmm...  data + behavior.  Maybe this CLR \ Managed Code thing that Microsoft is working on might apply?

So there you have WinFS...  WinFS is about transforming the file system from a “dumb as a rock“ byte stream into an active layer of persistent objects.  These persistent objects should be queryable in a rich fashion.  I should be able to easily go to a metadata catalog and get a list of all persistent object types.  I should be able to update these objects and be assured that any logic to validate the data is present in the persistent object.  And I should be able to update a group of related persistent objects in a transactional manner.

WOW.... that’s really cool!  Persistent objects with behavior.  What a concept!  Oh... wait a minute.  That sounds like MBF.  MBF is an environment which allows you to create a set of rich business objects (with behavior) which can be persisted to a database.

Hmmm...  Is there more to this story?  How do MBF and WinFS relate?  Inquiring minds want to know!

I guess you'll just have to wait for my next post.  (Don't you just love cliffhangers)!

Two weeks without a post... sorry about that!  I've been traveling all over the place collaborating with people on the architecture for MBF process \ workflow.  I'm in a hurry so I may as well post about my recent work in the process world. 

We believe that a strong process architecture is critical to building a successful SOA application.  SOA is about decomposing monolithic applications into finer grained services.  These services interact with each other asynchronously, across potentially long durations.  Something has to coordinate the work between services.  That “something” is a process engine.

If you look at Microsoft's shipping products, the closest thing to a “process engine” is BizTalk. BizTalk is being used today to successfully coordinate XML messages between autonomous applications.  That's great...  I like BizTalk.  It's being used by real people to solve real problems.

But that doesn't mean that it doesn't need some improvements to meet our future challenges.  “Coordinating XML messages between applications“ isn't quite the same thing as “coordinating work between fine grained services“.  My top three “wish list“ items for BizTalk in the future would be:

1. Make it hostable

Shipping versions of BizTalk always run in their own process.  (In other words, BizTalk is a “Server“... not an “Engine“). That's OK if you are willing to host your service inside BizTalk.  But what if you don't want to host your service inside BizTalk?  For example, Indigo makes it really easy to build your own NT service (or even just a command line application for that matter) that can act as a service and receive Indigo messages.  In this case I want my service to host BizTalk, not vice-versa.  This “build your own service“ model doesn't work well with BizTalk today.

2. Don't require XML messages

At first glance... this seems like heresy.  SOA is supposed to be heterogeneous, and XML is clearly the best choice for messages in a heterogeneous environment.   I'm not questioning that point.  However, what about optimizations when both the service and the service consumer happen to be homogeneous?  Suppose the service and the service consumer both happen to be CLR-based applications?  Indigo supports this model.  The service consumer can “send“ a CLR class as a message, and a service can “receive“ a message in the form of a CLR class.  The specific wire format is totally transparent.  Maybe Indigo goes to XML and back, or perhaps it uses a more binary wire format as an optimization.

Current shipping versions of BizTalk don't support this model.  Someone who wants to send a message to BizTalk must convert their CLR class to XML.  Then once BizTalk receives the XML message, you have to convert the XML back to a CLR class. 

Items #1 and #2 are especially difficult if you want to use process within a single CLR-based service.  All your code is CLR-based (C# or VB.NET), and you are running in a single process (a single service).  Yet to use current BizTalk you would have to constantly convert back and forth from XML just to “talk to yourself“.  Further, if your service couldn't be hosted inside BizTalk, then you would also constantly be jumping across processes as well.

Cross process calls and constant XML serialization\deserialization is expensive.  It's OK for very coarse grained interaction between applications (the current BizTalk usage pattern).  But it won't be OK in the future if we want to use process in a very fine grained fashion.

3. Allow schedule creation via mainstream languages

This one doesn't have a direct tie to SOA, but it is important nonetheless.  Schedules \ Orchestrations are created in BizTalk 2004 using a language called XLang/s.  Of course, sometimes this underlying language is hidden by graphical design tools.  But if the graphical orchestration designer doesn't meet your needs... you will be “coding” in XLang/s. 

Now there are some good arguments for having a specialized language for process expression.  Having this as an option is perfectly acceptable.  However, it's the only option in BizTalk 2004.  If you want to code an orchestration in BizTalk 2004, forget your C# \ VB.NET skills... you've got to either use the graphical designer or learn a new language.  This seems a bit harsh. 

So here is the great news....  The BizTalk team is well on their way to addressing all three of these concerns!  You should definitely check out their PDC session on “BizTalk Futures”:

PowerPoint only:

http://microsoft.sitestream.com/PDC2003/DAT/DAT321_files/Botto_files/DAT321_Parasnis.ppt

Streaming Audio \ Video:

http://microsoft.sitestream.com/PDC2003/DAT/DAT321.htm

As a quick summary the BizTalk team is working on creating a “BizTalk Engine” which:

1. Is hostable by any application \ service. 

2. Can send \ receive messages either as XML or as serializable CLR classes. (Supports Indigo as a transport protocol).

3. Can be programmed using mainline languages (C#, VB.NET, etc...) using familiar concepts like code beside (the same model used by Avalon, MBF, and many other Microsoft products in the Whidbey \ Longhorn waves).

This is about as far as I can go without violating NDA stuff... but be sure an look through their PDC presentation.  That's the most non-NDA information that you can get at this point.  But I can say that I am <very> excited about BizTalk's future.  These changes will enable MBF to support process at a very fine grained level, both cross service and even intra-service!  And because the process is expressed in mainstream CLR languages, the same developer can reasonably code both business logic and process flows in the same language.

Welcome to my inaugural blog post. My name is Tim Brookins, and I'm an Architect in the Developer Division here at Microsoft.

For those without context, the “Unfrozen Caveman” was a running Saturday Night Live skit about a decade ago. I work at Microsoft, so I haven't watched TV in the last decade to know if the skit ever runs anymore or not!

My area of expertise is building platforms for business applications. Some people simplistically think: “Here's a database (SQL Server), and a language (C#)... now go build your business application”. But in reality it’s a lot more complicated than that, at least if the application is significant.

All the major players in the business application space have traditionally built their own proprietary platforms to support their applications. For example, SAP has ABAP and PeopleSoft has PeopleTools.

Even smaller companies have made large investments in business application platforms. I was formerly a principal architect at Great Plains Software, and I led the team which developed our own client\server based business application platform called “Dexterity”. Along the way, we acquired or merged with several other mid-market accounting companies: Solomon Software in the US, and Navision Software in Copenhagen, Denmark. Guess what... They all had proprietary platforms as well!

These proprietary platforms are <expensive>. Both Dexterity (the Great Plains business application platform), and MorphX (the Navision Axapta application platform) are complete, vertically integrated environments. They have their own form layout packages, their own database APIs, their own scripting languages, and even their own debuggers. Think home brewed Access, Powerbuilder, or FoxPro... it’s not cheap to build something like this from scratch!

So why do companies do this? Do they really enjoy investing boatloads of resources into building infrastructure rather than building the business application itself? Of course, the answer is “No”. Nobody wants to spend money unwisely.

Some might argue that these companies built their own platforms to provide “API stability”. I'll be the first to admit the Microsoft hasn't done a very good job over the years of providing long lasting, stable APIs. ERP applications are huge. They have thousands of forms in the UI, thousands of tables in the database, and millions of lines of business logic. They take years or even decades to build. Completely abstracting an application via your own platform can really ensure that some unforeseen API change doesn't force you to throw away your business logic and start from scratch.

But I argue that the “API abstraction” was an important, yet secondary motivation for building these platforms. In many cases, the cost of building the platform was probably more than the cost of re-writing the application because of technology changes.

So what was the motivation? Well, we didn't know it at the time, but these proprietary business platforms were early attempts at what we would now call “Domain Specific Models”, with an accompanying custom Framework.  You can see these elements by simply looking at the hard drive of a Great Plains install. An installed Great Plains system has a main application exe (the framework) which is about 3 meg. Then next to it on the hard drive is a giant data file which contains the entire application, as data (domain model).

Now of course both the Navision and Great Plains platforms were early, baby steps toward a full featured domain model with framework.  I don’t want to over blow their capabilities.  However, even in their early form, they provided enough value to merit the significant investment to build the platform.  We'll learn more about the compelling value of domain modeling over the life of this blog.

So enough of the history lesson… what about the future?  We believe that there are several gigantic paradigm shifts occurring which will revolutionize business applications in the next decade.  I don’t have the space in this post to discuss them all, but I will touch quickly on two of the most important:  robust domain modeling in mainstream tools, and service oriented architectures (SOA).

We’ve established that pretty much every significant business application developer has historically invested in building their own platform.  These platforms were typically simplistic domain specific models with an accompanying framework, based on client\server technology. 

The emergence of SOA pretty much obsoletes these client\server toolsets.  Of course business developers can prolong the lifecycle of their proprietary tools.  They can keep building monolithic client\server applications and bind them together at the edges with some XML and a few web service calls.  But sooner or later the life support will give out and customers will see the duct tape and bailing wire holding these applications together.

Don’t get me wrong… SOA doesn’t exist today.  The current “state of the art” is exactly duct tape and bailing wire.  But SOA <will> exist in the future, and when it does it will drive a tidal wave of change through the “I built my own client\server toolset” crowd.

This blog will talk about this gigantic impending shift.  I work on an effort within the Developer Division to build a toolset and accompanying service oriented framework which supports robust business domain modeling integrated into Visual Studio.  The codename of this effort is the Microsoft Business Framework (MBF).