Dealing with the "Melted Cheese Effect" - Contracts

I recently published Dealing with the "Melted Cheese Effect", the first of what should become a series. Here's my attempt at the second article.

Contracts

Introduction

In the first article in this series, Dealing with the “Melted Cheese Effect" I talked about the interconnectivity of systems, and how that effects design, particularly when we want to make changes to the system.

I introduced the melted cheese effect. Seemingly autonomous components and services are connected much like strands of melted cheese may connect raviolis. Services have dependencies in often undocumented and unintended ways. This article discusses contracts as a way to describe these dependencies and make them more manageable.

Designing Service Interactions – from the Outside in

First, I’ll position contracts using a sketch of the design process.

Looking at the whole design process, and being aware that we will only seldom get the chance to go back this far in the design, it all starts with the business. It should be possible to divide the business into units, each with their own responsibilities and reasonably separate from one another. Each of these units requires IT to offer certain functionality. We can identify services to provide that functionality and we can group related functionality together to define the responsibility of the services. I like to think of these groups in terms of sourcing: can I imagine outsourcing this business unit and their services, or can I imagine handing it to a different group during a reorganization, merger or acquisition? This leads to a grouping of services, where service interaction between the groups requires more care than within the groups.

After identifying such groups, the design continues with more detailed business goals, with the flow of activity and information through the business and the identification of the role the services have to play in this. Consequentially, the design of the service should be outside in. The first thing to identify is what the service will be used for and then how it will be used. The design of the service internals, the object design, the information design, the database design are all a consequence; not the starting point.

After identifying the services and their responsibilities, the design process continues by adding more specificity, with the use cases and the interactions in which the service participates. This is important. Before thinking about reuse, one has to think about use. It’s not so important what the service exposes; it is important how it is used.

Services have interactions or conversations with other services and with their clients. These conversations are about something, they have a purpose. The goal of such a conversation may be to drive a business process forward, to execute a business transaction, to convey information or anything else. These conversations are formed by a sequence of messages between the partners in the conversation. We model these conversations in what we call contracts. The services together with their contracts provide us an excellent high-level view of the functionality and information flow, a map of the service landscape.

Conversations, Contracts and Policies

I call the party sending the first message in the conversation the client and the other party the service. When using the word client, keep in mind that the client may be another service. A contract describes the structure of such a conversation, of a conversation pattern. The interaction between a client and a service may seem symmetrical; after all, the two are just sending messages back and forth. However, the relation is typically asymmetrical. One takes the initiative and the other responds. The service typically doesn’t speak until spoken to. It just sits and waits. When it receives a message, a request, it processes the request and responds. As with interfaces, the service implements the contract (Class AService: AContract …). The client uses the contract (AContract aService = (AContract)new AService();). The client will see every contract, or more precisely every endpoint, as a separate service. It will not be able to see whether multiple contracts are offered by the same or by different services. So, from the outside, each contract looks like a service. From the inside, a service can offer multiple contracts each on its own endpoint.

Ideally, contracts describe the conversations, their interaction patterns, their messages, the information and the semantics. Note that a contract often only refers to the description of the messages and the information as described in WSDL, since currently not even a proposal to standardize on the description of the interaction pattern exists.

A contract models the conversation between services. It always models the interaction between two parties. Any party can be simultaneously involved in multiple conversations. These conversations may or may not be related and they may or may not influence each other. A contract models the possible sequence of messages sent back and forth between them as well as the possible payloads of these messages.

Policies, on the other hand, contain all the restrictions on top of this: the communication mechanism that is to be used, the security constraints that are required, who is allowed to participate in the interaction etc. There are efforts under way to put supporting standards such as WS-Policy and related domain-specific specifications, e.g. WS-SecurityPolicy, in place.

There is an overlap between the two. There are constraints that can be part of a contract or of the policies. We can summarize the above as follows:

• A contract is everything about the behavior of the service that is engraved in code. They are drawn up at design time.
• Policies are everything that can be changed without coding. They are typically detailed later in the process and they are easier to change.

Or,

·         The contract is the domain of the application designer. It describes the interaction between the client application and the service application.

·         Policies are the domain of the infrastructure designer. They describe the interaction between the client infrastructure and the application infrastructure.

Keep these responsibilities apart.

Keeping in mind the current standardization of web services, a contract consists of the description of methods in a service contract and the description of all parameters used by these methods in a data contract. Most of these parameters are business documents or business entities.

As I discussed in the first article, “Dealing with the Melted Cheese Effect”, the service should offer an interface and the client should bind to that interface. The Windows Communication Foundation, WCF (formerly Indigo), makes the same recommendation to define the behavior using an interface. By setting the ServiceContract attribute on that interface and decorating the methods with an OperationContract attribute, it becomes the service contract. WCF also offers a ChannelFactory that allows the client’s programmatic binding to an interface rather than to the implementation in the service.

Loosely Coupling Conversations

Various types of clients and services may use the same service. For instance, the sales system is used by the call center as well as by the sales team and it interacts with the customer relations management (CRM) system and together, this accounts for just a very small subset of the interactions. So, when services are often used by multiple types of clients and a change is made to the service, all those clients need to change as well, don’t they? Do we even know which parts of the service are used? Do we know which clients use which parts? Do we know whether those parts actually did change?

• Unknown dependencies present the tightest form of coupling.

Contracts help make the dependencies between services and their clients explicit. A service can implement multiple contracts and a client can consume multiple contracts. Each contract describes a particular set of dependencies. They offer a good foundation to track these dependencies.

·         We believe it’s useful to have separate contracts for each conversation pattern.

The same method may be offered in different contracts and therefore in different interfaces of the same service. Having multiple contracts decouples the conversations. One contract may change, while the others remain unchanged. The clients of that one contract may need to change as well, while there is no need to update any of the others.

The first steps to defining a contract are to identify:

1.      The conversations in which the service will have to participate.

2.      The interaction patterns.

3.      What the individual messages should convey.

4.      What information or entities are passed.

Then during the further design process, each of these will become more concrete. This is clearly an iterative process. From there we recommend you define a contract for each conversation pattern a service has.

A service may work on reaching a specific business goal. Goals such as negotiating or accepting an order, fulfilling an order or managing a customer complaint. A service may be involved in an exploratory communication. For instance, what are the most profitable outstanding orders? Who are the customers involved? Who is the account manager? A call center application may have conversations with our ECommerce service about customer creation, order creation or order inquiry. We may treat these as different conversation patterns and model them separately.

Most contracts will describe the interaction between one particular service and one particular client, but many contracts will describe the interaction between a set of clients and a service, e.g. the contract for general ledger postings is used to integrate invoicing, payment and incoming goods. The more clients, or partners in the conversation, or if there are clients external to the organization, the more important it is to reduce the dependencies and therefore to invest more in designing better contracts.

How many contracts should a service expose? We don’t know.

How many operations should a contract describe? We don’t know that either.

If we define too many contracts, we have a manageability problem and we introduce overhead. If we define too few, we loose our grip on the dependencies. We suggest a few things to look at when deciding on the contracts to define.

We should be able to identify the topic of the conversation and we should be able to give the contract a name.

·         If you can’t name it, you probably don’t know what it does.

This is a golden rule that applies at almost every level of design.

If there are multiple clients for the contract, their requirements should be similar enough to capture in one contract.

The clients of the contract should use the whole contract and not just a part of it. If one method in the contract changes or if one data schema changes, the client is affected regardless of whether that part is used or not. Therefore, the contract should be a logical entirety.

If we define a single contract for each service, we still haven’t gained much. Since we can define multiple contracts for a service, we can organize these contracts by conversation pattern. This way, we have a reasonable chance that a change in one type of conversation does not have to change the others. In other words, by defining a contract per conversation pattern, we couple conversations more loosely.

In some cases, authorization is another good consideration for defining multiple contracts. Often one role is allowed to retrieve specific information and execute specific actions that another role may not. In the past, I have written filters to filter out what the user wasn’t allowed to see and to intercept calls the user wasn’t allowed to make. This was sometimes complicated code and it was always a lot of configuration. Another drawback was that the service would return sparsely populated XML so that making correct and useful schemas was much harder, especially if the results were to be used by generic code such as Microsoft Office Information Bridge Framework (IBF). An alternative is to define interfaces for specific roles and to configure access to contracts rather than to individual methods and schema elements. This may be useful if conversations are role specific and identifiable at design time. This is less useful if the differences are more granular or when access rights are likely to be set and changed by administrators during the lifetime of the services.

Designing the interaction

There is a lot of discussion about the granularity of messages, but the question should not be whether messages are big or small. The question should be whether the message is needed or useful and whether or not it contains sufficient information. Sometimes a small message suffices, such as when validating a credit card.   Sometimes a huge message is required, such as when a subsidiary orders 6000 items from the head office. Of course, we want to avoid chattiness because the number of messages has more impact on the performance than the size of the messages. Thus, setting properties using a number of calls is obviously a bad idea; after all, this is about communication with a service and not with an object. Combining multiple messages into a single message may boost performance. An example of the latter would be sending a batch of offline requests to speed up the synchronization process.

Another constraint is that the contract should not expose too much of the inner workings of the service. In general, that means that the contract should describe business functionality and not technicality or implementation. A message may refer to an order, but not to a session; to a user, but not to a service; to a business transaction, but not to an ACID transaction; to an amount in some currency, but not to a float or a BCD (binary coded decimal).

Of course, there are different types of services and some services offer technical functionality. Depending on where the contract is in the hierarchy of services, it is more or less important to provide pure business functionality. The higher up in the hierarchy - for instance workflows or other business processes and the services called by those workflows and business processes - the more important it is to provide business functionality and hide any technical details. Lower in the hierarchy of services and components, technical functionality may be required and exposed. For example., the database service and the authentication services will not understand customers and orders. However, the services offering business functionality ought to express their contracts in business terms.

When designing contracts, it is helpful to follow a simple rule: If I can’t explain the contract to the business people, it’s probably not good. This is true for everything that makes up the contract. I should be able to explain the sequence of the messages as well as the content of the messages.

For instance, can the people in the department working with my service understand:

• Why we need the message?
• All the information contained in the message?
• What responses are possible?
• All the information contained in the response?
• What the potential exceptions mean?
• Can the users provide all the information in the message?
• Will they expect the response?

This set of questions automatically makes it less likely that the internals of the service are exposed. The internal codes and algorithms will have a more technical nature and are not easy to explain to the people executing the business. This would not exclude the use of enumerations in the schemas. If the enumerations map to terms the users can understand, a translation mechanism would expose the meaning of the enumeration. Such a translation can be cached to make the communication more efficient and language-independent.

There is not yet much tool support for the design of the interaction, nor is there a standard. It is possible to use plain English to describe the interaction and that is infinitely better than no description at all. UML sequence diagrams offer a good and more formalized mechanism. However, the steps from here to interfaces, implementations, and schemas are still mostly manual.

Some may see BPEL4WS as a way to define contracts, but we believe that it is better suited to describe a process. Processes interact with many services and therefore interact through many contracts, so there is a clear relationship. We would argue that BPEL4WS is better suited to design or implement the processes using the contracts.

On the bright side, there are some tools on the market today that take a stab at managing the consistency between contracts defined using those tools and the BPEL processes. We believe that more tools will be emerging soon.

The Message Document

Not that far back in history all business was recorded on paper and even today, many processes are still executed on paper. Government agencies are especially known for their use of forms. People fill out forms and send them to other people who then take subsequent action. The paper processes may be slower than their automated counterparts are, but they typically scale very well and they are reliable too.

When designing service-oriented systems, I like to compare their interaction with interactions between people using paper. Tellers in a bank, for instance, work with customers, handle money, accept new customers, deal with address corrections and address change, or at least they used to. For each of these requests or actions that would typically happen there was a standard form and that form would be transported to the back-office for further processing.

Dedicated personnel would collect the forms and take them to the back office . In the middle of the twentieth century and even today, this is still the case, although pneumatic tubes are used to transport the information as well. Such forms do not just contain the old information and the new information; instead, they contain business requests that have been designed ahead of time and each of these requests makes it very clear what the request is. For example is the customer moving or did the phone number change? These two requests lead to very different business processes.

Sometimes a simple form may not be enough and the request takes the form of a folder, but the principle remains the same: the supporting information is bound to a clear request. The teller may or may not know where the form is going, who is going to handle it or what the transport mechanism is. It’s not part of the form itself. It’s part of the message and the messaging infrastructure. Forms of one type may go into one tube and forms of a different type into another. Forms may be sorted into different boxes or the recipient may look at the form and start the appropriate action or hand it to someone else. It is almost like a service-oriented system. The method is the request and it maps nicely to the name or type of the form. The content of the form maps to the payload of the message and to the data classes passed as arguments of the method.

The message contains the request as well as the supporting information. The infrastructure interprets the message and takes care of delivery at the correct end-point. The request and the service’s internals are oblivious to the routing that took place, to encryption, authentication, and so forth. The request directly maps to a method on the end-point and thus to a method on the interface implemented by that end-point. The data contract models the supporting information.

The infrastructure and the business logic change at different rates. The business logic need not know the details of routing, encoding and encryption, but sometimes it needs to execute authorization, especially if the authorization rules are complex or are part of the business logic. So, is it a good idea then to define the caller’s credentials in the data contract and thus to pass them as a parameter? I would prefer not to.

·         I advise separating the method and the payload from the infrastructure.

I’d prefer that the infrastructure passes the credentials and provides programmatic access to the caller information, for instance in the call context. This allows us to change the authentication method or even to support multiple methods without affecting the service itself. This means that I would like the designer and developer of the service and of the client to be responsible for the interaction, the methods, and their parameters. If they need access to infrastructure, they should use the infrastructure. If they need the user’s name or role, they can call the infrastructure. I want the infrastructure designers and developers to be responsible for everything that happens between calling the method and implementing the method. If the infrastructure wants to use the header of the soap message, great. If they want to use the transport layer, that is great too. (Just don’t tell the application designer J.)

·         Soap headers are exclusively for the infrastructure. The application designers should stay away from them.

Creating the Contract

Today, the WSDL, with its embedded schemas, is the most important physical result of the design effort, it is the one artifact that can be used by both sides to steer the conversation. As we have seen in the previous paragraphs, the current schema standard does not support all of the design decisions, but it catalogs many important design details.

I cannot avoid entering the, sometimes religious, debate whether to design the schema first and then derive the code or to write the code first and then generate the schema.

In either case, you will have to look at both code and schema and iterate between them. Make sure to look at the code in all languages used, and make sure that the serialization and deserialization works in both directions and leads to the expected result. For any large project with Web services, it is highly recommended to have someone who understands the WSDL.

Schema first

The result of the detailed specification of the contracts is a set of WSDLs. Of course, it could be the starting point as well. It is possible to use WSDL and XSD all the way. You capture the data contract in XSD. Then you import the XSDs into the WSDL and add the interface descriptions. From these you generate the data classes as well as the endpoints.

This is a good and recommended approach. However, there are a few pros and some important cons to consider.

On the plus side,

• This approach gives you complete control over the contract. This is very important
• When multiple organizations are involved and change management is hard.
• When integrating between multiple environments, such as C#, XML and Java.
• Guidelines for this approach can be found on the Internet[1]. People following the discussions on the Internet will find this a natural approach.

The downsides are that

• It requires skilled people who understand WSDL and XSD and can design and maintain your contracts. The larger the projects, the more people with such skills are required.
• With the current toolsets this is tedious and error-prone work.

If you want to follow this approach, you will probably like the WSCF[2] toolset to counter at least the second of these.

Code first

I have worked with many customers, large organizations, who would prefer a different approach because they don’t have enough people with the required skill set or simply because they believe the schema first approach is too expensive.

In this approach too, the contract is the starting point and the WSDL is the resulting detailed specification. The difference is that the source code is used to generate the schema. The data contract is made by designing classes in C# or any other language and the mapping to XML is defined using attributes. This means that the data contract can support any feature that makes using the data easier. Data binding, sorting of collections, and other capabilities can be added. It requires some discipline to enforce consistency though.

It is possible to implement the behavioral contract directly in the code behind an asmx file. We are not proponents of doing this. We recommend defining an interface instead and implementing that interface in the code behind. The interface may then be reused. I often have one or more separate assemblies with the interfaces and the data classes. If the client uses the same language, I will reuse these assemblies on the client side to create a proxy that exposes the same interface.

Many developers prefer this approach because code is much closer to their daily work than schema is.

The main disadvantages of this approach are

·         Defining data classes by hand is a lot of work and changing of the definition often requires change in multiple places.

·         It requires some understanding of the schema to select the appropriate serialization attributes for all the data elements.

·         The compiler does not check the placing of correct attributes on all the data elements.

These disadvantages will be reduced, although not entirely eliminated, with the arrival of WCF.

Mixed approach

We believe that trying to get the best of both by mixing the two approaches may work well.

This may look like this:

1.      Model the data using the schema editor in Visual Studio. The XSD-schemas are part of the Visual Studio project. Use a custom tool to generate the data classes automatically from these and to keep schema and code always synchronized. I use XsdObjectGen[3] for this.

2.      Define interfaces in C# or any other CLR language and handcraft or generate the asmx code behind files from these.

3.      The system ultimately generates the WSDL from the code.

The alternative solution is to define a separate project for the data schemas and to generate the classes from this schema repository.

We are working on a set of tools to support both approaches. This set of tools can be found on our website[4].

Designing the data contract in XSD is not simple and we are considering building a tool to show the structure of the data and hide the complexities of the schema.

The future

I must confess that I have had and continue to have many discussions with my coauthors on the subject, even though we agree on the fundamentals. We do believe that it’s not about schema first or code first. We believe it’s about contract first and that one of the outcomes should be an adequate schema.

In our opinion, the discussion about schema first or code first is about which tools are most suitable for the design of messages and entities. With a good complete set of tools that allow the design of the interaction patterns, the messages, and the entities, and that would allow generation of schemas and code in various languages, the discussion would be moot. I know that I’m making a controversial statement if I say that we believe that a very good toolset would not require the designer to look at the schema ever, although it would support looking at it. The schema is for the runtime to interpret, not for humans. In many cases, you are satisfied to know that the tool defines a good data format.

Do a client and a service need to share the class definitions? Clearly not! .Net and Java can interoperate, but clearly do not share classes. Even in a homogeneous environment, there may be many reasons to have different class implementations between client and service. One may require data-binding, the other may not, one may require simple addition and removal of elements to a collection, the other may not, one may be used in a single user environment, the other may have to be highly scaleable and may have to be reentrant.

Do they need to share the same schema? Yes, clearly! Or, not so clearly? They must understand each other’s information and therefore there must be commonality between the schemas, but one may have a newer version of the schema, as long as the schema is compatible. But, this detracts from the point I really wanted to make: the schema depends on the technologies used. In a pure .NET environment, or a pure Java environment for that matter, we could, and probably would, optimized our schemas for that environment.

We would like to define the data contract at a higher level of abstraction. We would like to define what information we would like to send and receive. Then we would like to have tools that generate the classes most suitable for the target environment and for the target requirements and that would generate schemas most suitable for the technologies used.

We do not know exactly what this higher abstraction will look like, but we do believe that it is the future direction. We also believe that WCF will move us in that direction.

We believe that something similar will happen with the behavioral contracts and that definition and tooling will become available to define the interaction and generate the artifacts such as class interfaces and WSDLs.

Until that time, we believe that it’s mainly a personal preference. Simply put:

·         If you understand WSDL and XSD inside out, you may want to start with schema.

·         However, if you’re not an expert, we think that starting with schema may be an expensive proposition and it may be wiser to start with code.

Conclusion

Roughly half of the service-oriented world provides contracts for the other half to consume. This may not be entirely true, although I do not dare to make an estimate of contract reuse and sometimes these halves overlap. In many B2B scenarios and even in many inter-departmental scenarios, one party designs the service and another designs the client. Many others own both endpoints and control the contracts and the changes to them on both sides. Yet many others work on standard contracts like those provided for the health care industry. The more distant the designer of the service contract and the client of that contract are organizationally, the more negotiation it will take and the more expensive it will be to change that contract. The closer they are organizationally, the easier it is to manage change. Contract design is complex. Design contracts for the organizational distance that is required. With the current tool support, schema first is probably your best bet when service consumption crosses organizational boundaries or when interoperability is important. Otherwise, when the organizational distance is small and especially if you own both sides of the interaction, you may want to consider what I called the mixed approach.

In all cases, our recommendation is to design and describe the interaction as well as the expected business behavior. The schema in the form of WSDL is only one artifact.

The next article will go into the design of a simple set of services and the effects some design decisions may have on functional change.

Journal

I should have said where to expect the Journal 4.

The journal 4 will be on: http://msdn.microsoft.com/architecture/ feed: http://msdn.microsoft.com/architecture/rss.xml

A previous journal can be found at: http://msdn.microsoft.com/library/default.asp?url=/library/en-us/dnmaj/html/aj3intro.asp

Razorbills

For a long time now, I have been thinking about exposing the functionality of services. I called that the “what of services". The WSDL and the complete WS* stack to me are the "how of services", how to encode the message, how to encode the payload, how to encrypt, route, authenticate, how to transfer reliably etc. But not, what does the message mean, what does the service expose etc. The concept of describing this “what” of services, based on entities encapsulated by services, is the basis for IBF (Information Bridge Framework). I believe strongly that the concept of entities, views, relationships, references and actions, together with a metadata description is important by itself and that it can be applied in many more cases.

 

Then the moment came that Bill asked about IBF and asked how to move forward.

 

I had been walking around with the idea of writing a paper on the subject for a while and now I had a real incentive to do it fast. I had been looking for a title for a while and hadn’t come up with anything, until, that same morning I thought of the memo written by Adam Bosworth in which he pushed for using XML as the format between services on the internet. This was a very influential memo and it led to, or at least hastened, the SOAP and web services effort in the company. Adam had called his memo “Sea urchins”.

From my childhood I remembered a few little riddles:

Why did the fly fly? Because the spider spied her.

Why did the owl howl? Because the woodpecker would peck her.

And this one:

Why did the razorbill raise her bill? Because the sea urchin would sea her chin.

 

So, I decided to build on the success of “Sea urchins” and take the next step to “Razorbills”. I wrote the memo on one evening, worked it over the next day and sent it out hoping the name, and thus the concept, would stick.

 

I can’t say I got many responses from within the company other than from people in my own group. One of the responses was “A razorbill is it a penguin or does it just look like one?”

Well, I knew the answer to that one: It’s not a penguin. But, it was enough trigger to do a little further investigation.

The razorbill is a bird in the family of the auk. Even though the razorbill may look much like a penguin, it is actually a different species. Never the twain shall meet unless it’s in a zoo. The razorbill lives in the northern and the penguin in the southern hemisphere. Due to the similarity in environment, they have evolved to similar birds (ecological equivalents). According to some websites the razorbill is a threatened species. Apparently, the now extinct great auk or “Le Grand Pingouin”, of the same family as the puffin and razorbill, lent its name to the penguin.

Just a few links for those interested:

Penguin Post Newsletter Puffins

Boreal Forests of the World Bird Species - Razorbill

Tordmule, Razorbill (Alca torda)

http://www.frontier-travel.co.uk/wildlife/images/razorbill.jpg

Photos of Razorbills - Ardnamurchan Charters - West Coast of Scotland

http://www.west-scotland-tourism.com/ardnamurchan-charters/razorbills/razorbillFlying.jpg

http://www.west-scotland-tourism.com/ardnamurchan-charters/razorbills/razorbillFlying2.jpg

And, as you can see, unlike penguins, razorbills can fly.

 

This all happened a few months ago and in the mean time I spent a little more work polishing and then even more time waiting until the Journal 4 would come out… I’m still waiting, but it should be happening pretty soon now and Razorbills will be in it.

CRUD, finally

It took a while, but my “CRUD, Only When You Can Afford It” article has finally been published. I’ve been working with Jon Tobey on my articles to make them more readable and more consistent. I can tell you, when he returned my first draft (the blog I wrote earlier) there were so many questions in it. My first thought was that I had to rewrite much of it and I thought of the amount of time I would have to spend. But, it’s a good thing. If he asks me how I get to a conclusion, tells me that two sentences are contradictory, asks for an example or even asks me how this fits with one of my other articles or someone else’s article, then I have to take those questions seriously. If my wife reads it, she’ll tell me where she lost me, if Jon reads it, he’ll tell me why.

 

Jon also helped me with “Dealing with Concurrency” and with the “Razorbills”, which I hope will be published shortly in the “Microsoft Architects JOURNAL Issue 4”.

 

For those who read my blog, I hope they like the article better.

Razorbills - What and How of Service Consumption

Razorbills
What and How of Service Consumption

Why did the razorbill raise her bill?
Because the sea urchin would see her chin!

 

Less than five years ago, I read a Microsoft internal memo called “Sea Urchins”. This paper pushed for using XML as the communication mechanism between services. Since the “Sea Urchins” memo, we have made great progress in describing the how of services. With “Razorbills”, I propose putting a similar effort behind providing a description of the what of services.

I propose standardizing on a description of this what based on Entities, Views, Actions, References and Relationships. Service Oriented Architecture is as much, if not more, about consuming services as it is about building services. To enable better consumption I believe we need infrastructure that allows people that may be less technical but do have domain expertise (the so called business analysts) to design and build solutions. We need infrastructure that allows users of any application, structured forms as well as free form documents, to really use the functionality offered by services. Standardization on a way of describing services in business terminology will help do that. It will help business analysts and users to better bridge the gap between the structured and the unstructured, the formal and informal, both in content and in process.

How

Current service descriptions such as WSDL and the WS-Standards describe how to access functionality. Once the caller has decided which functionality to invoke, the WSDL describes how to encode the message, how to route the message, how to secure the message, how to authenticate the caller etc. It does not describe what should be called. For any dedicated client or consumer of a service that knows what to call, this is great. The infrastructure and the description are based on standards, so that there is much code reuse and many technologies can interoperate. A web of communication, based on a single infrastructure, can be created.

However, this implies that the logic of what is being called is coded in the application. When organizations want to be flexible, they may want to access services to get information, aggregate information, reorder information or they may want to change process flows. In order to do this, they need to rely on developers. This may make the desired changes expensive and, in many cases, too expensive. Organizations facing this problem would like to have these changes made by business analysts and even business users rather than developers. It would make change cheaper and it would shorten the cycle of change and make them more agile.

If our software could communicate this what to the business people, then they could define new aggregation of information, change or create the flow of processes.

If we describe what the business wants to do and we describe this in terms understood by both business people and software, we would not only have a description of the how, but also of the what.

This information is available today, but it is in the heads of the designers and developers at worst and in some design documents at best. The consuming applications are being written using this knowledge. However, it is not available to tools and other software that could be used by analysts and users to solve their challenges. We need to find a standard way of describing this information.

In the following I will explain the fundamental idea of such a description of what services expose and then I will highlight possible usage of such description. I believe and I hope that I will be incomplete; going down this road will undoubtedly provide us with more insight and more ideas.

What

Services expose business functionality to their consumers. Business analysts and business users often prefer to think in terms of the entities they use. They understand the relationships between these entities and the actions that can be taken on those entities.

Entities: Most services encapsulate entities. However concrete these entities were for the designers of the service, for the consumer of the service they are abstract. The consumer cannot touch them or manipulate them; the consumer does not know whether they are stored in a single table or in a set of files. The consumer however does understand what they represent: Customer, Order, Problem report, etc.

Views: The service exposes views on these entities. The consumer may obtain an address view, or a financial view, or a historical view on the customer. These views are snapshots of the entity, taken from a specific viewpoint. The views are concrete. They have a well defined schema.

Actions: The service also exposes methods or actions. Many of the actions are related to an entity. E.g., release order or upgrade customer are clearly related to specific entities.

Entities, views and actions are artifacts that we can describe, that a user can understand and that may help us describe services in ways that can be understood by business analysts.

I want to mention other artifacts that are less often used.

References: Many entities can be identified using key attributes. However, they can often also be uniquely identified using other attributes. The customer can be identified using his social security number, the tax number or the DUNS number. A list of customers can be retrieved via the zip code, the region or the account manager. We can formalize this through the concept of reference. A reference defines either a unique instance or a list of instances. A reference, like a view, has a schema.

The actions exposed by the service accept references to identify the entities they operate on. The references are a more formal way of defining possible input.

Relationships: Now, if we have a view of one entity, an order say, it should not be too hard to build a reference to the customer. In other words, we can build relationships that transform a view into a reference. These relationships can be defined using the schemas of views and references and these relationships are not constrained to a single service; they can define relationships between entities encapsulated in different services.

If we are able to describe these entities, the views on these entities, the relationships between these entities and the actions on these entities in terms that are understood both by the business people and the software, we would have a description of the what. This description can easily be linked to the existing description of the how.

Much of this information can be provided by the designer of the service interface in a similar fashion WSDL is currently provided. However, an extension to WSDL cannot solve all our needs. Relationships between entities offered by different services should not be stored in either, but would require storage external to these services. Instance specific information, such as the current availability of relationships and actions, as well as information created at run-time, such as tasks in a collaborative workflow, require run-time interrogation of the service and thus a run-time description. This is very much related to service contracts and the required infrastructure for providing run-time information about the state of a specific conversation.

Since views are what the designer exposes of the entities, much of the information is available in the systems today. Every service that was designed around entities provides most of the information needed and many other systems the analysis has been done and has been captured in some form of metadata.

Entities and Views in SOA

The differentiation between entities and views is important in a distributed environment. Data is copied between the systems and whenever the consumer of a service retrieves data, it will get a copy. When I request information about an entity, I will receive a copy. This copy is a view on, or a snapshot of, the state of the entity from a particular angle. I may receive the customer’s financial view, or the customer’s satisfaction view, or the customer’s contact view. These views depend on what I want to do and what I am allowed to see. When I receive that information only a few milliseconds later, the service may already have moved on and changed its internal state. The view may be stale even when I receive it, and the chance of it being stale increases the longer I hold on to it. The differentiation between View and Entity is an important recognition of the fact that the consumer has an outside view on the entity and that there is a time difference between creating that view and receiving it. It’s like taking a picture; the picture is frozen in time, whereas the subject of the picture may move on. If you take a picture of a building, you will only see the building from one angle. The building will not change after the picture has been taken, but the people on the picture, the clouds on the picture and even the trees on the picture will move. The snapshot is taken from a particular angle or viewpoint and frozen in time.

Because views are copies of data and may be or become stale, the concept of references is important. They provide access to the source o the information, to the entity itself. They allow us to convey and store information and let the service that encapsulates the entity deal with the concurrency issues. References are less likely to become stale. They will become stale only if the entity ceases to exist. Views and References are the carriers of the information. They are the basis for information integration and entity aggregation. Understanding their relationships to entities, their behavior and capturing their schemas should form one of the fundamentals of any distributed architecture and thus of any Service Oriented Architecture.

CRUD, only when you can afford it (Revisited)

So, I did a little more work on it and I hope I addressed the feedback in it. I don't think I'll be providing code anywhere soon though. I hope that the project that David Hill and I are doing together with msdn “Patterns and Practices”, we'll get to good examples for offlining and offline UI. The article doesn't really have a beginning, a middle and an end yet. A few more internal passes and I hope I can publish this officially.

After my article on Dealing with Concurrency: Designing Interaction Between Services and Their Agents (http://msdn.microsoft.com/library/en-us/dnbda/html/concurev4M.asp) and my talks at the TechEd in Amsterdam, I needed to express my thoughts on CRUD.

I concluded that article with the following recommendations:

• Confine pessimistic resource usage
• Make information predictable
  • Manage the time of validity
  • Personalize by assigning ownership
• Limit optimistic updates
  • E.g., Add rows instead of update fields
• Design business actions
  • State your purpose
  • State your preconditions
• Design postings
• Use the Journal pattern

When I wrote “Limit optimistic updates”, one of the things I meant to imply is to limit the use of CRUD. Don’t use CRUD if you don’t have to. In this article I try to point out in which cases you do have to use it because, I believe, that in those cases you should. Using replication is a great way to go if your problem allows it.

Why don’t I like CRUD?

Hearing people talking about CRUD makes me cringe; I feel the urge to react. So, let me get that off my chest first.

CRUD, Create, Read, Update and Delete is about working with entities and changing those entities by retrieving the data from a service, modifying the data by setting properties and then sending the data back to the service for update.

The reason for my allergic reaction may stem from the way I see developers use Datasets the wrong way. Such a developer will write a service that gets you a Dataset, the UI modifies the Dataset and sends it back. The service or even the stored-procedure checks the before and after image, the timestamp or the version … ouch!

Why not have a request specifically stating what the request entails and specifies when the update should be executed or refused? E.g. buy this when the price is at most x, rather than buy this when the version of the information retrieved in an earlier request was 30256917 or worse: the old address was …, the new address is … Did the customer move? Then we better start a complete process. Or did we just correct a typo in the contact information?

CRUD is data oriented. I don’t ask for a specific business action, I just ask the service to create an entity, or delete an entity, or I ask it to update the entity by giving the service the new data for that entity. The service derives from the data changes what action needs to be executed if more needs to be done.

Most order processing is not CRUD, or at least I don’t think it is. An order can be created offline and then sent (replicated if you will) to a service for processing. Processing of that order will affect many of the related entities. The customer information will be updated and more than only the year to date totals may change. The properties used for price and discount calculations may be updated, because the customer reached the critical order mass and is upgraded. Products may or may not be available; delivery dates may or may not have been realistic etc. CRUD to me means that the entities are read, created, updated and deleted. Submitting an order to an order system, regardless of the physical implementation has the semantics of requesting the service to accept and process a new order. Such an order is a complex request for a complex business transaction and I regard such a request as a “Journal”.

If you do not use a CRUD interface, but offer business actions such as upgrade the customer or set the new quantity for inventory, the service will execute the request and then, deep inside the bowels of that service, use CRUD on the database. If you offer a CRUD service interface, the service will in most cases, take the property changes and map them to logical requests that are executed and, again, deep down in the service use CRUD to do the database updates.

Problems with CRUD at the service interface level are that CRUD

• Looses the context of the change (e.g. the contact info changed because the contact has another function in the organization or the contact info changed because the department was renamed)
• Implies dependencies you may not want (e.g. the sales system refused the order because the price is not the same anymore, even though it has been reduced)
• Omits dependencies you may need (e.g. only place the order if the time to deliver is less than 10 work days)

Replication

Yet a lot of solutions have been built using the principle of CRUD and at least some of them were very successful.  CRUD has been used successfully for entity aggregation as well as for offline scenarios. Why did they choose to use CRUD?

Replication is the main answer; ease of interface design and implementation may be another. People want to use replication as offered by email systems and databases. Even though I will not go into the various ways that data replication can be configured in databases, I will look into how replication can be used.

Replication is about keeping two or more systems in sync while changes may occur on each of these. As we are all aware, in computer science there are only three numbers: zero, one and more. Hence, when I write about replication between two systems, feel free to extrapolate this to more.

When two systems, each with its own data store and each with its own business logic need to synchronize their information (merge replication or two way replication) I can either

• replicate at the data level, or I can
• replicate at the service level.

When replicating at the data level, the business logic in the services on either side have no easy way of protecting the consistency of the data; the data will just be replicated and the business logic will only execute when a conflict is detected, if at all.  This also implies a high degree of trust and thus coupling between the systems; the data in the system is changed without much scrutiny.

Data level replication has the advantage that I can use standard mechanisms such as database replication. This immediately leads to a couple of other advantages. These mechanisms are typically fast, they sometimes support synchronization through firewalls and, the design and coding effort is greatly reduced.

The alternative is to provide some additional logic to walk through the changed data on both sides and use the service’s published interface to replicate the changes. Normally this is more development work and it may carry a performance penalty. But, when you choose for the latter, you are not constrained to using CRUD; you may use any mechanism you want.

As a side note, when designing for replication, it is wise to assign ownership of the data. We often talk about single master. The essence is that for each piece of data a single owner exists, so that conflict resolution is made easier. Note, that a single owner can be a system or a real person; important is that there is only one.

Even though service to service replication needn’t go through CRUD, often entity services will offer CRUD semantics and replication mechanisms are often easier to build using those semantics. When a request comes in, the before image of the entity is retrieved, the changes are being applied, the after image is retrieved and both before and after image are stored in a change log for later use during the replication. This looks like reusable code doesn’t it?

I can make this even more complex by adding a data store in the middle and using data replication between. This would be useful in a scenario in which I use Outlook as the front-end to a CRM system for instance. Outlook and Exchange would replicate the data using the available mechanisms and Exchange and the CRM service would replicate the entities through the business logic of the CRM system. I will get back to this scenario later.

Similar approaches

Such solutions seem very reasonable. So why can’t we use CRUD or can we? Let’s compare the CRUD based service interfaces with a few other cases.

In object oriented design a class typically has a set of properties. These properties are accessors (get / set) to the private data. Properties are typically used to keep the internal state consistent; they are not used to trigger complex business logic. The set methods are typically held small and they typically have none or little effect on other classes and instances. If they were to change state in other classes as well, changing of a property would lead to a cascading effect of property changes and the system would become unmanageable. Properties are used by various levels in the design. It is important to note however, that not all of the set methods on these properties are declared as public. Only those properties that are without side effect when changed are typically made public. The other ones will be private if they were designed as properties in the first place.

For instance the Customer class in an insurance application may have an address property, but changing the address may result in an extremely complicated business process. Thus this property will typically be private, so the change can only be done after all the side effects have been dealt with. For an address change, the designer would typically offer a method, which internally will use the private property.

Similarly CRUD implies change of properties (Luckily it allows us to send all properties at once; otherwise performance alone would make CRUD unworkable). Property changes that have side effects, i.e. effects that transcend keeping internal state consistent should not be supported (be prohibited). For these, requests should be defined that convey the business meaning of those requests as well as the conditions under which they should be executed.

In working with databases CRUD is the normal way of dealing with data. But there are a few notable differences between talking to a database and talking to a service.

• Data in the database is typically normalized. Properties aren’t replicated; a single update typically suffices to maintain consistency. But, I don’t think it’s a major difference if we look at a single service. Good entity design should help and a service could (and should) provide the logic to keep the data consistent. It becomes different if the same information is held in multiple services; then logic external to these services has to keep the services in sync.
• Database updates are transactional. This means that multiple updates that are, at least from the database’s view, unrelated can be combined in a single all or nothing, or ACID, transaction. When one of them fails, all fail. In contrast, multiple updates to the service are not combined; logic external to the service has to maintain consistency. If the service only provides a CRUD interface, a business action that updates multiple entities must be split into multiple updates, each on a single entity. The service will not offer transactional consistency between them, simply because it cannot. The caller has that responsibility. But, in replication scenarios, the caller does not take that responsibility either.
• Updates to the database are not protected by business logic. That protecting business logic does the database updates and doesn’t require additional logic to be executed, much like the properties of a class in object oriented design.

I think all three points lead to the same conclusion. Services are not databases, but by only providing CRUD interfaces, they offer a poorer form of database like behavior. If services should offer business actions and business logic, they should be given the information needed to execute such business actions and ensure that business logic. The service encapsulates entities and it protects consistency through business logic. If the original business request affects multiple entities, a service can only provide complete service if enough information about the original request is given to the service. Then it can execute business logic, apply that business logic even across multiple entities and maintain both information consistency and transactional consistency.

When CRUD

Now, this may all be true, but is it important? As is always the case, sometimes it is and sometimes it isn’t. Architecture and design is all about making the trade offs. So, when is it important to expose business actions instead of CRUD?

• If it is important to maintain consistency or if there is a reasonable chance of concurrency issues and if it is not easy to resolve those concurrency issues try to use business actions and avoid CRUD. Booking a travel where it is important to have a flight, hotel and car or booking an order where both products and service are required at a specific time are examples of requests that require consistency. The simplest examples of concurrency issues are the general ledger and the stock on hand information; they are hardly ever, if at all, exposed through CRUD. Not because of dependencies, but because of the concurrent nature of updates.
• If, on the other hand, updates are seldom or are only done by a single person, or, if the updates consist of adding information rather than changing existing information there is not much of a problem. The contact information in a CRM system will not be changed often. The appointments in my calendar are typically only changed by me. Even if our admin has complete change rights to my calendar, the chance of both of us changing the same appointment is slim, because of the common business practice that she just does not change my calendar unless she has to. In many cases, concurrency issues are guarded better by business practice or habit, policies and process than by the software. The software should support these and possibly enforce these, but without them, the software enforcement makes little sense.

To summarize this:

• Only use CRUD when
  concurrent updates can be avoided because
  • updates are seldom, or
  • updates have only one source (person or system)
• Design changes as creation of entities rather than updates

Granularity or chattiness is neither an argument for nor against either approach. When the service exposes methods that accept complex requests and even combinations of requests (as in the Journal pattern in Dealing with Concurrency), requests may even be less chatty than when using CRUD. On the other hand, the synchronization mechanisms are often well tuned. And both approaches are obviously infinitely better than sending individual set-property requests.

An example

At Microsoft, the people in the field use only a subset of what the CRM system has to offer for most of their work. They keep their contacts up to date. They add opportunities and activities and keep these up to date as well and they read the customer information. What if we were to provide this to them via Outlook? Outlook clearly only supports replication, so that all changes made to local data would be replicated to the server and into the CRM service. Every context other than the new entities and the changes to the existing entities would be lost. Still, I believe it would be a good, even recommended, solution.

Customer information is provided in Outlook but cannot be changed through Outlook.

Creating and updating of contacts is not critical. Contact information is shared, but when the contact information is changed, it is a minor change without much business logic. When an update fails the user may receive a synchronization error and deal with it is he or she deems appropriate. The chance of such a failure would typically be small.

Creation of an opportunity triggers a business process or workflow. But the creation of the opportunity in Outlook provides enough information to that process and this creation does not depend on any other changes. Even if these other changes are related, the opportunity does not have a “transactional” dependency.

Adding activities is uncritical. Changing activities might lead to conflicts, but much like appointments, activities can be though of as personal information. i.e. the owner of the information is known and that ownership is respected by common practice. The activity may be created by one person and then owned by another, but that is normally arranged outside of the software.

More complex cases don’t have to be offered through Outlook. The original solution provides these cases. Most users, and especially occasional users, will have sufficient functionality through Outlook and can hook up to the corporate network in the seldom case they need more; more demanding users can still use the original solution and have more overhead, but also more functionality.

Best of both worlds

Until here I’ve discussed CRUD as an all or nothing approach. It doesn’t have to be that way. That would only be the case if you were to rely entirely on standard replication mechanisms. A good way of combining approaches is to replicate information from the service and to queue requests to the service. (You can optionally change the local data to reflect those requests, but if those changes are ignored and overwritten after the requests have been processed by the service these local changes don’t affect the solution.) The business actions in the queue will then be processed and these actions can follow all recommendations made in “Dealing with Concurrency”. Instead of queuing the requests in a queuing system, I can store the requests in a database table and have them replicated. Another way of looking at this is that I may use replication as a messaging mechanism. Processing the replicated requests is exactly the same as processing requests that have been sent to the service using mechanisms such as SOAP.

A variation on this is to model the requests as entities. The order is a good example of this. The order is essentially a request to the service. Modeling the order as an entity has the additional advantage that it has become part of the business system and that it now can be tracked and audited. Another example would be the address change of the customer. Instead of designing a request for address change, I could design an address change form that has all the required information. The form would state the reason, such as correcting incorrect information, a new telephone number or moving to a new location; it would have the desired date of change etc. Now, if the customer calls me six months later, I have the original request and the activities that I executed that relate to that request. The address change form is now an entity in my system. I can use many of the standard replication mechanisms to replicate orders and address change forms. They are new instances in my system or database and they do not introduce concurrency issues. Note however, that neither the local order entry nor the local address changes need to have any effect on the other local data. Processing the request (or new entity) by the service may affect the other entities and those changes will then be replicated back to the local store.

Suppose you want to write a solution for a service technician. The technician needs to fill out hours on the road, hours spent servicing, diagnosing and repairing. He or she needs to specify the material used, specify the condition of the devices, etc. The information about the customers, the devices, the maintenance contract per device, the prices for the materials, all can be replicated to the notebook. However the per customer service information that is entered consists of more than just an invoice; it can be a complex set of requests to the service that may contain a few problems on synchronization that need to be resolved before the complete set is accepted.

Not knowing your dependencies is the tightest form of coupling