Business Process Versioning - Updating Running Business Processes

Continuing my coverage of best practices and patterns in the Business Process Management (BPM) scenario I examine how the problem of versioning the business process has been addressed. Even though I have been able to base much of the content from a presentation I did on this topic six month ago this has by far been the hardest post to write for me so far, I have been working on it on and off for about a month without really being satisfied with the contents. But after another rewrite over the weekend I think I have managed to catch the essential parts of Business Process Versioning, as illustrated by the BPM scenario.

The Problem

A business process that contains long-running steps, e.g. async receives from external systems or delays, may be hard to update. One example of such a business process is when you involve human interaction in your processes, you simply do not know if someone will complete a task in five minutes or in two weeks. Another example is waiting for a response from an external system which will batch responses to be sent every night.

In BizTalk terminology you wish to deploy a new version of the orchestration implementing the updated business process and the first step is then to undeploy the old version, the problem is that if an orchestration has running instances you cannot undeploy it. If you have a frequently used orchestration with long-running steps there is a high risk that you always will have running instances of the orchestration, so you will never be able to undeploy the old version and hence never be able to deploy the new version...

There are actually two problems:

1. You cannot easily deploy a new version of the orchestration as the old version cannot be undeployed, this us due to that it has running instances which will have to run to completion first.
2. Even if you are able to deploy a new version of the orchestration (using two parallel versions) the already running instances will NOT pick up the changes you have made to the business process.

The first problem is probably a good topic for a future post but in this one I will address the second problem, as that is what the BPM scenario addresses.

An Example to Illustrate the Problem

Suppose you have an approval process with the following steps:

Request received => Notify approver => Wait for approval => If approved, send to supplier

Typically this process will swiftly run to the "Wait for approval" step and the it will have to wait for the approver to take an action. Then suddenly requirements change for the approval process, it is now also required due to SOX compliance that we log the result of the approval process. The process now looks like this:

Request received => Notify approver => Wait for approval => Log approval result => If approved, send to supplier

Assuming you are using an approach with monolithic orchestrations all the steps of the approval process will be implemented as a single orchestration. As a result you will not be able to update the business logic of the already running approval processes, even though they have not yet completed the "Wait for approval" step. Of course you can deploy a completely new version of the business process and have new instances of it pick up the changes, but you will still have the problem with the running instances.

The Solution - A Request Manager with Multiple Stages

To understand the solution you need to realize that a business process does not need to be represented as a single orchestration. This is probably nothing new to you as you may have used Call or Start shapes, but have you really considered splitting the main logic of the business process into multiple semi-independent orchestrations. That is what the BPM versioning pattern is about!

Stages

The solution to the problem of updating already running instances of a business process is to avoid building large orchestrations that include all the steps required for the business process, instead you should split your business process into multiple smaller orchestrations which (when executed in sequence) provides the required steps. In the BPM scenario each such step is called a Stage.

A key decision when using the BPM design is to decide which business logic goes into which stage. One way to do this is to do a prototype design using a single orchestration to understand the characteristics of the business process, and the split the business logic into stages using the following guidelines:

• Split along long running persistence boundaries, i.e. where you wait for async responses from other systems. Typically this is immediately after a request-response exchange.
• Keep atomic parts together. Parts that should be executed as a unit-of-work should be in the same stage, even if they don’t formally enlist into a transaction.
• Each stage will add overhead to the system, which will impact both performance and scalability. So to create a stage out of every single shape in your orchestration is probably not a good idea.

While I have not used this pattern in any real project yet I feel that keeping the number of stages low is key. If you create a stage for each of the logic part of the business process you will end up with a large number of stages that you will have to administrate and maintain. In the example above I would only create two stages: one with the "Request received", "Notify approver" and "Wait for approval" steps and one with the "Log approval result" and "If approved, send to supplier".

The key design issue with stages are that at every point where your orchestration does not immediately complete a step there is a risk that someone have deployed a new version of business logic that needs to be used by the already running business process. So I would concentrate on the the guideline: split along long running persistence boundaries.

Request Manager

The BPM scenario also require that you implement an über-orchestration that is responsible for driving the business process forward, it does this by calling the stages in sequence. This über-orchestration is called a Request Manager which does not contain any business logic.

When the business process must be started the request is routed to a new instance of the Request Manager (RM), each specific process will have it's own instance of the RM orchestration. The RM initiates the first stage and wait for it to complete or fail, it decides which by examining the message which is sent back from the stage. When the first stage has completed the second stage is initiated by the RM and then it waits for its completion. And so on… This way the RM doesn't really need to have any knowledge about the specifics of each stage, it is only responsible for driving the process forward by calling the stages one by one, as long as the are successful.

A key design requirement for the Request Manager is that it should be so generic and free of business logic that you would never have to version it. Because if you have to version the Request Manager while you have running instances of it you're back on square one, you have gained nothing!

Implementation

How do you implement this pattern? It may not be completely clear when you examine the implementation of the BPM scenario exactly how the implementation is done and which parts of the implementation is most important. Let me give you a few guidelines:

• The OrderManager project contains the OrderManager.odx orchestration which is the implementation of the Request Manager in BPM.
• The stages are implemented by CableOrder1.odx and CableOrder2.odx (in projects CableOrderStage1 and CableOrderStage2), but much of the business logic is actually implemented by orchestrations in the CableOrderActions which initialized using Call shapes in the actual stages. It is not necessary to look at the orchestrations in CableOrderActions in order to understand the versioning pattern in BPM.
• The message send from the Request Manager to each Stage is a multi-part message, see OrderMgrMsgType in the Orchestration View. The first part contains routing information used to find the correct stage, and the second part is an XmlDocument which can contain any type of message. Using XmlDocument is key to keep the Request Manager free from stage-specific implementation details which may change, and if they change there is a risk that the Request Manager in no longer valid and need to be versioned (back to square one!).
• The Request Manager does not contain any specific knowledge about the type of stages involved in the business process and not even the number of stages is hard-coded (it's read from the SSO).
• To initiate a stage it publishes a message to the MessageBox using a direct bound port (actually it uses Inverse Direct Binding which you can read more about here), and it is the responsibility of each stage to activate itself when its stage number is published. The stages uses a filter expression on the activating receive shape to determine if a message is targeted for them.
• There is a Loop-shape which executes while there are unexecuted stages. So when one stage has been successfully executed the next will be started.
• The implementation becomes a bit cluttered as it also contains an implementation of the Interruptable Business Process pattern (which I will examine next in this series on BizTalk scenarios).

One thing that I find a bit unsatisfactory about the implementation is that the stages are executed in sequences based on numbers (1, 2, ...). I would have preferred a somewhat more complex control process, I have one in mind and will think about it some more to see if it works.

Conclusion

Using the Request Manager - Stage to implement versioning of business processes a great pattern that everyone should know about, but you should only use it when you really need it as it complicates the design a bit.

The implementation of the pattern in the BPM scenario is not entirely easy to digest, as it contains implementations of other complex patterns as well. I'm considering providing a smaller and cleaner implementation which only focuses on the Request Manager - Stage pattern, but I won't promise to do this.