How do I construct callbacks to work over a load balancer without affinity?

Let's construct a scenario to demonstrate this question. I have three machines; call them X, Y, and Z. X and Y are together behind a network load balancer. This is a server to server communication scenario, where two servers are attempting to talk over a duplex contract.

One of the load-balanced servers, X or Y, is going to first act as the client. Pretend that X is the relevant server in this case. X calls a service with a callback contract to Z. At some point in the future, Z is going to respond on that callback to the load-balanced group. If X passed its real address to Z, then Z has no problem making the callback. If X gives the load-balancer address, then Z will sometimes pick X and sometimes pick Y. The load balancer is not affinitized to a particular machine. The interesting case is where we haven't pinned X as the instance to respond to.

What can we do to make sure that the request by X is correlated with the response by Z, regardless of whether that response goes to X or Y? Well, either one of two things needs to happen.

• Z can stuff all of the necessary context information into the response message so that any server could process the response without having to know about the previous conversation. This is essentially turning a stateful problem into a stateless problem that sends a whole bunch more data. This has turned out to be a pretty interesting solution from the HTTP developer front.
• X and Y can share a common, durable store of correlation information. This is typically a database, but we don't have to be specific about how X and Y share state between themselves.

If you picked something in between going totally stateless and having durable state management, then there would be some interesting implications. There would be situations in which the receiving server would need to invent correlation information out of thin air in order to properly interpret the message. You can fake this some of the time, but sooner or later you'll get caught.

Next time: Poison Message Handling