Change agents break the rules. They think on the outside. They point out the stuff that is fundamentally broken. Administrators follow the rules. They make more rules. They believe that order equals simplicity.
Is it possible for a change agent to become an administrator?
I've been tempted. I've thought "if they would only follow the 'new rules' then we wouldn't have this problem." It isn't true.
We can't make rules to change rules. A leader cannot 'govern' the way out of a broken situation. The only way out of disfunction is cooperation, teamwork, and leadership.
So if you really see your role as a change agent, don't enact rules and then whine when others don't follow them. Get in front. Communicate. Demonstrate. Sacrifice.
Only a leader can truly effect change.
In many organizations, EA is a sidelined process or a last thought. It is hard to be effective in that case. In other organizations, EA is a core part of IT planning and delivery. It is difficult to imagine EA having anything less than a pivotal role there.
The benefits of an Enterprise Architecture program are clear:
The longer term benefits are the really compelling part:
Sounds good, doesn't it. If you are still in the "I'm thinking about it" stage, then consider adding a step, about a year or two in, that looks at whether you are having the effect you should be having.
If your program is established and running, consider a process every 18 months or so to ask the same questions:
Why add this step? Because, as a change agent, you have to, first and foremost, believe your own story. You have to convince others to change by first believing that change is necessary. This is an elemental part of the conversation.
However, if you believe your own story, day in and day out, you are very likely to miss one aspect of critical thinking: self reflection. So, you need to force yourself, and your team, to self-reflect. Otherwise, you may think you have the best program in the world, only to wake up one day to find your program marginalized, reorganized, or dispersed.
If EA is a new bird at your company, or has been tried before with limited success, you must know that your 'hall pass' has an expiration date. You cannot expect to spend a decade 'experimenting' with changes to the processes needed to get EA off the ground. You don't have that long.
Focus on some key areas:
I know that this advice is general, but everyone's problems are different. At the end of the day, a truly effective EA program, added to an average IT organization, should ROCK THE WORLD. It can happen gradually. It can even happen in fits and starts. But know this: if you are running an effective EA program, the impact will be huge.
What is your impact? If it isn't huge, in terms of the kinds of systems you have or will have, it's because you aren't being effective.
Sometimes, the architect is not really relevant.
For architectural purists, this is a shocking thing to say. To them, I'd say that teams can choose to ignore their architect. In fact, in some situations, they are REQUIRED to ignore their architect. (case in point to follow). If that is the case, then creating a model, or delivering a review, is pretty pointless.
When are teams required to ignore an architect?
I've come across two situations. Perhaps there are more.
1. When the process used to select projects, fund projects, collect requirements, etc, has no mechanism for involving an architect.
If an architect delivers a model in this environment, it MUST be ignored because there is no team that is expecting to receive the model, or trained to understand it, or governed by alignment with it.
2. When the architect has stated goals that conflict with the desires and preconcieved notions of powerful people. If a person is used to funding IT projects without regard to anyone else's wishes, and he finds out that an architect is saying things like "build for the enterprise" or "create fewer, simpler, more consistent solutions," he can simply refuse to cooperate with the architect. The architect is outside the meeting loop, outside the process.
Delivering a model in this environment is pointless, because the team dynamics are oriented to listen only to the "customer" even though the money comes from the corporation, and the customer may not represent the corporate interests.
In both cases, the folks involved in development are not being mean or malicious by ignoring their architect. They are following the 'rules' that exist in the environment. They have no choice.
So if you find yourself in this situation, don't create models. You will be wasting your valuable time. Work on the real problem. Find the support you need to connect the process up, so that architecture has ties into project funding, and a recognized voice in project delivery.
Architecture makes lousy (and expensive) shelf-ware.
Udi Dahan posted an interesting reply to a recent posting of mine. In my post, I go into detail to present a scenario where two services are coupled because the business itself is coupled. He disagreed with my design and offered an alternative. I will discuss his alternative and show that our designs are similar but that mine is more stable and more appropriate to the specific example I described.
I'll see if I can add real diagrams tomorrow, when I'm on my own PC. Right now, we will have to suffice with 'text diagrams'.[addenda: I did my best to create diagrams. The large one may get cut off in your browser. Open it in a seperate window to see all of it.]
From what I can tell of Udi's model, it looks like this:
-- writes --> [co-op master db]
-- subscribes --> [partner change events] -- generated by --> [partner-master service]
-- writes --> [local partner data cache]
-- calls async --> [insert-partner-master]
-- notifies --> [original caller]
-- writes --> [partner master db]
-- publishes //change events// --> to all subscribers
My model was a bit different. Here's mine in the same, goofy, notation.
[composite orchestration: create-co-op-partner]
-- calls sync --> [partner-master] (retrieve partner id)
-- calls sync --> [add-to-co-op-master] (retrieve co-op id)
-- notifies --> [original caller]
Addenda: I created this diagram to show my viewpoint, in context of some callers. you may need to open this is a seperate window to see all of it.
Some interesting comparisons: my message exchange pattern (MEP) is less reliant on async calls. The assumption I made is that the orchestration itself is reliable, so if it cannot call one of the downstream services, the orchestration engine retries later (perhaps dehydrating as needed). This probably lowers scalability. On the other hand, my orchestration has two advantages: first off, it is both simpler to build and, in the 99% sunny-day flow, it performs far better.
So there are architectural tradeoffs between the two designs: Udi wins for scalability, while I get performance.
What is the cost of the scalability? Udi's design is far more complex and thus more expensive to build and own. Do we get so many new co-op partners every day that we need the added cost of Udi's design? I doubt it. Perhaps if the example were dealing with orders, but it isn't. It is dealing with co-op partner agreements... negotiated legal documents. Even very large companies may create a handful of these in a month. So the added complexity (and cost) produces no return on investment.
The most important difference, however, is not the use of sync or async services. In fact, both models assume that the orchestration lives in an async container, and if you started with my model, it would be a trivial change to move to async services and pub-sub. So, while I can chide Udi's design on the basis of cost, that isn't my disagreement with it. In fact, I quite like the notions of publish-subscribe and local distributed data cache. However, his model is not "elegant" in my opinion.
The source of my discomfort is the coupling. In my model, the 'create-co-op-partner' service performs ONLY orchestration. It makes no attempt to call a local database or store cache records. It calls only other services. This allows the fine grained services to be called directly by other consuming applications. In effect, my model allows the business process to be encapsulated and seperated from the fine-grained services that are called by it.
Udi binds them tightly together. In his model, a change to the business process affects all systems that call 'create co-op partner' while in my model, any systems that are consuming the fine-grained services would not be affected by the change. These are three different things: two fine grained services and one (composite) process service. Tying the process service to one of the data services just doesn't feel right to me.
Which one is better?
That's not an easy question. I sat at my desk for an hour before coming up with a situation where my model is better, but I don't think it is all that common of a situation. I will say that generically, I believe that decoupling these three things from one another feels better. That said: the best I could do to prove it is an odd case. here goes:
let's say that we are implementing the following process in our orchestration (both models have an orchestration. same process for both): 'Create-co-op-partner' service is called and passed a data document that describes a co-op partner. There is no 'master partner id' so we search against the master partner service (or local partner cache) to see if the partner already exists. It does, so we get the partner id. We then create the co-op partner with the partner id we found.
Along comes a change to the requirements. (no... that NEVER happens ;-).
Our business wants another business process for the 'spy toys' division. In this process, 'create-co-op-partner' will get a data document that describes a co-op partner. The difference is that in this model, we don't search for the partner first. If no master partner id arrives on the data document, we always create a new partner first, and then create the co-op partner record. Two different processes: two databases that need to be coordinated.
With my model, you simple create a new composite service that performs the new orchestration, call both fine-grained services, and move on. No changes to the existing services. No regression testing.
With Udi's model, you have two choices: either change an existing service to support both processes (and incur regression testing costs) or you create a new service that performs the new orchestration rules as well as performs fine-grained database work to add the co-op partner.
Let's say we have Udi's model in production and we assign this new requirement to Tom, our very cool support developer. Of course, our good support programmer choses to create a new service. He doesn't want to regression test thousands of lines of code that he didn't change. So he copies the existing service, changes the process code, and puts it out on the test server. Of course, Tom would also realize that he has copied the code for the fine-grained database work to two places. That code is not different between the services, but it could get there. Fixing a bug would have to happen twice. Bad. So Tom promptly creates a fine-grained service that both orchestration services will call and refactors out the common code. Viola' Udi's model just migrated and morphed into mine.
So with all due respect to an excellent architect, I say this: just as water seeks a level, design seeks stability. If you build a design that, when kicked with a change, immediately folds to another design, the second design may be more stable than the first. Why not start there?
The sync/async point is not meaningful. It is a tradeoff. I maintain that I made the more appropriate one given the tangible example at hand.
What do you do if two enterprise services share the same database?
I am running into this all the time. As we work to break apart legacy applications, we need to recognize that 'stovepipe' applications are written from the perspective of 'put everything we need in the local database.' That means that we can get really lazy, and require the database to do some things that it does well, like referential integrity, but which complicate integration.
I'll walk through the logic of breaking this apart by way of an example.
Example: if our company (Contoso) stores a list of all customers in Dynamics CRM, but we have an order entry system that is a custom app running in our Extranet. The app is used by our field sales team to enter customer orders. Let's say that we successfully work out the problems of getting basic domain data to the extranet. So we share the same lists of countries, or states/provinces/locales... data that changes rarely.
However, our legacy app was used to not only writing customers to the local database, but also reading customers from the local database. So we built in operations, like the following:
Joe is a sales rep for Oregon and Idaho. When Joe logs in to the legacy order entry system, we look up his region and display a list of customers in the region.
That assumes that we HAVE a list of customers for his region.
Clearly, the list of customers belongs to another service. We have written a 'customer lookup' service and a 'get-customer-details' service that connect to Dynamics CRM. So how, when changing our legacy app, do we break these up... or do we?
There are two design choices to consider here:
1) How much data from another domain is REQUIRED to maintain local functionality? (Correlary: is local functionality actually required or helpful?)
2) What is the cost and complexity of acquiring dynamically-changing non-local domain data?
Question 1 is really a requirements in business question, but it leads straight to a technical challenge. Let's say that the business wants to keep the following function: Joe logs in and sees customers from his territory.
Let's say that they also want to add the following capability: Joe can look up a customer from anyone else's territory and either create a local subsidiary or book a sale to the "non local" company (commission rules will apply later). He CANNOT create a new customer. (business rule).
So, in order to look up companies local to Joe, we have to know a couple of things:
a) For every salesman, what territories to they have? [[note: This information is probably also managed in the CRM solution or perhaps another system. It shouldn't be managed in the order entry system, but it is possible. Let's say that in the past, it was managed locally, but we want to move it to using the data out of CRM.]]
b) For every customer, what territory are they in?
c) What customers are subsidiaries of other customers?
d) We want to find any customer by name (text search).
e) What customer data needs to appear on the actual order itself?
f) What customer data is needed to allow or disallow specific products, or marketing programs, from the order?
I would suggest that the local data that we need to accomplish this is a subset of customer information. Not all the customer data is really needed to find a customer. I would break up the list above into two use cases: find-customer and enter-order.
For find-customer, I'd want local data that includes the names of customers, their customer id, their territory id, a parent customer id (for subsidiaries) and very little else.
In order to get this information, we could set up an event driven master data management pattern. When the CRM system updates a customer, it sends an event to an event handler running in Biztalk or some other free-standing component. (If your CRM system cannot send an event, then have your CRM system update a seperate table in a SQL Server database, and then wire up SQL Server Notification Services to detect the change and send the event... about 200 lines of XML).
Once the event handler gets the notification that data has changed, it passes it along to the subscribers. In this case, a web service running on top of our order entry system. That web service asks the CRM system for details about the customer: customer id, territory id, parent id, and customer name. Once it gets this data, it stores it locally. Nothing else. We have our data feed.
When Joe logs in, he can see a list of all customers for his territory because we can look locally for the list. He can even see other customers, subsidiaries, etc. Now, if he wants to search for companies by a field that is not in our local database, say number of employees, then our local app would call a service on the CRM system. That system would return a list containing (you guessed it) customer id, territory id, parent id, and customer name.
When Joe selects a company from the list, the app looks to the CRM system for actual customer details needed for the order header and for rules enforcement. There is no point in storing this data locally for every customer, although it may be OK to store it locally for this customer now that we have it, along with a 'cache date' so that the local system can use local data when it is not too old, and look up remote data when the local data is 'old enough' (configurable).
What about creating a subsidiary? The order entry system will need to allow Joe to enter data about the subsidiary, and to pick the parent company from the list of customers (as above). Then, it can either use a synchronous call to the CRM solution to create the subsidiary first, copying the customer id (created remotely) to the local database, or the app can create the subsidiary locally (using a temporary primary key) and pass an async call to an event handler to add the subsidiary to CRM. That event handler can then call back to the local system with the official customer id.
The cost of decoupling the databases, and removing a data feed from one system to another is the cost of setting up this Master Data Management mechanism that informs subscriber systems when a publisher system has changed data.
Note that once you set it up, it can be reused by any number of publishers and subscribers.
The key here is to get the remotely mastered data out of your local database. Let the services bring data to you. Don't go get it with SQL jobs. This allows you tremendous flexibility. Services can keep a solid interface without regard to underlying changes. You can move your CRM data from one CRM system to another without breaking connections to order entry, call center, billing, and many other integrated functions.
The downstream savings in integration rewrites completely pays to keep these services decoupled and to use events and messages to move data around, as opposed to direct database feeds.
I hope this discussion helps. Any questions?
Recently, I blogged that two coupled services should have declared, visible, and open coupling. I was promptly asked how.
First off, when you have two services, why would they be coupled? Isn't the POINT that your services are decoupled? Sure. That's the point. But the business doesn't always work that way. Sometimes, your services are coupled because the business is coupled.
Here's an example.
Let's say that our company (Contoso) has a couple of co-op marketing programs. This kind of program is fairly common these days. You make an arrangement with a business partner. They include a reference to your product in their advertising, and you provide some money to pay for the ad. A good example that most folks will recognize: Intel has a co-op marketing program. Any ad that specifically mentions that a machine has an Intel chip in it, and displays the Intel logo, earns a reimbursement from Intel for the costs of producing and running the ad. As a result, we all seen (and heard) the "Intel Inside" brand quite frequently. The program works.
Now, let's look at this from a business side. I'm going to pick on two closely related use cases. Signing up to be a partner of any kind, and signing up to be a member of the co-op partner programs.
Contoso has lots of relationships with partners. We have partners in the supply chain, in manufacturing, as well as in distribution and retail. Each of our partners needs to be uniquely identified, so we have a system where we record the basic information about a partner: legal entity name, DUNS number, Government Tax Id, Street Address, and other information needed to correctly identify this company. Attached to this record, we may have information about contacts, directors/owners, and people that we recognize as contract signatories.
Seperately, we will have a system that records the information specific to the co-op programs. It will contain the list of programs, eligibility rules, claim constraints, claim history, and measures of visibility and effectiveness of the co-op programs. The partner identified in this system has to also exist in the core partner system, but not the other way around. In other words, in order to be eligible, you have to be a partner in some way, but not all partners are eligible. The supplier of cardboard boxes should not be able to claim co-op marketing funds for our product, for example.
So, what kind of service is "create co-op partner"? First off, it is a composite service. It needs to orchestrate between two lower-level services. For the sake of argument, I'll call the lower level services "partner-master" and "add-to-co-op-master".
If I send data to a service that creates a co-op partner, and I don't know the id in the partner master system, then I need to look the partner up, and if they are not found, create them in the partner master system, then take their partner id and add them to the co-op system. I may not want to do any of that if they are not eligible to join the co-op system.
Adding data to the co-op system itself is done in the 'add-to-co-op-master' service. This service simply requires the existence of a 'partner master' id. Of course, to be defensive, it will need to check to see if that id is valid or have some other way to trust the caller.
And here's where we get to the notion of a declared coupling. The 'create co-op partner' service needs to have a way to declare, even if it is just in text, that it is responsible for checking the 'partner-master' service, looking for duplicates and creating the partner if necessary. There needs to be data that shows the coupling of this composite service. In the best of all possible worlds, This data would be available in the contract header information (although I don't know of any standard way to do that).
In addition, there needs to be a way for the 'add-to-co-op-master' service to trust the partner id that comes in on a call, without actually calling the 'partner-master' service. Perhaps it will only accept calls that have an ID if the call comes from a known composite service or from a known IP address? Perhaps digital signatures are required? Trust has to be established.
It is important that the fine-grained 'add-to-co-op-master' service NOT directly call the 'partner-master' service. This would be unnecessary coupling.
So, recap, the rules to making this coupling open and visible:
-- In the header of any composite service, declare the child services that will be called, with sufficient reference data for the system that is asking for data to drill down and find out about the called service.
-- If a service has data reference requirements, it can limit the list of eligible callers to a known collaborator (like a known composite service). It should not call other fine-grained services to validate inbound data.
-- The only services that can call other services are composites, and composites can only call other services in a rules-defined order. They should not perform their own data storage or data validation operations, relying only on underlying services to provide these capabilities.
It's not dissimilar to the notions of 'good practice' that evolved out of object-oriented programming when it started to get off the ground. Things like naming rules and style considerations, all of which were designed to make code easier to read and use. The same goes for service coupling. If we follow these basic rules, we can minimize coupling and keep it contained in an open and controlled location.
In response to a question about application-level authentication, I am updating this post to include a link to an MSDN article that discusses this aspect in rich detail. I highly recommend this 'Patterns and Practices' article.