Inside Architecture

Notes on Enterprise Architecture, Business Alignment, Interesting Trends, and anything else that interests me this week...

July, 2007

  • Inside Architecture

    Free Code - Getting IT out of the Applications business


    There is one big thing we must do if we are to make IT align with business strategy, we need to get IT out of the role of interpreting the whims and desires of the business.  The good folks in IT are really bad at mind-reading.  As long as we are in the "mind-reading" business, we will never be given credit for what we do well: automation.

    The answer: let the business folks write free code.  Not just any business folks.  We let Business Process Developers write free code.

    What is free code?  Free code is unmaintainable code that wires together service calls in a way that is inexpensive to produce.  Free code is mashup code.  Bugs can be fixed, but we don't really maintain it.  If we want to change free code, we write it again.  It was so inexpensive to build that it costs less to rewrite than to modify in any non-trivial way.

    Free code, in order to be truly free, needs to be generated from tools that are NOT coding tools.  In other words, software development environments are too rich for free code.  Why?  Because it is too tempting to build expensive code.   We need to differentiate, then, between the rich, highly designed, object oriented code that software developers produce, and the free code that business process developers will produce.

    Note: I said that free code is unmaintainable.  Code is unmaintainable because it's complexity exceeds the ability of a developer to maintain it. Let's dig a little deeper.  Why do we need to maintain code?  Because code is expensive to write.  Therefore, it is currently cheaper to fix it than rewrite it.  On the other hand, what if code were cheap, or free?  What if it were cheaper to write it than maintain it? 

     Then we would never maintain it.  We'd write it from scratch every time. 

    Sure, we can choose to write maintainable code.  We can use practices like patterns, object oriented development, and careful design principles.  On the other hand, we can give our business project managers an environment where they can describe their needs and code is simply expressed from those needs.  If the code that comes out doesn't meet their needs, the business process developer knows it the moment they run their code. 

    What is the value of doing this?

    1) Lower the cost of IT through reduced skill requirements.  The skill set of the Business Process Developer is different from that of a software developer.  Traditionally, we've sought folks with both skill sets to employ as software analysts.  This usually meant training someone.  What is wrong wit that?  Answer: We've created expensive specialists to overcome tool deficiencies.  Why not fix the tools?  Then we won't need the specialists that cost so darn much.

    2) The speed of development goes up.  If the business process developer can change the process wiring readily, then the software developer can focus on making the needed updates to the services themselves.  This removes the coupling between process and code that slows down EVERY project in IT.   

    3) Projects become more agile.  Since a business process developer can develop a mashup of services quickly, they can demonstrate that mashup very readily, directly to business stakeholders.  A change can be shown to the business folks quickly as well.  If the business needs change, or their understanding grows, and they need the services to do something more than they do, then this kind of agile process encourages rapid feedback to the developers who own the services themselves. 

    4) Solution quality goes up.  Since we can focus our deep design team on developing the services that the business process developers consume, we can improve the quality of those services independently.  This allows for better measurement of quality and an increased focus on the key quality measures inside each service.  Reusability is a natural outcome of high quality services.

    What does this mean for our tools:

    We need to seperate business process modeling from software development and produce rich tools aimed at the needs of the BPM practitioner.  Those tools need to start and end with an understanding of business capabilities, tied through to business processes, and down to events and business documents against a common information model. 

    We need our tools to reduce the leaky abstractions that we currently call 'services' by helping developers build services that are very simple to consume by the business process developers.  We need to capture these requirements and act on them through automated mechanisms built in to both the BPM environment and the IDE.

    What does this mean for our processes:

    The good folks in IT need to formally and officially take control of managing the common enterprise information model and the business event ontology.  If a business wants to change the data and event models, they need to work through a published process that allows and encourages consensus. 

    The good folks in IT need to formally allow business process developers to easily develop, test, and deploy their processes.  Deployment is a problem because IT folks normally just 'fudge' their way through deployment processes.  If we are going to let business process folks to write code that we deploy, then it needs to be very simple to deploy that code.

    Free code makes sense...  We need to align IT to business, and this is one very useful mechanism to do it.  It is time to stop getting in each other's hair.

  • Inside Architecture

    What you need to make middle out SOA architecture work


    In middle-out SOA, we want to do as little as possible from the "center."  The real value is at the edge, where the services are being created and where value actually lies. 

    Our goal, in middle out architecture, is to set up standards, mechanisms, and protocols that everyone will follow.  We will give names to services that need to be created, and we will describe a small set of standard message exchange patterns, complete with the names to be used at the endpoint.  Where possible, we will specify transport mechanisms as well.

    Important: in Middle Out SOA, the central group writes NO code.

    Let me give you an example.  In the metropolitan area around Seattle, there are quite a few suburban cities.  East of Seattle, across Lake Washington, is the city of Bellevue.  Next to it are the cities of Redmond and Kirkland.  These cities each began from their own centerpoint and grew outward until they collided in the same area.  The Microsoft campus is on a 'spur' of Redmond that is surrounded almost entirely with the city of Bellevue. 

    Bellevue intersection with Redmond Washington


    If you look at the map, you will see that there are PLENTY of roads that are either borders between the cities or cross from one city to the other.  As a driver who drives in this area, I can tell you that the roads have the same name on both sides of the city lines.  Normally, you do not know which side of the line you are on, or where the transition happened. 

    Why is it so seamless?  It is not overtly controlled.  The cities are quite distinct from one another.  Rememer, they grew from their own centers out.  So why is it, when they intersected, the arterials lined up and the names are standard and the roads don't change width or designation? 

    It is because the planning departments in these cities all work together on standards.  There is a standard numbering scheme for the entire of King county (5,974 square km) that allows all streets, even ones in rural areas, to have names and numbers that don't change from city to city.  The roads have a standard width, and roads of a particular arterial classification have standards for sidewalks and traffic lights.  I can find a business by its address, regardless of what side of the street that business is located on.

    My car is a data packet that travels over these roads, and I have the routing mechanism in my brain, deciding the route and alternatives as needed to find my destination address.  It works because those roads and addresses are consistent and integrated.

    The coordination between the two cities is practical and useful.  Each city has an ordinance requiring its planning department to cooperate with other cities.  That is important.  The ordinance is a fixed decision, made by their governing bodies, that gives their planning boards support and direction.  Without the ordinance requiring cooperation, passed by each city, the roads would not work.

    This is the fundamental idea behind middle-out architecture.  Know the MINIMUM set of standards that everyone has to agree on, and get the management to buy in to requiring that minimum amount of cooperation.  Then cooperate.  Each area is responsible for their own governance. 

    It is an inherently distributed mechanism, and therefore scales well.  Note that the minimum set useful in an enterprise is larger than the minimum set useful across the internet as a whole (at this time).  Mashups and SaaS services will not have a consistent mechanism for sharing data... at least not until customers demand it.  On the other hand, in an organization, consistency at the data level makes more sense and therefore a common event and schema model can be expected and used.

    Middle out is not new.  It is a way of describing what works in other areas.  It is only new to SOA because SOA is new to IT.  We are still learning what other folks already have learned.  We are still catching up.

  • Inside Architecture

    Bottom-up SOA is harmful and should be discouraged


    It's independence day week in the USA, so it is time for me to declare a little independence. 

    There are three schools of thought around "how to build an Enterprise Service Oriented Architecture."  They are:

    1. Top down - central group decides everything and the dev teams adopt them.
    2. Bottom up - central group provides a directory and dev teams make whatever services they want.  Dev teams go to the directory to find services they can use.
    3. Middle out - central group provides key elements of the interface, including numbering schemes, message exchange patterns, standard communication mechanisms, and monitoring infrastructure, and encourages the dev teams to use it to build services that can be shared.

    Top down works only in some organizations.  I imagine smaller IT shops (with 300 or fewer total employees) and shops with a well established heirarchy or very strong central leadership can leverage this one.  (I'm conjecturing, of course, having not worked in SOA in that setting.  It's an educated guess).  For everyone else, it is slow and difficult to follow the Top Down path. 

    Bottom up is an answer to top down seen as "the wisdom of crowds" or "economics in IT."  The idea being that many services can be developed without much central control, as long as they are in a directory.  Everyone would look in the directory to find the services they need and adopt the best or least expensive, and we'd get SOA adoption.  This one has the advantage of scale.  The larger the organization, the argument goes, the better the case for bottom up. 

    Middle out is the newest method developed by folks who prefer a blended approach.  I won't go into a lot of details here.

    The point I want to make today: Top down is slow, but functional.  You can end up with a SOA that is good for the enterprise.  Middle out has the same advantages and is a bit quicker. 

    Bottom up is actively harmful to the enterprise as a whole and should be discouraged at all costs.

    Of course, a lot of the literature in the SOA field says "build the catalog first," effectively supporting the bottom-up approach.  And that is a mistake.

    As I've said before, I am not paid to solve a particular system problem.  I'm paid to look out for the needs of the enterprise, to create a "gravity" towards the center to counterbalance the natural tendency of IT to develop only for the individual needs, even when it works against the strategic goals of the enterprise.

    Bottom up is a mess.  Let's look at the logical conclusion of bottom-up for a minute and compare what results we get with some very common enterprise strategic goals.

    Let's say our goals look like this.  This is a very simple set.

    1. Share a common concept of customer, order, sale, shipment, and inventory so that we can get a straight answer at the end of the day about the operational and financial health of the business.
    2. Couple loosly so that we can keep the costs of maintenance as low as possible and hopefully, over time, divert some of those expensive maintenance resources over to developing new programs. 
    3. Build your applications so that other folks can easily integrate with them so that we can add new capabilities to the IT ecosystem quickly. 

    On first blush, SOA supports all three. It is clear that you can get common data, loose coupling, and ready integration from SOA, so a lot of folks have cried "SOA is the answer!" and then stopped.  They didn't pick the path to take, so the debate rages on: top-down, bottom-up, or middle-out.

    The problem with bottom up is that you end up with systemic effects, not seen by individual contributors, that work against these same strategies.

    There are three ways that bottom-up SOA can go in your organization.  One, it is ignored (no one decides to participate).  Two, it is wildly adopted with some folks contributing and others consuming, and Three, it is adopted only be people who have services they want to advertise, but no one consumes them out of fear of taking a dependency.  (There is a common word for both One and Three: "failure".)  Let's look at Two: some contributors and some adopters.

    What are the folks contributing?  Well if we are creating a system, but we are not paid to think about the enterprise, then the system we are creating may, but probably won't, deliver capabilities that the enterprise needs.  The service will be 'local-specific'.  That means that the service will serve the needs of the paying customer (a business leader) but not the enterprise as a whole.

    So the services that are in the catalog, most of them anyway, are not really 'enterprise ready.'  That means that when others consume them, they are using identifiers that are not enterprise-ready, or they are using data in a manner that may or may not be usable in the data warehouse. 

    Remember that consuming a service is only demanded by a business process.  If it weren't for the need to change processes, we would never need flexibility in the first place.  If we do 'bottom-up SOA' without considering the business processes, that doesn't mean they aren't there.  It means we are ignoring them.  It is in some folks best interest that we don't look.  Consuming a service "automates and hardens" a business process, making it more difficult to change.  What if we are hardening really bad process?  Better, what if, in different parts of the company, we are hardening different processes? 

    In the past, I had a run-in with very strong-willed and opinionated leaders, both with IT and the business, who strongly defended their absolute right to "do things their own way."  Sometimes that's healthy, but in the cases where it is not, it can become a shouting match.  Bottom-up SOA avoids the fight because if the service is attached in the "right way" you use it, but otherwise, you write your own.  Fight avoided.  However, when a business leader wants to take a basic, core, fundamental process, and go their own way, that is not a fight you should avoid. 

    According to some fairly compelling research and literation in the field of business process automation, the most successful companies have automated their core business processes, which provides less flexibility, but gives them far greater agility when it comes to taking on new challenges.  The core stuff, the stuff they do every day, is routine, standard, and well thought out. It is a stable foundation, allowing the company to stretch in a dozen different ways. 

    "Bottom up" is simply a mechanism to allow those strong-willed leaders to create multiple infrastructures in the company "because they want to" without any consideration for creating a common and stable foundation for the business.

    So lets look at a bottom-up success story three years later.  A number of infrastructures have emerged.  You have one based on Division A's CRM system.  You have another that feeds Division B's ERP system.  You have a third in Division C to surround their mainframe Operations management system, and a fourth in Division D focused on their home-grown web-order management infrastructure.

    You have four legacy systems.  Integrating them will require a slow, painful, expensive infrastructure, in the middle, with huge amounts of semantics and rules needed to translate the data from one legacy infrastructure to another.  That central system will be necessary to support Business Intelligence and Financial Reporting (so you cannot avoid it) and it will limit the ability of the company to respond to the needs of the marketplace.

    You got SOA, but at a price. 

    Of our three principles above:

    1. We have our common understanding of customer, but it is so far away from the systems that create that understanding that it is useless for generating real value. 
    2. We have our loose coupling, within each ecosystem, but each one is at war with one another, vieing for resources and investment.  The money we saved by developing the individual infrastructures has moved to the central integration hubs and BI.  No win there. 
    3. We've got easy integration, as long as the system that needs to consume the data has the same semantic understanding of the data as the system that produces it.  Otherwise, consumer and producer cannot work together, and the cost of forcing an integration is high.

    We will have won the battle, but lost the war.

    This is not a scenario that I would wish on anyone (except perhaps my direct competition).  To be honest, I think my competition is smart enough to see this for themselves.

    Bottom-up SOA may work if your measurement is "get services up an running" (or it may fail).  If it works, it produces a monster that is not worth producing.  That is not success either.  Measuring SOA success by the number of services, or encouraging the creation of services without any form of central understanding of what the services should contain, is doomed.

    Avoid Bottom-up.  It is a pox on your house.

  • Inside Architecture

    Applications ARE the leaky abstractions


    Recently, a message called my attention to a classic article from Joel on Software on the Law of Leaky Abstractions.  It's a fun article and I recommend it heartily.  In this article, Joel argues that most techologies are designed to abstract something more complex under them, but nearly all abstractions are "leaky" in that they fail to completely hide the complexity underneath. 

    Therefore, you can teach a software developer student about the "higher level" stuff, but sooner or later, they will need to know more about the underlying technology to cope with some unexpected or misunderstood effect caused by a leaky abstraction.

    As a result, we cannot forget the underlying technology and "move up" to the new abstraction every time a new library or language appears.  We have to continuously cram more knowledge into our brains.  This makes all software development increasingly complex. 

    This little reminder of Joel's classic article came from a developer who complained that software can never be truly disposable, and can never be free, because software is the first, last, and only place where you can document the business process.  Every place else, including process diagrams and documents, are leaky abstractions.

    I believe that notion is backwards.  I believe that software is a leaky abstraction of the business process.

    Software, to our end users, is there to solve a problem.  The problem is caused by the business needed to implement a business process.  Software automates a portion of that business process.  In doing so, software must abstract the process details away from the users, hiding complexity.  But if the process is complex, or if the software is not able to abstract the process well, it is very likely that the complexity of the business process will "leak through" the abstraction.

    In other words, using code as documentation of a business process is a leaky abstraction. 

    This is just one of the reasons why I do not believe that code is a good place to document business processes.

    1. As explained above, software is a leaky abstraction of the business process, which nearly always starts with, involves, and ends with human beings accomplishing a business task. 
    2. Code is not accessable to most of the people who actually need the process to change to meet the needs of the business.  Especially Object Oriented code, where large portions of the functional bits are hidden in base classes or where a lot of code is spent dealing with Object Relational Impedence.  Therefore, the cost of changing a business process increases dramatically due to the need for specialized skills to understand the existing automation.
    3. Most software modeling technologies focus their strengths on the static and structural elements of software (class diagrams and deployment diagrams) while the business folks want to deal with dynamic diagrams (state diagrams and transition diagrams).  As a result, we've refined a language that the business doesn't speak.  When you reverse engineer a diagram from code, you usually only get the unhelpful (to the business) static diagrams. 
    4. Code is an indirect representation of the business process steps.  A software developer, or team of developers, designed a technical solution to a business problem.  There were probably many ways to perform the particular task in technology, yet they chose one.  This narrowing is fine when going from general (business process) to specific (implementation) but when going back to general, as we would need to do when using code to document a business process, we would inherently add value to the choices already made.  This is flawed thinking that constrains business flexibility needlessly.

    You see, my notion of inexpensive software is not just to make it easy on developers.  My notion is to create an abstraction that is so good that it provides over 90% of all business problems a "minimally leaky" abstraction.  Of course, developers (not just product teams within MS, but IT developers in the field) will need to create this abstraction for our business users.

    I believe those abstractions exist in the services spanning layers predicted by Middle Out SOA Architecture.

    Of course, there will always be the need for "very deep" software development to expand the boundaries of technology or to create altogether new abstractions for the business to use.  We don't put developers out of work.  On the contrary, we allow them to lift themselves out of the details and into the language that the business speaks: business process.

    Software is the leaky abstraction.  It makes poor documentation for a business process. 

  • Inside Architecture

    How is Business Process Management related to Service Oriented Architecture?


    I'm at a large training conference this week, doing a presentation on my ideas around Middle-out SOA and abstractions in the center.  I got a rare opportunity: a smart consultant pulled me aside and asked me this question:

    How is Business Process Management related to Service Oriented Architecture?

    The great part about being asked that question is that I had to work to answer it, which tells me that I hadn't spend nearly enough time thinking about the relationship between the two.  The consultant who asked wasn't trying to show me up, nor was he asking for a basic course in SOA.  He had his own ideas, and he wanted a sounding board.

    So we stood at a white board and wrote, and drew, and compared models and described ideas.  What we discovered, first and foremost, was this: we both had the same answer.  We had picked up our words from different places, so they were different words, but once we wrote the models up on the whiteboard, the biggest difference was that I drew the SOA stack horizontally while he drew it vertically. 

    Turns out, when you dig down, these two activities are twin brothers.  Their parents are the same, and they have many similarities, but the serve different needs in the enterprise.

    • Business Process Management is the organization of business capabilities (people, process, technology, and data) so that common mechanisms can be discovered, simplified, and improved.  The result: a simpler, more efficient and more rational portfolio of processes.  This reduces cost and increases business agility.
    • Service Oriented Architecture is the organization of technical capabilities (activities, events, documents, and data) so that common services can be discovered, simplified, and improved.  The result: a simpler, more efficient and more rational portfolio of services.  This reduces cost and increases business agility.

    The similarities are striking because these twin brothers have the same father, data, and the same mother, the business event.  To understand how they relate is to understand their parents.

    First, the father: the common data model  

    As I have blogged before, we need very little at the "center" in order to make Enterprise Architecture work.  Probably the most important thing we need is a common data model, so that we all know what data we are working with.  Doesn't have to be a big data model.  On the contrary, the common data model should be as slender as possible

    The common data model should contain ONLY enough information to define the things that must be present to define a business document, to identify the unique key for that document, and to define the relationships with other business documents.  It can be developed over time, with some parts vague (just relationships between data areas) while other parts are tangible (thus allowing it to be developed iteratively, as needed by projects).

    To extend this 'family' analogy a bit further: The brother of the data model, and the uncle of our twins, is the business document.

    A business document is the collection of data items that are needed to characterize an object used by the business to support the business capabilities.  These are the things the business acts upon: products, services, programs, invoices, orders, shipments, payments, and many more.  Every ERP system vendor understands the concept of the document.  It is at the heart of what they manage.

    And then the mother: the business event ontology 

    And just as the father of the twins is data, the mother is the business event.

    A business event is an event that defines the boundaries between business processes.  At the instant that a business event occurs, a business document changes state. 

    If you truly want to create a rational heirarchy of business processes, and you want to look for both commonalities and variations, you have to know where a process starts and where it ends.  Otherwise, you could find yourself comparing a single apple to a grove of apple trees. 

    To compare two processes, they must both start with the same starting event and traverse to the same ending event.  We do this in our language when we say "Let's look at the Order to Cash process" meaning "Let's look at the process that begins with the creation of the order (event) and ends with recognition of revenue (event)."  You need the ontology of events.  This is often left out of literature on Business Process Management, because you can diagram a process that spans "from anywhere to anywhere."  That's great, if you want to create a diagram.  But if you want to capture knowledge, and you want to rationalize or simplify processes from around the company, you need these events as your boundaries. 

    Marriage of the two

    Before you manage the processes, and before you design your services, you need these parents.  They have to have time to grow up and become mature, or their children will have no direction, so get consensus that the data entities are correct and their relationships are correct.  Get consensus that the heirarchy of events is correct and the names of the business documents are consistent.  There is no point in doing BPM if you don't know what your data is, and you don't know what the events are.  The same holds for SOA. 

    This is not a top-down design, but it is a top-down constraint.  If you build your SOA before you understand the data, you are creating multiple versions of the truth.  If you attempt BPM before you identify your events, you will burn huge efforts to compare apples to apple trees, producing no value.

    Now, let's all sit together and share a meal

    Like all brothers, SOA and BPM don't always get along.  They will have different viewpoints of both data and business events.  These two activities are usually performed by different people, in different parts of the organization.  Frequently, the are not connected in any specific way.  Their managers may be different, and their measurable metrics may be different.  Sometimes, they don't even see how closely related they are.  Yet, there is a link, like the link between twin brothers.  An obstacle for one is an obstacle for both, and victory for one directly benefits the other.

    Who works to get these two brothers to agree?  The Enterprise Architect does. 

    Through both SOA and BPM, we get exceptional value for the enterprise.  Understanding how they are related helps to build the common shared understanding that these two activities need to truly succeed.

  • Inside Architecture

    What are the requirements for enterprise event security?


    In a SOA, to be effective, we need to share both data and events.  Events, as I have discussed before, are the messages that applications send to the common infrastructure to inform subscribers that well-understood business events have occurred.  This includes things like "a sale was made" and "a product was shipped."  It may also include things like "an employee has ceased working for the company."

    The last example above, an employee quits or is fired, is interesting.  From the employee's standpoint, it may be a private event.  If they were fired, it may be quite embarrassing, not something they want broadcast.  On the other hand, from the viewpoint of the company, it could be of very high importance that the applications in IT be immediately notified of the employee's dismissal, so that all permissions can be revoked, and workflow processes involving that person can be redirected to another person or team to accomplish.

    Revocation of credentials: we've got Active Directory.   That one is fairly straight forward for a Microsoft-based IT team.

    Changing workflow: most folks screw this up.  I've seen my share of workflows.  It is rare, to say the least, to find a workflow system where the business told IT to prepare for the day when their employees would be fired.  Mature systems usually add the event handling somewhere... sometimes it is entirely manual, occasionally requiring a data support analyst to write a SQL Query to reassign tasks from one person to another. That's a mess.

    So clearly, we need to share the firing of an employee as a broadcast event in the IT infrastructure.  The question is this: do we secure it?

    The ramifications of placing security models in an EAI space are oddly different than similar situations in an application.  If an app knows that Joe can access Contoso's sales record but Mary cannot, then Mary doesn't see it. Pretty simple.  However, the app knows.  We, as developers, can still build in referential integrity around the fact that the sale was made.  By extension, if an event occurs in another system, this system may need to know about it, even if NONE of the users of this system need to be informed about it.

    It may sound striking, but when you look at it, it makes sense.  Applications have their own identitiy, and their own security.  They are full actors in the system.  An app can see an event.  In fact, I would say that all apps that support the enterprise security model should be allowed to subscribe to ANY event. 

    Of course, I said the key words: apps that support the enterprise security model.  What does that mean?  That means that there is a way for an event to either carry security in the event message or in the subscription process, so that the apps that subscribe to the event respect the right of the enterprise to control who, among the human population, can see the event.  It's a little like saying that the app has signed the corporate NDA and will respect the wishes of the company to keep some data secure, even from its own users.

    In other words, it is OK to allow our apps to see things that we, as human users, cannot see.  Certainly, the support analysts will need the right to investigate exceptions and find data errors.  Other than exception handling and auditing, it is entirely reasonable to restrict events to be viewed by the app, but none of the users of the app.

    So, how do we correctly share and manage event security?  That is the question I'd like to answer.

    If you know of a good answer, please drop me a link.  I'm investigating two avenues:

    1. Including the security policy in the event message itself, and/or
    2. Including the security policy in the subscription, so that the subscriber has a consistent policy to follow for all events.

    If you know of a good answer to this question, please share it.

Page 1 of 4 (20 items) 1234