Architecture for the Real World

[OBA Solution Framework Index]

Let us say that you've decided to build an Enterprise Architecture that supports all five layers of the architecture that I laid out earlier - presentation, collaboration, process, services, and data. Let us next say that you've also picked the Microsoft stack as the technology platform for this, and that you have decided to build an Office Business Application. That's great, but then what next?

Let us assume that you start from a set of monolithic applications as below. You are faced with a number of choices now. Do you start tearing apart your existing applications? Do you start to create new OBAs from scratch? Top-down? Bottom Up? You know that there will be a transformation, but will this be business led or IT led? 

One approach is top down - often taken when transformation is business-driven. Here much time is spent on analysing business processes across the company, doing extensive use-case analysis, and then designing OBAs to match. The focus is on designing applications and services to promote re-use. The risk here is that a lot of time / budget might get spent on analysing as-is business processes, without measurable returns.

In companies where transformation is IT-driven, then a bottom-up approach is often used. Here people talk of SOA, but what gets produced are sets of web services that map directly to existing business applications and processes. The risk here is that no new business capabilities, or value, is being created. 

There is a different way - called 'spiking' - where you nail down a fixed vertical scope across all five architectural layers. This scope should be part of a project to deliver a new cross-functional business process that cannot be easily done with existing applications. Next you follow an iterative approach, and at each iteration you expose core IT assets, you compose a new solution from them, and then you plug this solution into information channels easily accessible to users.

Why is this called 'Spiking'? The goal is to get enough breadth in the process, that the solution provides measurable business value. At the same time, there need to be enough depth so that significant architectual / organizational concerns are addressed at each of the tiers. At a high level, the approach for each iteration (or spike) is to do the following:

1. EXPOSE: standardize core IT assets that make up the solution - e.g. documents that make up the process, business events for that process, data models, performance models, web services for integration to existing Line of Business systems, etc. 
2. COMPOSE: you assemble the new solution by combining different assets. Composition could be at any of the five tiers, e.g. composing web parts in a portal, or composing calls to business services into workflows that orchestrate them. Here you are setting up the server side information architecture (e.g. site hierarchies, document libraries, workflows, connection libraries and business entity definitions).
3. CONSUME: you connect the new solution to information channels that are accessible to those roles that make up the process. This means connecting client applications (web based, thick client, Office Client based) to the server side solution that you assembled.

This approach differs from a top-down approach by leveraging the fixed scope of a 'spike' to ensure quick business value, while still mitigating technology risk. This approach differs from a bottom-up approach by ensuring that back end services are aligned with the way that information is being consumed by specific roles.

[OBA Solution Framework Index]

I'm posting this from Singapore where I did a couple of sessions at their Strategic Architect Forum yesterday on Office Business Applications, composite business solutions, and how to build them. 

Lately a number of folks that I've talked to, have been wresting with the practicalities of building a SOA (Service Oriented Architecture). While SOA holds a lot of potential, there have been a number of large SOA projects where benefits have not been in line with expectations. This is especially true of Big Bang SOA, where large re-engineering projects have been kicked off. InformationWeek in an article here points out some troubling statistics with Big-Bang SOA (e.g. 24% projects short of expectations, 41% projects with cost overruns AND failure to meet expectations).

I think that usually SOA failures arise due to inadequate consideration to how real people will use the system. There is an inordinate emphasis on building 'perfect services' rather than supporting the decision making processes of information users. So lets see here, what a "People-Ready SOA" might look like.

I once met with a large semiconductor company that wanted to introduce a new kind of product to market. In about 18 months they had retooled their manufacturing facilities to make these, and set up new sales & distribution channels. However at that time, their IT systems were still nowhere close to being able to take orders for the new product, plan production and then drive their manufacturing execution systems. They had a familiar problem - legacy business applications that were monolithic - that prevented change rather than enabling it. It was hard for them to build new cross-functional solutions.

The problem is that most applications are architected in isolation, and offer few opportunities for reuse. For example, let us assume that you have systems that look like this.

So you now want to unlock business value through SOA. Usually business justification for this is some combination of the 3 A's (Agility, Alignment, Adaptability) - see paper by Dr. Hau Lee on this topic. For me, this translates to the ability to quickly and easily build new cross-functional solutions to keep up with business needs.

Let us suppose that you now start to build a SOA. The first thing to watch out for, is to avoid the following (very common) mistake. You've replaced applications with services - but you haven't actually improved anything. Although for your troubles, you have introduced more complexity through a new technology stack.

So you might next want to refactor services for reuse, as shown in the next figure below.

This leads to a new kind of problem. No matter how cleanly you design your services, it is hard for business folks to relate to them. Have you ever tried explaining an API, or a WSDL doc, to somebody in the business? Also, you want to separate out things that change frequently from things that don't. In other words you want to externalize the points of variability in the system, so that they can be changed easily without long drawn development cycles.

So you add in a business process management tier next. This is a set of technologies that let you design and orchestrate interactions across services. The advantages of doing this are:

1. Visualization of process - visual representations of business processes that are first class citizens of the development process, rather than dusty documents 
2. Align IT resources to business imperatives - if business processes are made explicit, then it is much easier to align IT resources with business needs 
3. Monitoring - explicit processes enable monitoring of activities, which leads to oportunities for optimization and improvement
4. Enables change - it now becomes much easier to change processes and / or the business rules that drive them. The idea is that changes to core services are made relatively infrequently, and are done incrementally when needed - most changes are made in the process tier (if services and process tiers are designed correctly).

But there is a problem with doing this, as shown below.

Usually BPM tools work best to handle automation of services, processes and systems. However people and computers don't work in the same way. People need an environment that is flexible and dynamic, that can handle unstructured information and documents, and that allow them to easily change the way that they receive and consume information. Computers deal with structured information and processes.

So you need to design an environment that enables collaboration and human decision making. Neither of these come from the services tier, or the process tier - which mans that you have to explicitly account for this in a separate tier as shown next.

What are the capabilities, that need to be delivered by this collaboration tier? Some of the capabilities provided by SharePoint - which is a good fit for the collaboration tier are as follows:

1. Self service content management and BI - ability for users to create, store and share information e.g. documents, lists, reports, wikis, blogs, etc. with versioning and security.
2. Security - ability to partition information by process, role, organizational hierarchy etc. - and set up appropriate access rights.
3. Portal - this provides a customizable / personizable interface for users to publish information, and to search for information. Users have the ability to compose project / role / activity specific pages, sites, and dashboards. Emphasis here is on "employee self-service".
4. Solution services - forms services to easily create and deploy forms, and Excel services to turn spreadsheets into server side components for reporting, calculations etc.
5. Communications - track inbound emails for archival, send outbound emails for alerting / notifications, integration of portal to real time messaging channels.
6. Connections to lower tiers - back end data, services, and structured business processes. These connections are set up by IT, with role specific access controls. Once set up, the connections can be used to access information in the portal, e.g. in reports, documents and dashboards. 

You may notice that I have chosen not to consolidate at the data or presentation tiers. The architecture should allow for federation of data, allow for composition at the presentation tier, and support multiple channels of information consumption.

What are the implications of this architecture? Well firstly, you need to choose a platform that supports this kind of decomposition and assembly. Not coincidentally, the five tiers here match up with the five tiers for an OBA here. So the platform for building an OBA will deliver all elements of a "People Ready SOA".

Once that is done, the architecture will provide the following benefits:

1. Separation of concerns
2. Loose coupling
3. Patterns of reuse
4. Enables composition
5. You don't need to write code for everything

[OBA Solution Framework Index]

Consider composition of information in the Office Client. People are used to thinking of Word, Excel and Outlook as desktop productivity tools, but they can now serve as containers for data coming from back end servers (e.g. ERP and other Line of Business applications). Consider the following diagram.

You can see how Office 2007 client applications are now customizable / extensible containers for business logic. You can use VSTO (Visual Studio Tools for Office) to extend Office client with application specific add-ins. These add-ins can present the user with information and actions in the context of his current task, and in the context of his role. The following figure shows how this might look like.

But if you do consider Office 2007 client applications as containers, then the question remains - what gets deployed into these containers? Now that Office documents are represnted as Open XML, you can drop custom defined XML blocks into documents - as in the figure below.

This means two things for solution developers - firstly that client side application add ins can operate on this custom XML block. And secondly, all kinds of interesting new opportunities have opened up for server side automation that process this custom XML block. You can now do the following:

1. Extract information from back end systems, e.g. by a workflow running in SharePoint that gets this information via the BDC (Business Data Catalog). You could build up a custom workflow activity library for this purpose.
2. You can embed this data back to an Office client document for off-line use. This document can reach its intended user via either a pull model (retrieving it from a SharePoint library), or via a push model (sending it via email). The data could be stored in the Custom XML schema embedded in the Open XML document (i.e. payload).
3. You can use Office applications like Excel, Word, and Outlook, with appropriate add ins to provide client side actions and services that make use of this custom data.

[OBA Solution Framework Index]

What is a composite application? In the simplest sense, it is a cross-functional solution assembled from pre-existing building blocks. These building blocks could be services, workflows and activities, documents, application blocks (e.g. portal web part), rules etc.

What are the components in an OBA that can be assebled into a composite app? These are:

Steve Ballmer took the stage at the Software 2007 Conference to talk about Microsoft's software + services (or S+S) strategy. As part of that presentation, Steve talked about Office Business Applications (i.e. OBAs) as composite applications that allow information workers to use information in back office applications such as ERP in new ways. ZDNet published a report on this two days ago - which is available here.

I'd like to thank Steve and ZDNet for setting up this posting, because on that page, ZDNet has two pictures of Steve, and ALSO an early version of the following diagram showing the internals of an OBA that I had created while releasing the OBA RAP for SCM (Reference Application Pack for Supply Chain Management).

What does this diagram mean? In my posting OSF 5 I talked about the fact that OBAs can deliver solutions that provide business capabilities for transactional needs, decision support needs, and collaboration needs. You can think of these as next generation composite business applications. However in order to deliver solutions that provide these kinds of business capabilities, you need a platform that provides technical / infrastructure capabilities. Those platform capabilities fall into five categories - that correspond to the five tiers that I called out in my posting OSF 2 - presentation, collaboration, process, services and data. Do you need all of these tiers to build a business application? Not for all kinds of applications ... but you if you do want to build a next generation composite business app that combines transactions, decision support and collaboration.

So the diagram above just breaks down the technical capabilities of the Microsoft platform into those five tiers. How does this fit the composite application paradigm? Well each of the boxes in this diagram represents a container, that provides particular application services, and into which content and data can be deployed. The different bits of content and data represent components that make up the composite solution. Components can be assembled into different groupings - each of which is a composite application. Each of the containers in the diagram above can service only particular types of components.

For more information on composite apps, you can check out the MSDN architecture site on composite apps, as well as the Architecture Journal issue on this topic. For an article on OBA and composite applications, you can check out an earlier article of mine in the Journal.

In later blog postings I will go into more details on each of those boxes, and show an end-to-end example. A list of links to the composition abilities of those various boxes are provided here - and as I post on those topics I will update this list.

1. Composition in the Presentation Tier
   1. Composition in Office Client applications
2. Composition in the Collaboration Tier
3. Composition in the Process Tier
4. Composition in the Services Tier
5. Composition in the Data Tier

[OBA Solution Framework Index]

[UPDATE: For more info on the "OBA RAP for Price Management" (OBA type #5 below) announced yesterday, read the blog posting by Moin here.]  

Traditional business applications have mostly either provided collaboration (focus: information sharing), decision support (focus: BI, reporting, planning, event management), or transactional support (e.g. data entry).

However the rich capabilities of the Office system enable applications that can combine elements of collaboration, decision support, and collaboration into a single, unified user experience.

For example, the following screenshots show examples of different types of OBAs, and illustrate how collaboration, decision support and support for business transactions / process are brought together.

EXAMPLE: An inventory analyst gets an email (COLLABORATION) with information on a stockout. The email shows him tabular / graphical data in Outlook. He can click on the custom ribbon for Procurement, to pull up a list of Purchase Orders in a custom task pane filtered by the item / location / date of the stockout (DECISION SUPPORT). He can then select an order to expedite, which opens up a form within Outlook for him to make changes to the order (BUSINESS TRANSACTION).

This is one type of OBA, where the Office client is connecting to back end systems for data. A list of different types of OBA is as follows:

1. Task-centric application built into a stand alone Office client with custom task panes, custom ribbons, and application-level add-ins. For example, a data visualization / analysis application built into Excel.
2. Customized office client connecting to back end systems to retrieve data (e.g. inventory analyst application in Outlook above, or Duet v.1.0)
3. Application built into the SharePoint portal (Windows SharePoint Services or Microsoft Office SharePoint Server), with all data stored in SharePoint lists, or as documents stored in document libraries. I met a customer (large manufacturing company) that had such an OBA to manage their protfolio of R&D projects - with a SharePoint site set up for each product that was being worked on. 
4. Application built into the SharePoint portal (Windows SharePoint Services or Microsoft Office SharePoint Server), with connections to back end data stores. The user experience for such an app might look like this:
   
5. End-to-end business solution that is spread across Offfice client apps and SharePoint server. Such a solution would have all the five tiers I listed in my posting here - presentation, collaboration, business process, services, and data. Some examples of these kinds of OBAs are in the OBA Reference Application Packs (RAPs):
1. OBA RAP for Supply Chain Management
2. OBA RAP for Loan Origination
3. OBA RAP for Store Operations
4. OBA RAP for Price Management
6. Hosted OBAs used as the delivery mechanism for SaaS (Software as a Service). You can think of Office Live as a great example of this - which is hosted on WSS, and which offers business applications for small businesses. This might be a great opportunity for hosters offering SharePoint to add differentiated lightweight application templates. More on this in a later blog posting.

[OBA Solution Framework Index]

Consider a team that is trying to deliver an OBA for a particular business process (e.g. to manage procurement). This could be an internal in-house IT development team building a standardized business process for enterprise-wide deployment, or it could be a solution provider who is building a packaged application to sell to customers. In either case, development of the OBA is a separate, decoupled activity from actual deployment of the OBA. So the solution provider will not know the end users at the time that the solution is being built. Instead the solution provider should work in terms of personas and roles, and package OBA artifacts (e.g. document templates, reports, dashboards, actions, workflows) accordingly. Doing this in the form of an OBA actually makes it easy to customize the business solution at the time of deployment, to manage change post-deployment, and for end users to personalize the solution extensively at any time.

In most business processes, people and systems interact by exchanging documents.

It should be straightforward for a business analyst to identify the roles that make up the process, and to create personas for these roles. For example:

 Name: Alicia   Role: Purchasing Agent

“I order materials and supplies. I follow up on PO confirmations and partial receipts. I also research suppliers to get the best quality products at the lowest price.”

Name: Eduardo   Role: Order Processor

“I am responsible for the physical entry of orders as well as other sales support tasks. I take orders passed to me by the sales rep as well as repeat orders directly from customers.”

For each persona, it should be possible to inventory their needs for this business process:

1. Document templates (e.g. forms, spreadsheets)
2. Metrics, reports, dashboards
3. Workflows
4. Task lists
5. List of alerts / exceptions / notifications to be monitored

These artifacts can be built into a set of role specific SharePoint personalization sites - one for each persona. These sites could then be packaged as a set of Site Template files (i.e. .stp files) that make up the business process. During deployment of the solution at the end customer site, the SharePoint administrator would deploy these site templates, and then create personalization sites from these on the shared services provider using the admin console.

What is the advantage of this? This allows end users to automatically discover links to role specific sites on their personal 'My Site's. This can be set up after deployment, by the SharePoint administrator, who would set up 'audiences' that map to each role in the deployed solution through rules on properties stored on User Profiles. Then the personalization sites could be configured through the admin console, to automatically appear on 'My Sites' for those users who fall in each of the configured 'audiences'.

[OBA Solution Framework Index]

Office Business Applications (or OBAs) represent a good way to provide solutions for cross-functional business processes. Office Client applications allow users to interact with systems from using familiar interfaces. Office servers bring together structured and unstructured processes, data and documents. However it is important to understand that the platform capabilities required to support a business process, are more than just what you get with the Office System. The good news though, is that the Office system is a good way to surface capabilities from other platform technologies e.g. SQL Server (data, integration / reporting / analysis services), BizTalk Server (business process management / monitoring), Active Directory (identity management).

Before going into details on how to do this, let us look at what the anatomy of a business solution looks like. This will be done from the perspective of a software solution provider - packaging an OBA solution for subsequent deployment and implementation at a customer location. This high level overview will set the stage for more detailed drilldown in subsequent blog posts.

Any business solution will need to include some (or all) of the following elements. All of these elements can be supported through an OBA.

1. Roles: Most solution providers won't know up-front who the users in the system are going to be. However they can (and should) partition artifacts of the solution by role. Later when the OBA is deployed, and actual users are assigned to roles, then these users should automatically have solution artifacts for their roles become available to them. Another benefit is that knowledge networks within the enterprise become explicit (e.g. who knows what, how to search for persons by skillset).
   Implications for OBA: Consider role specific SharePoint dashboards and personalization sites. Later when people are assigned roles, then links to their dashboards / personalization sites should automatically appear on their 'My Site's i.e. auto-discovery of tasks, documents, reports, ... etc that are part of their job.
2. Information Channels: Different roles can receive information through various channels e.g. through a web portal (i.e. ASP.Net, SharePoint), through an Office client document, through a mobile device, through a smart client application, through IM (i.e. Office Communicator). The business solution not only needs to support these multiple modes of information distribution, but also extends the reach of these front end systems into data stores in the back office.
   Implications for OBA: Consider the information architecture holistically e.g. design SharePoint site hierarchies and document libraries to match business processes, roles and organization structure. Key artifacts (e.g. documents, dashboards etc.) related to particular business processes can be arranged in the context of this information architecture. Office client applications can be extended through process specific add-ins that plug into these information structures. Consider using VSTO SE to build these add-ins.
3. Process: There is a spectrum of business process from collaborative (e.g. people-to-people interactions) to transactional (e.g. structured people-to-system, and system-to-system interactions). Any business solution needs to be able to represent business process that enables the right level of visibility and monitoring.
   Implications for OBA: For document routing within a team or organization, consider using Workflow technologies within SharePoint. These might be simple sequential workflows that extract information out of documents that get added to document library, or might be complex state machine worflows that coordinate tasks / document routing across subordinate workflows. For managing the lifecycle of business processes / business entities (esp. when coordinating across multiple groups or organizations), consider using BizTalk to handle business processes external to SharePoint.
4. Rules: Often business processes are fairly static. Once modeled, they are not updated very frequently. More often the dynamic elements of a business process are the business rules that make up the process. Consider externalizing rules from workflows / orchestrations, so that their lifecycle can be managed separately from the lifecycle of the processes that they support.
   Implications for OBA: For workflows that are embedded in SharePoint, consider management of these through a separate rules service - with a deployment mechanism into SharePoint.
5. Events: Most people tend to suffer from information overload as part of their job. A typical response to this is to handle business process through a process of Manage-By-Exception. Therefore a business solution might choose to support management of the lifecycle of events / exceptions that are part of the business process. The lifecycle of an event typically is Detection - Notification - Resolution.
   Implications for OBA: Consider modeling events within SharePoint through a Task libraries that 'own' a particular business event, and reports / dashboards to provide visibility. Consider managing the lifecycle of these events through State Machine workflows running within SharePoint. Alternatively, the lifecycle of these events could be managed by the BPM server (e.g. BizTalk) external to SharePoint, and SharePoint would just act as the visibility layer (for reporting / task management).
6. Data: In some sense, data is the fuel that drives any business solution. For solution providers, there needs to be a well defined data interface, that clearly marks out the data entities which are consumed in the business process - and which need to be mapped to back end data stores at deployment time.
   Implications for OBA: Consider modeling business entities, and the actions that can be taken on them, using the BDC (Business Data Catalog, a shared service within MOSS). Solution providers can map these to WSS lists, and then these could be mapped to other back end data stores at deployment time.
7. Performance Management: In order to support a complete business cycle, a business solution needs to enable roles to plan activities, carry them out, measure the impact, and make appropriate course corrections. These are covered through the following four steps - Monitor, Analyze, Plan, Execute. Implications for OBA: Personalization sites that support particular roles in a business process, should provide artifacts that support each of these four steps e.g. reports / dashbords for monitoring, links to activities for execution, and links to spreadsheets for analysis and planning. These artifacts should plug into the data / information / eventing channels described above. Also consider using Performance Point that plugs into SharePoint, and that provides capabilities for three out of the fours steps listed here (monitor, analyze and plan).

[OBA Solution Framework Index]

An OBA is an emerging class of business application that bridges the gap between productivity tools like MS Office, and back-office systems such as ERP and other line-of-business applications. My colleague Mike Walker discusses this in more detail in blog postings here and here - so I'm not going to talk about what an OBA is. I'll just focus on what makes architecture for an OBA different from that of traditional enterprise applications.

Most business applications fall into one of three categories today:

1. Transactional apps (traditionally data entry such as processing orders)
2. Collaborative apps (traditionally email, FTP, or IM)
3. Decision Support apps (traditionally BI tools for reporting, or analytics) 

An OBA is different - you can think of this as a composite that combines elements of ALL three of these kinds of applications, and in ways that can be personalized by end users. For example an OBA might be a set of role-specific dashboards that offer document sharing and management, reports, forms, and real-time communications - all in the context of specific business functions.

Now consider a traditional 3-tier architecture for business apps:

This model works well only for the first kind of applications - the transactional ones (e.g. data entry). However it incompletely describes applications for decision support or collaboration. The problem is that it doesn't account for the differences between human workflows (e.g. people-to-people interactions) and system workflows (e.g. automation of business proceses). Human workflows involve people and roles, while system workflows involve applications and services. Human workflows are flexible and dynamic, while system workflows are prescriptive and formalized. Human workflows revolve around unstructured data and documents, while system workflows involve a different set of structured and / or transactional data.

So an architecture for an OBA needs to accomodate five separate tiers as shown below. This does not mean that all OBAs must have ALL five tiers, although this might be expected from OBAs that  deliver composition across all three types of business applications above.

So for example, the five tiers in this figure could be mapped to specific platform capabilities as follows.

[OBA Solution Framework Index]

The set of blog postings tagged OSF will provide architectural guidance on how to build / package / deploy / deliver business applications on the Microsoft Platform, using Microsoft Office (Client / Server) and other platform technologies. These solutions are called OBAs (Office Business Applications) as they heavily leverage the new technologies in SharePoint v 3.0 (MOSS / WSS) and the Office Client. However OBAs also pull through capabilities of a number of other platform technologies e.g. SQL Server (and IS, RS, AS), BizTalk, Performance Point, and the core .Net 3.0 Framework.

The OSF is not a product, nor even an official MS code name, but instead a unified set of architectural guidance (plus ref. architecture and ref. code) for building OBAs. I work on solution architecture in the Microsoft Architecture Strategy Team. This team (AST) works with customers, partners and the larger architect community to help come up with innovative ways to put our different technologies together. Over the past year and a half I've been looking at the architecture of business applications in the enterprise, and focussing on OBAs. As I've talked to different architects who are trying to build their own OBAs, I've found that I usually get a common set of questions. So over the next month or two, I'm going to put out a set of blog postings on this topic - each around a different question - and I'm infomally calling this the OBA Solution Framework.

I'm going to focus on three broad areas in my set of postings:

1. What are the kinds of solutions that I can build as OBAs? What are the application patterns?
2. What are the infrastructure / operational implications of OBAs?
3. What would Management / Governance of OBA solutions look like? For example, SDLC (Software Dev LifeCycle)? ALM (Application LifeCycle Management)?

Needless to say, feedback / discussion / comments on any of the postings in the OSF are very welcome :-). Also my views are that of a solution architect, and not the official views of the various Office development teams.