I have been meaning to write this post about software factory verticalization for some time now. I actually started this post at the start of the year, but we've been busy writing a number of MSDN white papers recently which have gotten in the way, which in retrospect has been good because some enlightenment has come my way about this since then.
The title of this post would not mean anything if you didn't know what 'horizontal' and 'vertical' software factories were. In that case then for simplification, ‘Factory Verticalization’ merely means ‘Factory Specialization’. Ironically enough though, most of the factories being build today are ‘Horizontal Factories’, so specialization in these cases means ‘Factory Verticalization’.
In almost all my interactions with customers about factories I am constantly reminded that ‘verticalization’ is the one key principal aspects of software factories today which is largely going unaddressed (and poorly understood). I am convinced that there is a clear and immediate requirement that we need to deal with more effectively in the present. I go so far to say that without catering for Verticalization of any factory, would seriously limit the adoption of factories in the future. It is such an important aspect to provide for if we are to realize the full vision of product lines and asset reuse.
What are these axes?
Before I get into this lengthy discussion about different aspects of verticalization, I think I need to explain what we mean by 'Horizontal' and 'Vertical' software factories in order to set the right context for the discussion.
In software factories we use these terms (vertical and horizontal) in the respect as they use them to describe markets. In software development these terms are generally well understood to differentiate broad skill-sets (or more general capabilities) typically focused at particular technologies and platforms, from the skill-sets/capabilities applied to specific industry domains (e.g. finance, manufacturing, retail, etc). The intersection of these axes in software engineering, in theory at least, applies the expert technical solution people to build instances of solutions for particular vertical domains, under the guidance and knowledge provided by the industry domain experts. The assumption is that the horizontal assets (people and artifacts) can be reused (with some specialization) across many vertical domains - it’s really a synergy of the two axes.
If you think about it more deeply, when applied, vertical domains provides a specific abstraction of the solution that can then be mapped down to multiple horizontal domains. Which is starting to sound a little like what part of the vision of MDA was attempting to do, but we are not going into MDA versus Software Factories in this discussion (you can use your own blog for arguing that issue!).
Specialization/Customization
In order for a horizontal software factory itself to be reusable in multiple vertical domains (which is where it is ultimately destined for) there has to be some level of customization that allows it to be specialized towards any particular vertical domain. If we don’t allow this, then we simply can’t adapt a factory to create specific enough product lines, which means we are limited to the productivity we can achieve with it as a reusable asset. That’s not to say we can’t. We can do the product lines, but the overhead of doing so is much larger than it needs to be, because some things in the particular domain will either be common, or specific across the whole product line for that domain. Remember, the power of factories is in how specific they can be.
A practical example:
Take a web service software factory (you can take the new version of the Web Service Software Factory (Modeling Edition) from the patterns & practices team as an example). One point of variability of the product this factory creates (a web service) would have to be the data contracts (Service Operations use Data Contracts to define the schema of the messages the operations exchange). The assumption the web service factory builder made (accommodated for) when he designed the factory was that each instance of a data contract created by the factory could be wildly different between instances of the web service solutions the factory builds (hence the point of variability). Therefore the factory provides a ‘view’ (as we call it) to expose the data contracts and allows the developers using the factory to configure them (might be using a DSL or XML file for example). Now, let’s use this factory to create a real web service.
The Modeling Edition of the Web Service Software Factory predictably uses multiple DSL’s to define data contracts and service operations.
A service operation in a financial web service would certainly have a different data contract than a service operation for a retail (shopping) web service (in fact both the service contract and the data contract are variable). The factory correctly defines data contracts as variable and as such provides views (abstractions) that allow you configure them different for these vastly different domains.
Now, suppose your company (assuming you want to use this factory to create web services for your customers) only builds financial web services for your banking customers. You are experienced experts in this industry domain, and you know what finance customers want in the financial web service solutions. Finance web services would be limited to certain functionality, and compliance and industry standards, and as such the industry has defined certain specific data standards for the data contracts and service operations you should use in your financial web services. Suddenly, pieces of a web service are very predictable (not variable and common), and you wouldn’t want to invest time and money into configuring them with the factory to the same known value every time you go to create a finance solution with the factory. What you would want to do is tell the factory that this variable piece can be predetermined for certain types of financial web services. And you wouldn’t want those using the factory to bother with those parts. Furthermore, some parts for this domain, may have special names. For example, messages in this domain are known as ‘advice’.
So you are likely to want to do is customize the basic (horizontal) web service factory in two main different ways, and create your own very specific financial (vertical) web service factory that creates only financial web services the way you like them. Specialization through Customization.
Firstly, you might want to predefine parts of the solution the factory is building, without having your developers wasting their time configuring them. You may also want to rename the parts of architecture used in factory so that finance domain experts understand the moving parts better. Let’s call this type of specialization, the ability to fix certain variability points to fixed values.
Secondly, there may be modifications to the how the factory creates its web service solution for your particular finance domain, for your particular organization. Perhaps the code templates need tweaking or modifying, or need to use some of your organizations binary assets (i.e. frameworks for example). You might need to add other aspects to the architecture of the web service that are specific to finance web services, and have these new aspects exposed by the factory. Let’s call this type of specialization, the ability to extend certain variability points.
To do both these types of things, you would want to customize the (horizontal) web service factory and create your own vertical factory variant specific to your financial domain.
In the cases where you want to fix some point of variability, you want to achieve higher productivity in using the factory, and tailor its naming to suit those who understand this domain better. In the cases you want to extend some point of variability, you want to achieve flexibility, extensibility and reuse. In practice you want the ability to alter the base (horizontal) factory and thus create your own vertical factory.
(see later for practical means to do this today)
You might be surprised to know how many people desire this specialization capability. And then you might be more surprised just how hard it is to customize a software factory to be able to achieve this today, based upon current technology. I am confident there is a good amount of evidence at Microsoft patterns & practices to support this view, from customers trying to use their factories.
Let's assume then, that any instance of a software factory used in real development will have a vertical factory component and a horizontal factory component – whether intentional or not. The final resulting factory will be a synergy of both these axes. We are going to expand further a little more on horizontal and vertical factories in the next sections, but I first wanted to introduce the parties who play this factory building game.
In practice, there are several parties involved in building factories since they consist of both vertical and horizontal factory assets. Since once we get into this, things get interesting as far as who does what. Consider the parties involved in the software factory development lifecycle (building, using, consuming etc), as follows:
- Those that own the factory, and develop/maintain it – we will call these the engineers (i.e something a software vendor would do)
- Those that use the factory to create products – we will call these the developers (i.e something a solution integrator would do)
- Those that consume the products – we will call these the customers (i.e. something a business or end-customer would do)
Clearly we have other parties involved here too, such as:
- Those that define the requirements for a factory – these are typically the customers, and developers.
- Those that own the assets of the factory itself – these are typically the engineers.
- Those that own the assets of the factory-made products – these may be the engineers, the developers or the customers.
Horizontal Factories
Let’s just have a deeper look at what horizontal factories are, where they come from, why they are popular now and what they mean in the bigger picture.
Horizontal factories usually provide particular patterns, architecture or technology implementation. This would help explain why most of the factories we see today are horizontal factories.
Primarily, from a technology perspective, patterns, and architecture (models) are highly reusable for whole classes of solutions. Those who want to provide guidance on correct architecture, patterns and technology use would provide these initially. Which would explain why Microsoft has tackled these first.
From a marketing perspective, if you can call it that (which is really isn’t), one of the primary reasons Microsoft currently releases only horizontal factories is that: at present, software factories are a new concept to the industry, and Microsoft is leading the charge here. It makes perfect sense that the first factories from Microsoft must have some large significance and impact and benefit to the existing marketplace. So they chose horizontal factories to tackle first. I am not saying this was ever consciously intentional by Microsoft patterns & practices, but it just makes sense that that more people who can gain value from these new ‘things’, the larger the impact and better the chance of success of them (via feedback channels) since this was new ground to cover. By the way, patterns & practices goes after solving significant real world enterprise problems, so picking the horizontal domains of the software factories that they did covers a large proportion of the application types being built today in the enterprise, and the source of a great deal of pain for most enterprise developers.
If you think about it, if they had instead chosen to implement only specific vertical factories, say they did a ‘financial payments web services’ instead. How limited would that impact be with general enterprise development? Not really applicable to most enterprises, all that guidance goes to waste away from the general enterprise development populace. These would only solve problems for niche markets, and would make it much harder to leverage reusable infrastructure and assets for other similar verticals. Not much feedback could be collected from that and this would slow the progress of software factory development.
Reuse
Now, whilst it is feasible to create vertical factories initially without creating horizontal factories first (i.e. very highly specific factories), (a) you still have the same set of horizontal technology problems to solve (albeit at a different level of abstraction), and (b) you won’t have much to reuse of your factory in other similar domains. Reuse is a key goal in the game of factories, and by the nature of the domain specific beast it is by design that specialized domains (assets and models) aren’t easily generalized. You will notice that as your assets become more specialized, reuse of them rapidly declines. (This is exactly way they are so powerful, because they are so specific).
Architecture, patterns and their models are highly reusable across multiple more specific domains.
Horizontal factories are (should be) characterized by being very customizable to be specialized towards any specific business domain (within the general domain they address). The value of them today is that they proffer reuse, and convey foundation patterns and reusable logical architecture.
Although many of the current generation of factories do provide mapping from domain model to physical solution, their most valuable assets are in fact the domain models, patterns and architectures they codify.
The new Web Service Software Factory (Modeling Edition) also provides very valuable infrastructure assets that ideally should be part of a general software factory platform (for all factories use), and this infrastructure is also highly reusable, but for the purposes of this discussion should be considered as part of the base infrastructure that all factories are built upon.
So how reusable is all this stuff? For the purposes of the discussion following, you can consider that are really two layers combined together in most horizontal software factories today.
The top layer of the factory cake is the domain models that describe the patterns and architecture of the solution domain. These types of assets describe the logical architecture of the product being built.
The second layer of this cake contains the mapping from logical architecture to physical solution domain. Since there is this mapping between logical and physical solution, this mapping may well vary depending on the underlying implementation created upon the architecture.
Now, I don’t want to set the expectation that this mapping can or should be platform independent (such that MDA promotes) since, platform independence is a much larger challenge than technology or implementation independence. And one that practical software factories in general avoid. Platform independence requires platform independent architectures, and effective specific software factories will define successful platform specific architectures. Remember, software factories are very solution domain specific. Platform independent models are likely to be too abstract to become practically useful for product line development.
The second layer is where you map the given architecture defined in the models to executable code (compiled or otherwise). This could be source code, configuration files or other solution artifacts in most cases. This mapping will very much depend on the solution based assets you use in the particular solution the factory builds (such as frameworks, class libraries etc), and will be typically composed of source code templates.
As an example, the web service factory could create web services that could be implemented on several different frameworks and class libraries. Perhaps my enterprise organization has already invested heavily in some framework or class libraries assets which support web service implementations and we need to reuse those assets because of corporate standards, IT investments and the like.
In defiance of current thinking, as we will see later, I assert that it is not always a valid assumption that the person creating the top layer of the cake will be able to control the orchestration of the physical assets used in the mapping of the second layer of the cake. Thus the second layer of the cake needs to be highly customizable, but yet remain specific to the architecture and platform used in the actual solution.
For many of the Microsoft factories you see today from patterns & practices, the basic assumption they make is that the horizontal factory defines the solution based assets as well. Which is fine, as long as that is possible and feasible in your particular enterprise to buy into, and eat, the whole cake as it were. In other words, that you are able to adopt lock-stock-and-barrel not only the architecture but the underlying technology solution based assets (i.e. code templates, libraries and frameworks and technologies used). However, as I discovered in some world markets this assumption does not always stand as far as IP ownership of these implementation assets goes, and those building the factory will not always be able to assume the solution assets. Some customers just can’t adopt and deploy the latest and greatest either.
We will come back to this last point a little later in the challenges discussed in the next sections.
Vertical Factories
As described before, vertical factories are the specialization pieces, and should almost always be the end-developer factories used to build a solution instance. Again, this is not to be taken to say that you can’t actually use a horizontal factory as is – you can. But it is very likely in most cases you will want to refine these horizontal factories for you particular organizations solutions. A horizontal factory can be viewed in this respect as an unrefined factory. (Sounds harsh, but you get my drift)
Vertical factories will not in all cases be built upon horizontal factories, but regardless, will more than likely always contain horizontal assets (remember that a factory is also an asset). These horizontal assets will be harvested from the patterns and architectures and implementation layers from existing solutions that the factory also prescribes in its products.
Personally, my opinion alone, I believe the future of factories will provide design principals and practical means to compose multiple horizontal factories together and define the specialization of them contained within the vertical factory (entity of some kind) that actually builds the solution. I guess time will tell if and how that materializes.
To expand our factory cake picture from earlier, it will be the vertical factory that provides the specificity of the top 2 layers of the cake; both the logical architecture and the implementation mapping. In other words, the vertical factory will contain the specific stuff to the particular instances of the products being built, and that specific stuff could be customizations to both the top layers of the cake (provided by most art by the horizontal factory).
In extreme cases, the vertical factory may provide the entire second layer of the cake, and the horizontal factory may only define the top logical layer.
Vertical Challenges
So let’s have a look at some of the practical challenges raised by practical implementation of vertical factories, and how the above model of a factory can be used to aid in the solutions to these problems.
Who owns the factory?
Today, we often think of a very simple ownership/use/consume model when it comes to factory authoring, factory use, and the consumption of factory built products.
We tend to think that one party (the vendor for example) will create the factory and another party (the solution provider) will use the factory to create products for another party (the end-customer). We gave these parties names at the start, and will use them here for clarity. We will say that the engineers create the factory, and the developers use the factory to create products for the end-customers.
For a practical example today using a classic outsourcing model, Microsoft patterns & practices (vendor engineers) create the (horizontal) software factory and ACME Consulting (solution integrator developers) use the factory to create products for their enterprise customers (end-customers).
We have discussed the extension this model in this article to include the development of a vertical factory (which contains the customizations to the horizontal factory). In this case our simple model changes a little since it will be the solution integrator (engineers) who aggregate the horizontal factories and create/compose a vertical factory to be used by their developers, since they understand the various vertical markets they have end-customers for.
When dealing with IP ownership of the assets involved in factory building it is often assumed that the party building the horizontal factory are the same party defining the logical architecture and the mapping to physical assets of the products the factory builds (i.e. the vendor engineers). We also assume that this party is the same party that actually owns and defines the product assets (i.e. class libraries, frameworks etc.). As we will show, in some markets and situations this cannot always be the case.
In our modified model now, assuming the creation of a vertical factory, the party that creates the physical assets could be either the author of the horizontal factory (vendor engineers) or the author of the vertical factory (solution integrator engineers). In reality it will be a mix of both in most cases. So, this implies that the engineers of the solution integrators will provide customizations in the vertical factory to be used by the developers in their organizations. So far so good?
So now, what happens in the cases where the end-customers have existing solution assets they need re-used in their products that the solution integrators are building for them, because of corporate policies or existing technology investments? Furthermore, what happens when these solution assets are protected by intellectual property (IP) constraints? For example, in cases where the solution integrators don’t have unlimited access to these end-customer protected assets?
I recently visited one such global market in Japan where that was the case, and I see there are other such cases, for example in government classified projects.
Then who owns, develops and maintains each piece of the factory puzzle?
If you are paying attention you might say “no factory deal! – go build a one-off solution!”. Since, we are basically saying that the solution integrator needs to build a custom vertical factory specific to this particular customer’s specific solution domain needs, which can’t achieve economies of scale or scope for them. A factory solution cannot help here, unless the end-customer will require multiple product instances to be created for them. However, if they do require multiple instances to be created for them, then the extra investment could be worth it to the customer. But because of IP issues again, it would appear then that it could only be the end-customer who could build the entire factory stack because only they can see the whole picture and have access to all the necessary pieces?
But then this does not hold true in reality, since the end-customer is unlikely to have the technology resources to build and maintain a factory. So it would appear that the solution integrator, with the end customer, under a special IP agreement would need to create an end-customer specific vertical factory tailored to creating products specifically to that end-customer.
Using the modified model described earlier, this division of labor (between solution integrator and customer) becomes feasible when you can separate the logical architecture from the physical mapping, and this is the basis for enhanced model (cake layers) of a horizontal factory - due to the separation of concerns.
How do you verticalize today?
Verticalizing a factory today can seem impossible given the tool-sets we have at our disposal.
For most customizations of today’s factories you need to ‘crack open’ the factory and mess about with the source code internals of the recipes, DSL’s and the like. This is not a guided experience, and requires very deep technical knowledge of the factory, and deep technical VS extensibility skills. Not an easy ramp-up for those new to factories.
Extending a factory today is a function of what external interfaces the factory may expose today (if any at all!). For example the patterns & practices Web Service Software Factory does a good job of providing a number of extensibility points and interfaces, both via a programming model and configuration. But anything beyond what it supports at the programming API level, and you'll have to crack it wide open, effectively voiding the warranty and support policy of the factory. It’s wild-west country from there onwards.
Ideally, what we need is firstly a structured authoring environment to build factories to have a well defined structure, with tools that create standard factory asset types (recipes, DSL’s, templates and the like). These tools would be an abstraction levels above the DSL’s and GAT recipes we build today. Basically they define a meta-model (architecture) of your factory product that can generate the required assets such as recipes and DSL’s and the like. This is basically a part of the software factory schema, transformed into the factory template, that has been quite difficult to support with today’s tooling and technology.
Once we have that though, for verticalization purposes alone, we would need a way of formally declaring the points of variability of the product architecture of a base (horizontal) factory. These declared points of variability could then be provided with default values/views/assets by the base factory. Or the factory may declare them ‘abstract’ requiring vertical factories to be built to subclass them and extend these points through customization interfaces.
We would then need a ‘customization authoring tool’ that we use to load the base factory into, and extend these pre-defined customization points with standard tools, that creates for us a set of customized assets output into a vertical factory in some form.
In this way we would not have to crack open a software factory any alter recipes or DSL’s just to predefine say, the data contracts of a web service product. Instead we can provide a simple subclass that hooks into a predefined point of variability customization interface and either provide fixed factory configuration, or extend the architecture of the product.
However, I feel this customization capability may be too far out for us right now. Instead we need to look at practical means today to achieve the same net effect (albeit a less guided experience).
Practical Verticalization Today
In absence of authoring tools that allow us systematic guided and structured customizations of existing factories and their assets, we still need to find ways of achieving verticalization with known techniques and tools today.
When you consider what verticalization really means you’ll understand that specialization can imply both simplification of, and extensions to, base models.
One practical means to achieve this is through model transformation.
Few people have entertained the possibility that DSL instances can render other DSL instances. Since you can transform models to create other textual artifacts, and a DSL is by default persisted simply as XML, it is entirely feasible to have one model instance generate another model instance. In the terms being discussed here, it is feasible to have instances of vertical models transformed into instances of horizontal models.
For the case where the vertical factory fixes points of variability of a horizontal factory (such as fixing the data contract of our web service), the vertical factory can provide simplified models defined in the horizontal factory. Once configured, these vertical models could then use built-in transformation rules (containing the fixed configuration) to generate the pre-configured horizontal factory models.
By the same means, your vertical models can extend the base horizontal models and expose new aspects of the architecture for your particular vertical domain. These vertical models can again then either be transformed into pre-configured horizontal models, or can be transformed directly into other solution artifacts.
Basically the vertical models contain metadata that is used to render the horizontal models, and developers using the factory would only need to configure these vertical meta-models.
It's not actually that simple in practice, you still need to manage synchronization between higher level (vertical) models and lower level (horizontal) models, which is not that trivial. And you have to somehow enforce a policy of not editing lower level models.
Your vertical factory could also contain modified mapping assets such as modified artifact templates and solution based assets if that type of customization is required.
As a follow-on example, our financial payments web service (vertical) factory could simply be an enhancement to the original (horizontal) web service factory, just simply adding additional DSL’s, and modified artifact templates.
These additional DSL’s would define domain models that were a simplification of a general web service architecture. Perhaps devoid of specific data contracts and service operations. Perhaps only offering the user basic configuration settings (metadata) such as web service name and ‘payment advice’ message names for example. In addition, it may, for example, define additional configuration a special security authentication model that only banks use over and above the standard ones the general web service factory supports.
Again all this can be configured within this new vertical model (a simplification and extension to the base models provided by the horizontal factory). When this vertical model is saved, it could for example, automatically generate the solution artifacts for the data contracts and service operations, (whether they are defined in DSL’s or other solution artifacts). These artifacts could be hidden from the user by some means (perhaps hidden solution folders in Visual Studio) and the developer prevented from editing them (perhaps source control locks and the like).
In these ways a solution integrator can take an existing horizontal factory (created by a technology vender), and customize it easily by adding new DSL artifacts to it to expose only the required extensibility points pertinent to the particular vertical domain. Then hand it off to their developers to use only the vertical abstractions to create the specific solutions.
In this way, they very quickly increase the productivity of the factory developers at the minimal amount of customization development and cost, without having to crack open the horizontal factory and understand all the internal workings of it.
Of course there well maybe other types of customizations that cannot be easily achieved without cracking open the factory and modifying it. For example, modifying some recipes.
I would love to continue the discussion, and hear some of your thoughts on practical verticalization techniques.
Clearly this was never going to be the short post I am striving to write one day - bummer, maybe next time...