Jezz Santos : EFx Software Factory

EFx Factory Futures

jezzsa — Thu, 18 Jan 2007 00:50:00 GMT

Well, its time to (officially) announce the end of further development of the EFx Factory prototype! and open a new chapter on software factories for our customers.

[In actual fact, not really hot-off-the-press news, this post has been pending getting published for some months now.]

The pioneering work on EFx has not been in vain. The good news is that many of the innovations, learnings, patterns and assets, from this factory are to be absorbed into future factories from Microsoft, thanks to the patterns & practices team. You can expect to see some of the more powerful features exemplified in EFx (such as: Product Modelling, Runtime Integration, Technology Separation and Extensibility for Composability) in the next versions of the Service Factory. The gap you see now between these two factories (discussed in detail in How does EFx differ from Service Factory?) will start to close now, as we consolidate our experiences, and ramp up with a new software factory platform - go the Service Factory!

The stuff in EFx was too good to ignore, (our customers really wanted to get their hands on it), and its only been a matter of finding the right channels to get this out publicly (to have the largest impact on our customer problems), which has really been the major challenge to date. Now that we have found that channel, you should expect to see even more innovations and compelling features in the new generation of software factories to come.

As for myself, I'll continue to work passionately on software factories, in the hope of 'making sense of it for you' (the tag line of this blog).

There are still many challenges in making enterprise development a cheaper, easier and more predictable experience with reliable results using factories, and hopefully I'll share with the community some of the past experiences, patterns and techniques in this area that we have identified as key with building your factories.

Looking Forward

Currently, this industry is on the brink of the adoption of software factories, and we currently find ourselves in a limbo, waiting for a platform to assist with the vision and boost adoption.

Many customers have identified and committed to a strategy to move forward with software factories, but without support from the platform, and experience and knowledge transfer in this domain, this task at hand seems ominously difficult, or hopelessly un-informed.

Those of us that took the plunge earlier on, without any formal platform, can testify that building factories is not an effortless undertaking. The rewards of those efforts are starting to come to fruition for the betterment of others. The earlier attempts, in various forms, should begin now to coalesce into a shared experience, and impact the future direction of this space.

Building factories today is hard. It's both hard to know what to build, and hard to know how to build it. Also, the current platform for building them is difficult to manipulate and esoteric. All this gets worse when you know we are in the shadow of a new platform that will soon to leap frog us forward again. But there are incremental advances being made all the time by many different people. It's going to take sometime before we have a fully realised vision, but this is no different than any other critical innovation in our industry, we will do it in small (verifiable) increments. Hopefully, we will be providing a bunch of tools to make this more achievable in the near future.

Most people have not witnessed the power of a real software factory yet. For most, it's hard to understand what they are and what benefit they offer the current software development market today. For those of us that have witnessed them, the potential is sobering and the benefits crystal clear. It's easy to see therefore what the future of factories can bring this industry, and how that will shape the future of software development. I hope most of you will start to witness that in the coming months.

FAQ - How does EFx Factory differ from the Web Service Factory?

jezzsa — Wed, 17 Jan 2007 12:50:00 GMT

Naturally, I get asked this all the time.

What are the differences between the 'EFx Factory' and software factories such as the 'Web Service Software Factory' from patterns & practices?

[Now, I do need to say this, I do work closely with the patterns & practices teams who built Service Factory, in fact, I am still a member of the expert advisory board on that project. So, I do need to be careful how I word this :) or Don and Tom will kill me.]

My aim here is to give a fair comparison between the two factories from the point of view of the customers using them and market assessing them. What is available today, and what customers can expect to see from factories in this space in the future (especially from Microsoft). Therefore, this is not about how they were implemented technically, but more about how they are designed, and how they are used to do what they do from the customer end-user perspective.

I thought this was going to be a brief post, but I was wrong again. (I'll get there eventually).

[I'm using colour to distinguish the answers for each factory, and aid readability (hopefully). Not here of course, but below.]

Backgrounder

The Basics

The Web Service Software Factory (Service Factory) is a released public available deliverable from the patterns & practices group. Currently, it's in version 2.0 and addresses building secure ASMX or WCF web service layers and building data access layers.

The EFx Factory is a limited, unreleased, prototype offering developed within Microsoft Consulting Services. Currently, version 0.9 and not complete or released. It provides the end-to-end architecture for building both Client Applications and Web Services with a choice of any combination of technologies.

Both factories target similar problem and solution domains and are both founded upon well known .NET architectural patterns and other assets and deliverables from the patterns & practices group.

Let’s have a look at both factories from a high level, and then we’ll consider the differences in detail in the answer below. If you are familiar with these factories already, then skip this part and move straight to the answer section below.

Web Service Software Factory (Service Factory)

The objectives at the start of this project (circa start of 2006) were to make building web services easier, quicker and provide the right guidance and patterns to implement them. At the time of inception, the application of Software Factories was not well understood and the DSL toolkit was in its infancy. GAT was a recent enabling technology investment made by p&p which steered them to more automation than modelling approach. The p&p project team (consisting of some web service pattern legends: Jason Hogg, Don Smith et al. and a number of key industry expert advisors and leaders from our partners and customers) had heaps of valuable web service guidance to put into the factory.

The approach taken was focused on implementing web service patterns with ASMX, and now WCF technology, and on providing the factory user with the source code and solution structure to flesh out the service functionality. Using available p&p deliverables such as GAT, a set of Guidance Packages (GP's) do the code generation from a set of recipes and wizards which capture the user’s settings and generated configuration and source code (via the Text Templating Engine). From there, the developer is expected to enhance the generated source code in order to complete the solution. It also includes recipes for automating some common development tasks. As well, the deliverables include a great deal of invaluable documentation, reference implementations, and guidance for designing and creating web services and implementing web service patterns today.

You can find more details about the features of Service Factory here.

EFx Software Factory (EFx)

The EFx Factory evolved from an Architectural Application Framework called 'Enterprise Framework' (circa 2003) which itself is founded upon Enterprise Library and a number of other application blocks. The project was initiated internally within Microsoft in response to demand from customers and partners for a implementation of the .NET architecture and enterprise development patterns prescribed in written form from the patterns & practices group. Over a long period of time, the framework has encapsulated these patterns implemented on the .NET framework and has refined class libraries to aid the construction of client applications and services according to these architectures as the .NET platform has evolved. The net result of using the architectural application framework was that a developer need only write a small amount of source code to complete a service/application. The software factory that evolved from this framework (circa mid 2005), focused upon the modelling, tooling and automation of the variability points of the architectures of the client applications and services. The framework provided or pointed to most of these abstractions that would have otherwise have been coded directly to the framework class libraries. The primary focus of the factory has been architectural guidance, usability, and the separation of the technologies that would be used to implement various layers of the architecture. In effect, the factory allows any technology, pattern or strategy to be used to implement the various architectural layers, which promotes re-use and composability of factory variants.

You can find more details about the features of the EFx Factory here.

Answer

So let’s have a look at the key differences of these two factories, in detail.

[For the purposes of this comparison, we are only going to compare the ‘services’ that are built with the EFx Factory to the ‘service’ that is built with the Service Factory.

The EFx Factory actually assembles not only ‘services’, but also ‘client applications’ and also ‘systems’ which are a combination of multiple client applications and multiple services together as one product.

Furthermore, the EFx Factory is somewhat of a different generation of software factory, being much closer to implementing the pillars of factories (Model-Driven Development, Architectural Frameworks, Product Lines, Guidance in Context) and raising the level of abstraction to logical design rather than physical design, so it's not really a level-playing field to do an exact comparison.]

Factory Mechanics

Service Factory is the following:

A set of Guidance Packages, one per technology and architectural layer: ASMX, WCF, SQL. These create initial solution structure, and primarily generate source code artefacts for each layer directly into the solution structure, invoked from projects in the physical solution structure.

EFx Factory is the following:

Guidance Packages to create initial solution structure, and create the logical work products of the architecture, invoked from a view of the logical architecture of the product, DSL models and Visual Studio Application and System designers.
DSL designers to visualise and define the 'specification' of the internals of the applications and services.
Factory runtime tools & runtime services to provide, view, and manage the Product Model, exposing a public automation API of the factory and its product.
A pluggable technology extensibility mechanism (models and programming interfaces) for other 3^rd party factories to plug in and extend and enhance the provided technology-independent models - enabling re-use and composability.
Integrated extensions to the Distributed Application and Systems designers of Visual Studio, enabling top-down development of the service.

Assets

Both factories deliver guidance assets that automate the development of the factory product (the service).

The guidance assets included in Service Factory provide invaluable guidance for service implementation of WCF and ASMX technology web services, and data mapping to and from a SQL database. These assets include a number of key innovations in web service patterns and secure web services with class libraries to implement common concerns in building services (i.e. Exception Shielding, Versioning etc.).

The factory delivers comprehensive written guidance on web service architecture, and message exchange patterns, to aid in completing the services which it initiates.

It also includes reference implementations to act as examples of what the factory can create using this guidance.

The EFx Factory core assets are based upon service oriented, and enterprise architecture patterns, separating the architecture and design of a web service from the technology implementing it.

The factory includes an Architectural Application Framework, itself an abstraction of Enterprise Library, and this framework underpins the models and artefacts which the factory creates.

The factory also includes a reference implementation, but minimal documentation of its use.

Solution Structure

Both factories provide solution templates that determine the initial physical solution structure (solution folders, projects, and layout etc) of a service.

Both use recipes to define the structure and naming of the solution and its artefacts. The physical solution structure is viewed using the familiar ‘Solution Explorer’. Both factory's solution templates can be customized, in both structure, layout and naming.

Both factories use metadata on the created artefacts (solution folders, project etc).

In Service Factory, this metadata determines the ‘responsibilities’ of the containing artefact (solution folder, project). The metadata denotes the type of physical artefacts that are contained in the container artefact, and furthermore determines what actions (recipes) can be applied to them.

The assignment of these responsibilities is configurable by the factory user.

In EFx Factory, this metadata exposes the meta-model of a logical work product the factory is assembling (i.e. a service interface or business component, etc). In other words, a containing artefact (solution folder or project) is the physical representation of one of the logical work products of the service (i.e. Service Interface, business components, data contracts etc).

The metadata describes the actual state and properties (configuration) of the work product, and the containing artefact (solution folder, project) contains the physical artefacts of that work product within it.

The work products of the service are configured in a separate view of the factory’s logical product in another window called the Product Explorer. This view shows the logical architecture of the service.

The EFx Factory maintains a runtime mapping between the work product in the ‘Product Explorer’, and its artefact realisation in the ‘Solution Explorer’. This separation (physical from logical) enables the factory user to re-arrange the physical structure to focus upon the deployment of the solution, rather than mixing that with the logical architecture of the service.

Artefact Generation

Both factories use GAT recipes and wizards to automate the creation of artefacts or work products of a service.

The Service Factory uses recipes and wizards to either generate source code, or to configure generated source code (on compiled CLR types), and automate some common development/debugging tasks. Source code artefacts are generated and assigned directly to the solution structure.

The EFx Factory uses recipes to create the work products in the logical architectural model, and recipes to configure the metadata of the work product in the model. In this way, the logical architecture of the service is separated and abstracted from the physical implementation of it.

Some work products of the service are represented in more detail using DSL models, and recipes are used to configure those models directly.

The factory also uses recipes to automate the restructure and renaming of physical artefacts representing the work products. For example, renaming a work product will require renaming certain physical artefacts and namespaces.

The EFx Factory defers and delegates source code generation of the work products to 3^rd party pluggable factories at designated times in the development lifecycle of the product. The factory itself is responsible for the generation of the solution structure artefact containers (solution folders, projects), and the assignment of generated source code artefacts (from 3^rd parties) to these artefact containers. This process is managed through the factory runtime.

Models, State and Process

In Service Factory, the source code is the 'model' of the service (albeit a 'weak' model). When creating the work products of the service (data contracts, service interfaces etc.), a recipe is invoked from the physical solution structure (solution folder, project in Solution Explorer) according to the assigned responsibilities of that folder or project. (See previous sections). The wizard alone provides the abstraction of, and the configuration for the 'work product' (although there is no formal concept of a 'work product' in Service Factory per se). Once the wizard is completed by the factory user, it generates the required artefacts straight into the current solution folder or project.

There is no ‘state’ of the service expressed anywhere - except in the source code of the service. The factory (not surprisingly) provides no round-tripping or reverse-engineering of the source code, so there is no way to view this 'state' at a simplified (abstract) high-level.

As there is no state, and the recipes are keyed from the assigned responsibilities on an solution artefact (project or solution folder), it's difficult to determine what activities have been performed, and what needs to be done to complete a service. The user must be aware of (or will have to discover by right-click trial and error) which recipes are assigned to which solution folders or projects. Furthermore, they must be very familiar with the process of how to complete a service and in which order they have to execute these recipes to complete the service.

Furthermore, the user must know at what stage of the overall process they are in, as this is not depicted anywhere. This is problematic when development of the service is shared between team members.

If the user needs to change something about the service that was generated by the recipes before, they either must re-run the recipe, and recall how they configured the recipe previously beforehand. The recipe will overwrite any customization to the generated code (if any) although the user is warned of this. Or alternatively, rather than re-run the recipe and recall all the settings used previously (usually in the case of a small change), the user can modify the generated source code by hand to make their change.

The Service Factory does provide additional Code Snippets to create the most common artefacts. However, these last options are the realm only of technology domain experts, and the benefits of automation that the recipes provided is really lost.

The EFx Factory provides (and persists) a model of the entire product (service) and all its work product instances. The product is managed in the Product Explorer view.

The primary work products have graphical DSL designers in which their architecture is defined. Recipes are used to configure these models. Because of this visual abstraction, the user (and his teammates) can more easily visualise what state the service is in and what steps needs to be taken to complete the service. (This is aided by validation, discussed later).

To create or modify work product instances, recipes are invoked from the models themselves.

The models provide the appropriate abstraction of the components of the service, its' variability points, and the state of the product and work product instances. The recipe wizards reflect this state when they are run, and instead of generating code directly, these recipes act upon the work products and product model itself.

To enhance the usability, and increase the productivity of using these models, and creating work products, the factory provides a number of specific editors, recipes and wizards to configure these models.

Abstraction & User Level

In the Service Factory, because the 'model' of the service is in the source code, source code is essentially the medium that the factory users must manipulate to create, configure and complete their service. This is a low-level abstraction of the solution domain.

All manipulation of the service is done via the 'Solution Explorer' view, which is the detailed physical view of the solution, (and best suited to display the deployment of the solution, not it's logical architecture). As long as this is the working view, the level of abstraction will always be at the source code level.

As there is no state of the service, and no state of the process to completing the service, the factory user will always have to be a 'developer' class user familiar with the the overall process of constructing a service from the current state of the source code (as a whole).

In most cases, to modify or enhance this 'model' this developer will require expert domain knowledge and experience of the technologies implemented by the factory.

The 'Guidance Explorer' does help provide a list of steps to complete certain processes, (and a log of the recipes executed previously), but it still does not provide a clear, reliable picture of where in the process the developer is and what needs to be done next.

In the EFx Factory, the model of the service is visualised in DSLs and in the Product Explorer. The model offers high-level abstraction of the web service architecture and patterns, and holds the current state of the service. Furthermore, these models do not depend on any technology.

To change the service the user simply uses the visual abstractions (models, explorers) used to view the models.

The factory user can be any technical user that is familiar with the patterns and architecture used by these abstractions.

The user can create the specification of a service without ever having to have any detailed experience of the technologies used to implement it. However, eventually, a technology domain expert will be required to apply and optimize the various technologies at some stage, but this activity can be deferred till later in the lifecycle of the product.

Designer Integration

The Service Factory does not leverage existing models or facilitate high-level top-down development of services.

The EFx Factory integrates and extends the Distributed Application and Systems designers of Visual Studio (Team Architect Edition if installed). The factory provides additional shapes to represent the implementation of a service, and enables implementation of the service straight from these diagrams using the project templates provided by the factory.

From the Distributed System Designer, the EFx Factory allows the user to drill into theses shapes to expose its' internal specification.

Validation & Completeness

In the Service Factory, since a weak 'model' of the web service is effectively defined in source code artefacts themselves, Service Factory can do little (apart from depending on the language compiler) to validate whether the web service source code artefacts have been created at all, in the right order, correctly or completely to form a valid and complete web service. Although, in principal it's possible to create recipes to perform this kind of validation at a course grain level.

The WCF guidance package does add FxCop code analysis rules that help verify the completeness of some WCF related code.

The EFx Factory has a comprehensive model of the service as a whole and the individual work products. The factory performs comprehensive validation of this model before any source artefacts are generated from it. In fact, this validation is a prerequisite to generation of source artefacts. The factory prevents the user from making invalid architectural configurations, and ensures the work products are configured correctly and completely.

This validation guides the factory user in pre-empting the tasks required to complete the service.

Architectural ‘concerns’ (cross-cutting concerns) for each layer in the architecture are defined in the model and enforced through validation, ensuring that these concerns are addressed appropriately, and the service complies to these enterprise-level concerns.

Architectural Coverage

The Service Factory guidance packages provide automated guidance assets (recipes, templates etc) for the generation of the Service Interface and Data Access Layers of a service. It addresses the middle layer Business Logic implementation and wiring up of the Service Interface and Data Access layers through written guidance, and customization of the source code. The Service Factory provides no guidance about architectural concerns of these middle layers, although there is plenty of written guidance available elsewhere on this.

The EFx factory provides a comprehensive architectural model of the entire service end-to-end.

The EFX factory models all the architectural layers, and their individual cross-cutting concerns (like: Authorization, Exception Management, Logging etc) at all layers in the architecture.

The models define the interaction between the components (method mapping & entity mapping) in the layers. The wiring up of the Service Interface to the Business Logic Components is automatic.

Since Business Logic implementation is difficult to model (too many technology patterns and strategies). The factory instead offers a pluggable technology strategy to implement this layer. This strategy then wires up the business logic components to the data access layers. Common strategies are: custom coding, sequence diagrams and workflow activity diagrams, each of which is provided by a 3rd party factory that can use any pattern or technology to model this layer.

Service Implementation Technologies

The Service Factory comes in two technology flavours for the service interface- WCF or ASMX, and one flavour for the Data Mapping - SQL. In order to create a 'specification' of a service, you must make a technology choice before using the factory. Once that choice is made, you are tied into that technology implementation – it is very difficult to change that technology decision later - and you'll need to start from scratch again. There is no de-coupling of technology implementation from architecture of the service – so you can’t create a design and specification of your service without first choosing a technology.

The EFX Factory separates the architectural specification of the service from the technology used to implement it - using a technology-independent model.

By providing these technology-independent models of the service's internal architecture, and a rich extensibility mechanism for other factories to plug into this factory, the EFx Factory enables the layers of the service to be implemented by any technology.

Furthermore, there are several layers in the architecture that can be extended, primarily: Service Contract Implementation, Business Logic Strategy and Data Mapping Implementation.

In each of these cases, a 3^rd party software factory specialising in a particular technology and architectural layer, can plug into the EFx Factory and implement that layer. They do this in a number of ways:

Enhancing the technology-independent model views with technology specific implementation metadata, commands and UI cues.
Providing separate technology specific views when that technology addresses work products not present in technology-independent view, or when the technology is best presented in a different view from that provided by the factory.
Providing technology specific implementation of the layer.

The EFx Factory, through technology separation and this extensibility mechanism, promotes 3^rd party domain experts to produce the authoritative abstractions and implementations of specific technologies of these layers for re-use. These implementations, themselves software factories, will then address and promote a standardised model of these layers. This then promotes re-use of these software factories in other similar architectures of other software factories.

Composability & Re-use

In Service Factory, the recipes to create the structure of a service are based upon a specific technology. Because of the lack of a 'model' and separation between the service specification and the technology implementation of that service, the recipes are bound to a specific physical implementation which they create – they assume certain source artefacts (CLR types) in a certain structure. This makes the assets and recipes very hard to re-use in other factories with different physical implementations.

The EFx Factory separates the model of the service from the implementation of it. The implementation is provided by 3^rd party factories based upon the logical architecture of a service layer. When these 3^rd party factories create physical artefacts for the layer, (specific to their technology), they don’t populate the physical solution structure directly, since they have no coupling to or knowledge of that. Instead, they pass the artefacts to the factory, and the factory, through its physical<->logical mapping determines which physical artefact container (solution folder, project) these artefacts will be assigned to.

Since the 3^rd party factories are only bound to a logical architecture, they can be re-used by other factories sharing the same logical architecture. This means that other factories can be composed using these factories.

Availability & Support

Service Factory is a freely available, released, documented, tested and supported deliverable from the patterns & practices team. It will continue to evolve as the .NET platform and emerging software factory platform evolves. This deliverable will continue to receive support from the patterns & practices team, and the development community as a whole.

EFx Factory is not finished and is not released to the general public. It was prototyped internally at Microsoft, but has not received the necessary development, testing, documentation resources and support to move forward as a releasable deliverable.

Summary

The Service Factory represents the first generation of software factories from Microsoft, which has made a significant step towards the creation and automation of web services.

The Service Factory is primarily focused upon the automation of web service implementations and patterns for specific technologies, using a wizard-based code generation process.

Its' very valuable guidance assets provide the best patterns and guidance for implementing web services and managing their lifecycle and operation.

This factory is available and supported publicly today, and is successfully being used by many of our customers in production to help them build web services more rapidly today.

The EFx Factory represents a prototype of a new generation of software factories within Microsoft, which has pioneered many of the concepts representative of future software factories from Microsoft tomorrow.

The EFx Factory is primarily focused upon logical architectural design, designer integration, automation, usability and architectural re-use and composability.

It provides a managed automated environment that uses and leverages many abstractions, models and extensibility mechanisms to present an end-to-end logical architectural view of a product, enhancing that with a rich design experience for the user.

It has no dependence on technology. Instead, it provides architectural-based technology-independent models that can be extended by custom technology-specific providers, promoting re-use and innovation in the community.

In turn, the core factory and the technology-specific providers can be re-used and packaged up to compose other factories to solve specific solution domains.

This factory is not publicly available and remains an unfinished prototype.

Moving Forward

Despite the significant advantages, and advancements that the EFx Factory offers in automating the assembly of service oriented client applications and services, Service Factory (and Web Client Factory), because of their availability, support, resource investments and ongoing development will remain the premier software factories available to Microsoft Enterprise partners and customers moving forward in this space.

FAQ - Isn't that MDA?

jezzsa — Thu, 30 Nov 2006 19:01:12 GMT

Considering the EFx Factory and its core composability features with extensible technology-independent models, a few astute people soon get around to asking the following question:

Isn't the EFx Factory just another implementation of MDA?

Backgrounder

Model Driven Architecture® (MDA) is a methodology created by Object Management Group (OMG). Their definition of MDA can be found from their FAQ page. (excerpt below)

"The MDA is a new way of developing applications and writing specifications, based on a platform-independent model (PIM) of the application or specification's business functionality and behavior. A complete MDA specification consists of a definitive platform-independent base model, plus one or more platform-specific models (PSM) and sets of interface definitions, each describing how the base model is implemented on a different middleware platform. A complete MDA application consists of a definitive PIM, plus one or more PSMs and complete implementations, one on each platform that the application developer decides to support."

There is also a good wikipedia definition here, with more references to other sources.

The EFx factory defines architectural based models (such as the one here) for parts of its product model. These are based upon common architectural implementation patterns (such as SOA contract patterns and the .NET distributed architecture). These models define views which are technology-independent abstractions of an underlying architectural framework that actually helps implement the products of the factory. The underlying framework itself, defines abstractions of the .NET framework class libraries and Enterprise Library.

The factory has been designed to define 'base models' of the internal architecture of applications and services - called 'specifications' which have no dependency on a particular technology or implementation pattern. The technologies that are used to implement the specifications are provided by other software factories that are plugged into these base models to allow extended technology-specific models, abstractions, patterns, meta-data and artefact generation from other factories suited to this architecture. This means that the technology-independent models can be specialised towards a specific technology or technology strategy. This approach prefers an extensible technology implementation to a prescriptive technology implementation.

In effect, the models and abstractions that the EFx Factory exposes, defines a technology agnostic model, where specific technology models can be provided and overlayed by other 3rd party factories. This mechanism is outlined also in this article.

Answer

The EFx Factory is not an instance of MDA.

From the definition of MDA above, it might appear that in fact, the approach taken by the EFx Factory modelling strategy, would appear to be similar to that of MDA® - that could only be true if you equate a platform independent model (PIM) is the same as the approach as taken by EFx factory which is providing an extensible technology-independent model.

The truth of the matter is that the EFx Factory approach is in no way derived from an MDA approach, nor uses any processes, technologies nor standards that are defined by MDA, such as UML, MOF or XMI.

Using MDA as an approach in the context of software factories has several challenges that should be considered in any comparison between software factories and MDA. Jack outlines some of these in the FAQ section of his book 'Software Factories'. However these issues are more related to MDA as an alternative approach to Software Factories as a whole, rather than a specific instance of a factory - which the EFx Factory is.

You might also want to read about the concerns around MDA here which appear to address some of the undelivered promises of MDA.

Technology-Independent Models

The EFx factory does not provide a platform independent model (PIM). Rather, it does provide extensible, domain specific technology-independent models. I am not just splitting hairs here. The models define specific extensibility points based upon architecture, that can be addressed by other pluggable factories ('factorettes' in this context) that either provide a technology-specific implementation or provide strategies to implement them. This is not the same as the platform independence described by MDA.

The plugged-in factorettes provide their own abstractions of a technology-specific view of the architecture that is exposed through an extensibility point. These extensibility points (variability points) have been determined by identifying which parts of the architecture require specific technologies or implementation patterns. The factorettes provide the abstractions, user experiences and implementation of that extensibility point. The abstractions provided by the factorettes are then integrated into the technology-independent base models to provide a technology-specific view of the architecture. That view can be either overlayed onto the technology-independent view, or displayed separately. It is then the combination of the views provided by the factory and the factorettes that provide the final user experience and actual implementation of the factory products.

Instead of MDA, you might say that the EFx Factory subscribes to a Model Driven Engineering (MDE) approach - which would be more accurate. It's worth noting that the approach used by this factory has evolved from a grass roots approach of iterating automation and modelling of an established application framework use, which is more in line with what software factories promote.

Multiple or Single Architectural Models/Views?

jezzsa — Wed, 22 Nov 2006 14:30:00 GMT

This was to be a FAQ, but due to the fact that the practices around this are unproven yet, there is no definitive answer to this question, rather a discussion of approaches, issues, guidelines and raising general awareness.

At the moment, there is quite a buzz about the correct usage of DSL's and modelling domains appropriately in the context of software factories.(within Microsoft at least, I would expect the rest of the DSM community have a more mature view on this).

We are starting to see an emergence of software factories of different varieties. Much of this innovation is growing from an absence of a formal framework for building factories and mechanisms to stitch them together, with navigation etc. (which may one day be available to us all).

A lot of the innovation is being put into how to describe a whole domain with multiple models, and when and how to relate those DSL's together. Services and practices are starting to emerge such as the integration service introduced by Mauro and Victor in their article on Integration Scenarios which allows you to reference elements from one model to another. Other factories, like the Ordina Factory, are using other techniques to achieve the same ends.

There appear to be two extreme views on how to model any part of a domain (sub-domain). On the one hand, some folks are using very few (or single) models to cover the sub domains. Others are decomposing the sub-domain into very many models.

Now, we need to be accurate here. There is a difference between the actual model and view (of the model). Ideally, you could have multiple models with multiple views, each view considers a whole model or part of a model. In some technologies a view can span multiple models also. Its the model that describes the logical architecture, and the view that provides the interface of that to the user. For the user, its the view that's important.

[It should be noted at this stage, that multiple views for one model is currently not supported in the Microsoft DSL toolkit. Views to span multiple models is also not supported currently. Only one view per model is presently supported.]

In the context of architectural guidance in a factory, imagine we are modelling a service, for example. Architecturally speaking, this is composed of several abstraction layers, and few deployment tiers. Lets consider the following standard .NET architecture for service orientation from patterns & practices. Since this is quite a popular architecture to create factories around at present.

Considering the abstraction layers in the logical architecture, the objective is to decompose the architecture into layers, with differing concerns, all in the scope of the whole service. i.e. Service Interface, Business Layer, Resource Access layers and their component layers.

Now undoubtedly each layer of the logical architecture is concerned with different aspects of the overall solution, and further, each layer has more than one concern to address, as well as the variability of that layer and its components.

The question is how best to apply DSL's to model these concerns and variability?

Jack addresses the same question is his article on bare naked languages.

There are a number of patterns that address these kinds of architecture, and its typically these patterns that determine the answer to this question.

However, I assert that basing your models and views upon these patterns alone, may not always be the best approach to exposing your DSL's to the users.

Multiple Models

To be a purest about separation of concerns, when analysing a problem domain, you would take the approach to define a single model for each layer which addressing a small number of concerns and the variability of that layer and its components. Or perhaps even, a model per concern!

Ideally, you would present several different views (if possible) of the model perhaps each one exposing each one of the concerns, or maybe just one view for all the concerns (which is the case today with using the DSL toolkit).

Then you would develop an appropriate mechanism to reference components from one model to another.

For example, a separate model for each of the layers:

A model for the Service Contract, and Service Adapter,
A model for the Business Logic implementation
Perhaps a separate model for the Business Entities
A model for the Data Access Logic
A model for the Service Agents

That's 6 models, but some could be combined. So in reality 4-6 models to cover the whole architecture.

All models address the concerns and variability of their layer.

Single Model

A more natural (perhaps less purest) approach, would be to define a single model for all layers which addresses all the concerns and variability.

This does not necessarily infer that appropriate emphasis has not been given to separation of concerns. It just means they are defined in the same model instead of separate models.

Ideally, you would present several different views (if possible) of the model perhaps each one exposing each one of the concerns or aspect of variability, or maybe just one view for all the concerns and variability (which is the case today with using the DSL toolkit).

For example, a single model for all layers. (example here from the EFx factory).

Considerations

Considering these two approaches (in this context).

These approaches aim to achieve the following:

Multiple Models

Maximum physical separation of concerns.
Maximum physical de-coupling of the layers.
Maximum flexibility in reuse of one model in another architecture.
Facilitate better source control in team environments

As developers can work on different parts of the architecture in isolation from others.

Single Model

Maximum holistic description of the architecture (big picture view).
Maximum usability.

The challenges with these approaches (in this context):

Multiple Models

No holistic view of the architecture. The inter-related pieces are presented in different views.
The interface to reference dependencies in other views is usually poorly visualised/represented.
Information to perform common user tasks maybe spread throughout several files.

Single Model

Complexity of diagram can be high, if the abstraction of the model is low.
Team development on same model can be challenging, as the view/model could be used by different roles addressing different aspects.

[If you have any other pros and cons to add I be glad to add them.]

Learnings?

So what can we learn from this? I believe neither of these extreme approaches is ideal. The answer lies somewhere in-between - a compromise of both approaches.

The emphasis of the views that you provide for your models, needs to be high usability. A software factory is a productivity tool for users that should not have to have the level of knowledge and experience that those building them have. So even though your model(s) employ the best architectural practices in separation of concerns, and pattern application etc, these should not necessarily be forced upon the users. Instead, these patterns and architectural practices should be abstracted in the views, in a way, that makes it easy for the users to get the job done in describing the solution domain. (Its the classic: Usability <-> Flexibility trade-off struggle)

Perhaps building the models and the views are two different roles of defining a model for your domain?

Here are some things to consider to make the modelling solutions (in a factory context) more usable:

Use fewer models when the abstraction is high and variability is low. Use many more loosely coupled models when the abstraction is low and variability is high.
For single models, use multiple instances of the model file to partition the solution into bite-sized chunks, and use cross-model references where applicable.

This will assist in team development.

For multiple models, provide holistic views of the other views that bring them together to form a composite 'zoomed out' view of how the models relate, so the user can get a feel of the whole picture rather than one aspect of it.

This will enable the users to stay focused on the objectives of the solution.

For any approach, analyse which layers are more tightly bound to others, and require less flexibility in reality - from the users perspective.

For example, in the above example, it is unlikely that the 'Resource Access Layer' is going to be used separately from the 'Business Layer', but it is highly coupled to it architecturally speaking. So it would make sense from a usability perspective to combine these layers into a single view.

For any approach, use views that expose particular concerns or aspects of the architecture.

A good use of these views is by role, so for example, a security expert can view the cross-cutting concerns of authorization across the whole architecture, without having to understand all other aspects or abstractions of the architecture.

[Again, if you have any other considerations to add, I be glad to add them here.]

FAQ - Round-Tripping

jezzsa — Fri, 03 Nov 2006 20:39:00 GMT

The first question I always get asked:

Do you support round-tripping of the generated code your factory creates? In other words, do you support a developer going into the generated code, modifying it in an arbitrary way, and then read back those code changes and their intended meanings into the models used by the factory?

[Quite often the asking party has the handy Class Diagram functionality of Visual Studio in mind when they ask this.]

Backgrounder

It works like this for our particular factory (the EFx Factory):

The factory user designs the logical components of the applications and services in architectural layers using a logical model explorer and visual designers. Each component is part of an overall (combined) meta-model that has a set of metadata (properties) associated to the elements in it. This metadata defines things like the naming, structure and concerns of the component within the architectural layer and ultimately governs how that component should be implemented. This diagram shows one of the designers used in this process.

Of course, these metadata 'properties' provide an abstraction to the actual generated solution artefacts and it is defined in architectural terms, since we don't want the factory users to have to be expert C# solution domain experts to use the models. That's on of the powers of using abstraction your factory.

[As a side note: In this factory, we do allow and support (even facilitate) the developers hand-crafting the business logic in source code, since we don't feel we can sufficiently model business logic successfully or completely for all cases to suit all solutions. The hand-crafted custom source code is referenced by the models and maintained separately from the generated code, so won't be considered as part of this discussion.]

At any given point in the design time of the logical models, assuming certain constraints and certain validation rules are satisfied, source code and other artefacts are generated from the model and designated to the appropriate container (project or solution folder) in the physical solution. It is then subsequently compiled into assemblies at a later stage.

Answer

The answer for this factory is: No

We are not saying it can't be done; just saying that the considerable effort required to support this is wasted when:

The Model is King. The model is intended to represent the architecture, its components and the architectural patterns it implements. In this factory, all the components (with business logic components being the exception) are modelled in the architecture and configurable to suit certain identified variability. This 'specification' is meant to define the specification of the solution, not the source code it generates. Since those interested in the abstractions it provides will not necessarily be interested (or skilled in reading) low level source code. The model is king, and the authority on the actual state of the factory product. All analysis and generation tools read and write to these models.
Abstraction assumed to be high: In this factory, the generated solution artefacts that represent the physical incantation of the models is provided by pluggable providers (we call them 'Artefact Generators' for obvious reasons). These providers are intended to implement a pattern or technology to satisfy a set of concerns for a set of components at a particular layer of the architecture. For example, we would have one such provider for a service to implement the Service Contract and Data Contracts using a given technology, say - WCF. Now these providers themselves may be mini-factories ('factorettes' as we call them) with their own abstractions that remove the requirements for the factory user to be experts in implementing their best practices or patterns in specific technology terminology or using a low level tool – like source code. The providers are expected to hide that complexity, and only expose a simple interface to configure it, in generally understandable terms and patterns. So, the abstractions they provide may not be as simple as perhaps the Class Designer in Visual Studio (which could be said to have little to no abstraction).
One-way mapping assumed. In following with the previous point, but more importantly, since the providers are to implement a technology based upon a meta-model of components and metadata in a well defined architectural model, they are encouraged to provide more than one set of patterns for their technology which vary and are each optimized depending on the configuration of that meta-model. You can imagine this being the same way that source code compilers implement different byte code sequences depending on how the developer wrote their source code. One pattern may not be most optimal for all configurations. So how do you determine from the modified source code which pattern is in use, and how that change in source code should be reflected in the model? It's the same problem most people see when reverse engineering assemblies using a reflector. You never get back the original source code with its abstractions in place – because there is a one to many mapping between source code and compiled code.

The next point is (philosophical)?

This factory does not want to promote source code development, favouring more powerful abstractions, visual modelling and extensibility (except, as we mentioned above, for the business logic – which we do want the factory users to define in source code). So, we don't expect the developers to tweak the generated source code.

Having said that, there are legitimate reasons they might want to do this - customization. The assumption here is that; in modifying the generated source code your intention is to define a variability point (or configuration of a variability point) that the factory does not cater for currently. In these cases, this deliberate action is interpreted as a case for further factory customization – which is a valid reason to create a variant of the factory to accommodate this variability. Its part of the natural development cycle of a factory.

Just as an interesting point of strategy to identify these cases, this factory employs a validation rule that checks the last generated source code artefacts for modifications, before re-generation. Then, if any modifications are found, the generation process is halted and the factory user is notified of the change. In those cases the modification needs to be resolved with an appropriate manual action by the factory user in order for generation to continue.

FPM - Case Study Part 2

jezzsa — Tue, 22 Aug 2006 14:00:00 GMT

Edward and myself have made some major headway on exploring and validating some experimental tooling for the authoring factories with the 'Factory Product Model' using the p&p Service Factory as a test case.

This work is based upon learnings harvested from authoring the EFX factory itself, and generalising those, and the design, into reusable tools that can be used by others to author their own factories.

The p&p Service Factory is subject of the current experiment at present.

Despite the fact that it was not designed with this product oriented approach in mind, it is nonetheless turning out to present no significant issues from what we are concluding at present. We are able to use the logical concepts that were originally used to describe the work products of this factory, and we have been able to define a model of the factory product, that could be used to generate the factory schema, solution templates, artefacts and runtime and tooling for navigating it.

Edward has been instrumental in this work, and has posted the second part: Factory Product Model, Case Study - The Service Factory Part 2 in this series. Have a look there for a first peek at the progress.

We will soon be posting more details about how the FPM designer has developed, and how it is used to define the factory product model. Then how the factory author would define their product model from the logical work products that the factory creates. Then we look at the configuration they define for these Work Products that enabless the designer to generate the assets, work products and schema of the factory.

In the next article, I'll explain how the FPM designer works in more detail, and how that is then used to generate the runtime factory tooling used by the factory user to manipulate their instance of the factory product.

[I have to mention here, that this work is experimental and does not necessarily reflect the actual approaches that may be taken in the future Software Factories tooling in Visual Studio.]

The 'Meta-Model' and its relationship to the 'Factory Schema'

jezzsa — Tue, 01 Aug 2006 06:41:00 GMT

In the previous article, I talked about the power of the factory meta-model, and why you should have one in your factory. In this article, I want to describe how the meta-model is related to the factory schema, and how that can benefit both the factory author - defining her factory, and the factory user - operating his factory.

So what is this meta-model, and how is that related to the factory schema?

A software factory creates, or outputs a software 'Product'. The 'product of the factory' represents the solution domain of the problem the factory is trying to solve. The meta-model is a model of the factory product.

Defining the Product of your Factory

In order to achieve high levels of applicability, usability, productivity and efficiency, the factory can only address a finite set of the variability of the problem domain, and through careful application of constraints and relevant architectural design patterns - define an FPM to model the product line architecture that addresses this problem domain. The product model therefore, provides a relevant abstraction of the solution domain, which can then be used to define the factory viewpoints, assets, using automation.

The factory should be able to offer the most optimal technology/pattern to implement the product for the specified problem by baking-in the experience of the factory/tool builder gained from their experience of repeatedly solving these types of problems in the solution domain. This means, that if the factory changes its configuration, the factory may choose to use a different technology or pattern to solve the problem. (kind of the same principals a language compiler uses).

In order for the factory to adapt effectively to change in problem specifications, and adapt to changes in evolving solution requirements, the factory must maintain and persist a schema of the product that can be configured/re-configured at any time by the factory author to regenerate the assets for the factory. In turn, these assets re-generate a new solution efficiently and predictably for the factory user.

Starting with the 'Factory Product Model'

In order to do this, the factory runtime environment needs a 'logical model' of the product it creates - a 'meta-model', or more specifically a 'Factory Product Model' (FPM). (see previous article on the Factory Meta-Model)

This FPM, is used by the factory author to design the architecture of the software product the factory produces, in common terms to be understood by the factory user familiar with the problem domain.
We must try not to pollute this model with details about the physical implementation of the solution domain, instead choosing an appropriate abstraction of that. The factory user should not be assumed (by the factory author) to be an expert in solution implementation using the various specific technologies or patterns of the solution. Instead the FPM should describe the architecture of the product, in general solution domain concepts, easily understood by the factory user. Correctly done, this is one way to achieve useful abstraction using the factory.

The FPM describes the things the factory will configure and produce - the 'work product types' - and their relationship to the product of the factory. Furthermore, this FPM describes the hierarchical relationship of these work product types and how they are composed of other work product types, ultimately resulting in the product. In essence the FPM describes the logical composition of the factory product, devoid of the physical implementation and physical structure of the solution. More precisely, the FPM describes the 'Product Line Architecture' of the factory.

You can see below an example of a DSL designer that a factory author could use to define an FPM for their factory. This designer could equally be implemented in a hierarchical tree view type control.

This particular diagram shows the FPM for the EFx Factory as an example. In this case, the product created by the EFx Factory can be either an Application or Service or a System (composed of Applications and Services). Each Application or Service is itself a Module that can contain one or more Subsystems, and each type of Module (Application or Service) implements its own specialisation of a Subsystem.

From the toolbox in the designer below, you can define the product line architecture in very simple terms. The Product, a variant of the product (in the product line), Work products used to describe the logical components of the product, and relationships to represent inheritance and composition.

Figure 2. Illustrates the top level 'Factory Product Model' design for the EFx factory.

The entire logical model of all the factory (all work product types) are represented in this diagram but not fully described in this diagram. Each Work Product will be assigned an asset to build them. Some of the work product types are better described by other separate models (DSLs). The FPM designer has a shape called a 'Model Concept' (green shape) which denotes that a work product type is described in separate model (DSL), which may offer more detailed work product types (white shapes) and relationships for that specific domain. For clarity, we could call each of these model files a 'Work Product Type Model' (WPM). These WPM models will ultimately be implemented as a single model file (DSLs) in the factory solution.

The other 'Work Product' shapes (yellow shapes) and 'Product Variant' shapes (brown shapes) in the diagram represent work product types that will ultimately realise themselves, physically, as either a solution, solution folders, projects or individual project files artefacts.

[A Work Product is physically realised as an Artefact. That may be a single artefact (e.g. source file). It may be artefacts in multiple files. Many parts of a single file, or multiple files (e.g. a partial class definition spread across one or many files). Maybe it's no physical artefact at all or maybe its configuration on a remote server somewhere. Not all factories create source code.]

Therefore the entire FPM of the factory can be visualised in one file, and implemented across several model files: Product Model + Work Product Type Model(s).

[Note. This is not the view the factory user will see. Instead the factory user will be provided a 'view' over this FPM similar to the one here, where they can create and configure the 'instances' of the work products of their instance of the factory. This is called the 'Product Explorer']

The Factory Schema

The FPM provides is powerful authoring experience for the factory author to define the product. A powerful user experience for the factory user creating an instance of the product.

The FPM (as a whole) provides a view across all the Work Products types, their Assets and mapping to the Artefacts from the factory schema. You can think of the FPM as a view of the factory you are building - a ViewPoint of the tool that lets you build your factory (that tool is the 'Factory-Factory' which you use to author your factory).

In fact, if ever there was a 'Factory-Factory-Factory' built (i.e. a factory that defined the schema for factory building :) ), that would contain a ViewPoint called a 'Factory Product Design'. This viewpoint would be defined as the following:

The WorkProduct would be the 'Factory Product Model'
The Asset would be the FPM Designer
The Artefact would be the FPM Designer DSL file - MyFactoryProduct.fpm (in our case)

The FPM becomes a logical starting point for the factory author to begin to describe and their work products of the factory schema and their artefacts and assets needed to do that. It is simply one of the views of the factory schema. There will be others too that view the schema from different authoring aspects.

Generating the Factory Runtime

Since the FPM describes the work products and their assets, it can be used to generate runtime user tooling such as the views required to view the product (the 'Product Explorer') and launch the assets to create the WorkProducts. This tooling would then be used by the factory user to 'operate' the factory and manipulate the factory product. This tooling is then packaged and installed with the factory into the factory user's IDE. The factory user can then explore parts of FPM (and schema) visually in a number of different tool windows (views) and assemble the factory product, using the associated assets (templates, recipes, models) for the work products from these views.

[I'd like to specially thank Martin Danner for providing valuable feedback to this article that expressed some of the points here more clearly and concisely. And to be fair, a lesson in concise, precise articulation. You can expect to hear more from Martin in the near future on factories.]

The Factory 'Meta-Model'

jezzsa — Mon, 31 Jul 2006 22:31:00 GMT

Hopefully in previous posts I have been able to help de-mystify what exactly the factory schema is and how you should be thinking about it. If you are still confused about how to build a factory and how to create the factory schema, you are not alone.

My good friend Ed Bakker and I have been working together towards improving the user experience of factories, primarily for the factory user, and also for the factory author. He has just posted on some of his own thoughts in this area (Software Factories - The Next Generation).

I wanted to expand on some of those thoughts here from a different angle, in a couple of articles and focus a little more on the schema and meta-model of a factory and how they are related, the EFx factory has a bit of a head start on most of the emerging factories at the moment.

A Solution-based Approach

The first software factories on the market right now like the 'Service Factory' (from patterns & practices team) do not define or provide or use a factory schema, nor offer help on what one should look like, or how it works. They do not provide a 'model' per-se of the problem domain of building a service as a 'product' of the factory, and therefore they don't require the need for a factory schema.

Instead, the approach it uses is to define sets of losely related GAT recipes to create the physical components of a service by directly generating the source artefacts right into the solution.
These recipes are expected to be executed, by the factory user, in the right order, on the right physical solution elements, then somehow the user is expected to know what was created and further more go back into the physical artefacts, manually, and edit them (using code snippets and other recipes). It's a job for a skilled expoerienced developer who has detailed knowledge about the how the factory works, and deep knowledge of the solution domain as well.
It is the output of the recipes (source code) that maintains the *state* of the service being defined, and therefore it is difficult to re-configure the state of the service with automated tools. So to be clear, there is a 'solution model' maintained in the source code. But there is no provision for reverse engineering the 'solution model' source back to a logical 'problem model', that can be modified and re-generated easily by the factory user.

This class of factory is able, to describe the work products (physical artefacts) of the factory, and the assets that create them (tools, recipes etc), but they offer little formal schema of the factory or visualisation of that schema, and rely upon written guidance as a means to relate one work product to another.
In other words, there is no 'problem model' of the product of the factory. There is a 'solution model' persisted in the physical solution structure and its artefacts, and the factory user is expected to be very familiar and concerned with the physical model. This design thererfore only offers very little abstraction of the problem space.

A Different Approach

An approach which is more in-line with factories of the future, and one the EFx factory took to the same problem domain, is to use a model driven approach to model the service and its variability points from end-to-end. This therefore, offers an abstraction of the problem domain to the factory user. The advantages of this approach are numerous:

The Problem Model

The EFx factory employs a runtime model (a Meta-Model) of the 'product' and its' work products that the factory produces, as well as the architectural relationships between them.

The factory is then able to display a logical 'view' of the things (work products) it produces (a product model) in a hierarchy that describes how they are composed and related, offering implicit guidance on how to construct and compose them.

Separating the Implementation from the Configuration

In addition, the EFx factory offers an extensible plug-in mechanism for 'Artefact Generators' which are used to generate the source artefacts that are represented by the model (once the model has been configured).

The model describes the architectural make up of the service and the relationship of the components within, not the technology implementation of it.

Technology implementation is cleanly separated by this design, and only applied (postponed) till when the factory user is ready to select a technology based upon changing requirements of the solution.

It is these 'Artefact Generators' (seperate specialised pluggable packages) that determine how best to write the source artefacts based upon the current configuration of the model, and in all but the rarest cases, these artefascts will require no further editing by the factory user. In fact, in this design the factory user, performs little if any coding. The coding know-how of the solution domain is encoded into the 'Artefact Generator' itself.

Seperating the Logical from the Physical

Further advantages of this approach are:

The factory user no longer needs to be concerned about working with, or configuring, the physical artefacts the factory (artefacts generators) produce, nor modifying them.

The factory maintains a logical model of the product and its work products, and this logical model maintains a runtime mapping to the physical structure of the solution. The factory is able to determine where the generated artefacts are placed in the physical structure.

The factory can do this because it attributes meta-data properties from the model to the physical solution structure containers (Solution Folders, Projects, Files), designating them as a specific work products in the product model.

Therefore, since the this product model provides a seperation and mapping to the physical artefacts that are produced, so the solution structure can be independently manipulated (re-structured) to define a suitable physical deployment model of the product, rather than attempting to describing the logical structure of it. The advantage of this is clear.

The mixture of these two concerns (logical structure vs. deployment structure) often leads to poorly named and structured solutions, and much debate on what physical structure to use per solution/project/developer. Now, you can address correctly the concern of the deployment structure of the solution - independently from the logical structure of the problem.

You can see the relationship of the Logical View and the Physical View of the product of the factory in the picture below.

Figure 1. Illustrates the 'EFx Solution Explorer' logical view of the solution, and its relationship to the physical structure of the solution in 'Solution Explorer', and meta-data attributed to the work products of the factory.

Architectural Correctness and Guidance in Context

The fact that the service has been modelled from end-to-end and the variability of the problem domain is configurable in a persisted model - offers additional advantages:

The model already describes the relationship between the logical concepts of the architecture, and therefore can validate them at an appropriate time in context with each other, preventing invalid architectures and illegal configuration of the work products.

The user is no longer expected to know how to compose a single work product (with a set of related recipes), since the model describes and maintains the configuration of the work product. The selected specialised 'Artefact Generators' determine the correct and most efficient means to construct it, and create the physical representation of it.

The work products can be re-generated over and over again by re-configuring the model, producing reliably, highly predictable and architecturally correct artefacts every time.

For more details on the other features of this factory and their advantages see the EFx Architectural Guidance Software Factory article.

The EFx Architectural Guidance Software Factory - Recently Published Article

jezzsa — Tue, 25 Jul 2006 23:35:00 GMT

Just a link to the article, as it does'nt appear in the blog roll: http://blogs.msdn.com/jezzsa/articles/677177.aspx

The Web Service Software Factory (Service BAT)

jezzsa — Sun, 09 Apr 2006 20:41:00 GMT

Very recently p&p announced the 'Web Service Software Factory' (formerly the 'Service BAT') to the public community (official announcement here). The name is still not nailed down yet, but its drawing in close.

A number of us in the community have been busy helping the p&p team refine the requirements and design of this guidance package and basically representing you guys - the community, in shaping this offering. The outcome of this project is set to be an exciting offering from p&p and a god send for developers building services today.

At present the offering is in its infancy, but rapidly progressing. We are all hoping that when finally released this package will provide a huge step towards proving a factory for our development community that exubes automation, guidance, patterns and tools to speed the construction of services.

It's quite an interesting position to be in with both authoring the EFx Factory and advising on the p&p 'Service Factory' project, since they both share the same overall vision. However, they do differ somewhat in their scope, approaches and end goals. The EFx Factory has been a gradual evolution from its beginnings as an application framework on .Net 1.0, into a automation toolkit and now a software factory on .Net 2.0. EFx provides more of a model driven approach to constructing Service Oriented applications and services. It is geared towards providing more architectural guidance, abstractions of the solution and separation between design and technology implementation and is based upon creating components on top of an application framework, (itself based upon Enterprise Library).

It is yet to be seen what future direction the 'Service Factory' will take, which will be further influenced by the wider development community now. However, the EFx Factory has been useful in providing an alternative viewpoint to the problem space overall, and although not directly used in the development of the 'Service Factory' offering, EFx remains as an influencer in its further development.

There are a number of key influencer's in this project (Jason names a few here), that provide various key parts that the toolkit covers, and these guys are all providing valuable input into the project - so check out what they have to say.

Edward currently owns the web space on the details of the 'Service Factory' at the moment, and is going to provide many more details in the future. So no point me repeating them for you - hook into his blog.

The EFx Software Factory

jezzsa — Mon, 06 Mar 2006 21:44:00 GMT

Its been a while since I talked about progress on EFx and its factory.

We have got to a point now where we can start to demonstrate the factory and how it operates. We still have some way to go to complete the implementation of the finer details, but a picture speaks a thousand words, so I am now in a position to show you some of what it looks like currently.

(I am gonna do this in several articles, so we are just gonna start with the big picture stuff first)

Factory Background
(Thanks to Edward for reminding to put his section in here - duh!)

If you have been following the articles on EFx earlier on here you'll know that EFx is an Application Framework for building enterprise distributed applications and services. In actual fact, like most frameworks, EFx helps builds 'baseline' implementations. That means it provides all the frame-working code and the common stuff developers need to build a real implementation. EFx has been successful in helping provide the architectural structure and class libraries to enable rapid and predictable development of these types of applications and services. One of the advantages of this framework is that it leverages Enterprise Library under-the-covers to provide access to common service providers and application blocks. EFx also provides a number of extensions to Ent.Lib to 'round out' the functionality it provides.

So, one of the major issues with EFx (as with any other framework) from the practitioners point of view, is that they had to learn quite a few new concepts about using a framework, and actually crafting the architectural solution structure (solutions, projects, source files etc) was pretty tedious and of course error prone. Not to mention if you changed the name of anything, you had to update a number of source code artefacts all around the place. These issues are not specific to this framework at all, but in fact any Enterprise solution development, with or without a framework in place. The other issue, which was a side effect of using a framework is that the code that the practitioners have to write starts to become very boiler plate like, and again tedious. (Since many of the coding concerns have been absorbed into the framework and architecture, that's the point after all).

So we decided to create an automation toolkit over the framework that builds these apps and services using design surfaces and meta-data definitions (Modelling), that outputs the source code artefacts (Automation) to free the developer from learning all its intricacies (Abstraction) and leverage domain specific knowledge and experience in areas where technology is key to providing a resolution (Extensibility). Our timing was impeccable since at the time of this creation, the Guidance Automation Toolkit (GAT) and the Domain Specific Language (DSL) toolkit were under construction too.

So by combining elements of those toolkits and some other select features of Visual Studio 2005, we have created an 'Architectural Guidance Package' - a 'Solution Development Construction Toolkit' for creating Enterprise Service Oriented Applications and Services all within your favourite development tool - Visual Studio.

Factory Principals and Goals

The guiding principals and goals of the factory are many, most driven from practitioners using EFx before it became automated, and some driven from the new 'Software Factory' initiative emerging within Microsoft. I'll list a few here so you get the basic idea:

Automated Guidance

Avoiding the 'Visual Studio Stare'.
Provide an obvious means of starting with solution development design process without a requirement for up front detailed design documentation.
Knowledge transfer from the guidance to the developer throughout.
- Allow the knowledge that went into the creation of the guidance emanate through as learnings to those using it.
Make solution development a creative, visual, repeatable, predictable and enjoyable task.
- Provide guidance that models human solution development, so that it 'feels right' and follows a predictable and familiar path - discoverable.
Allow solution development to evolve as the requirements for the solution evolve - agility.
Capture the design in artefacts that remain relevant throughout the solution life-cycle.
- Focus upon capturing the intent, the assumptions, decisions 'the unquantifiable experience' of the implementer.

Abstraction

Allow the developer to maintain a holistic 'point of view' of the solution, so they don't become deeply focused and entrapped by specific technology implementation and its intricacies and constraints.
- It is solution development after all!
Provide design artefacts that capture the intention of the design, at a sufficiently high enough detail to avoid technology constraints.
Let the tool do the hard work, automate where possible, create where not.

Extensibility

Leverage and extend existing familiar domain specific languages where possible and appropriate
Don't create yet another unfamiliar experience.
Integrate with other specialised guidance toolkits that focus on a specific pattern or technology, and leverage that guidance
Don't attempt to specialise in all areas, or encapsulate all aspects of solution development.

Customisation

All guidance artefacts should offer a level of extensibility and customisation to suit a variety of styles, processes and organisations.

Together these guiding principals have driven the development of this factory towards the goal of automating and guiding the solution architect and developers through most of the aspects of creating enterprise applications and services.

It should be mentioned at this point that due to the issues discussed in File | New | 'Blank Problem' that we are not attempting to automate everything - don't fret. We recognise that the one thing we cannot automate today is the business logic of the system, for this we allow the developer to still write code that does this, but for all other aspects which amount to mostly architectural concerns, we describe these at a higher level of abstraction and therefore are able to provide an guided automated solution.

Design Time Experience

The factory provides a design time experience that is intended to be familiar to the user, by utilising the design tools available in Visual Studio today.
It 'configures' Visual Studio to provide a customised environment that enables the development of solutions specific to the domain of constructing end-to-end enterprise applications and services. The toolkit does this by leveraging several technologies that are integrated into Visual Studio including the following:

The Guidance Automation Toolkit (GAT). GAT is built upon Guidance Automation eXtensions (GAX) which provides a foundation for defining recipes and executing custom actions within the Visual Studio IDE.
The Domain Specific Language Toolkit (DSL). The DSL toolkit enables the creation of domain specific languages to model software concepts, easily.
Visual Studio Application & System Designers. These designers enable architects to define systems based upon the high level components of a system and allows the definition of deployment settings and constraints that can be validated at deployment time. The designers provide a visual representation of the components and the relationship between them, and provide several different viewpoints of the system and how it is deployed.
The Managed Package Framework. This framework allows custom packages to be built and provide Visual Studio integration elements such as custom commands, menus, editors and tool windows.

Generation Time

Since one of the objectives of the factory is to reduce the maintenance/improve the quality of the code it generates, and provide reusable components, the factory generates source artefacts (applications & services) that are built upon the application framework foundation (EFx).

These source artefacts are created by the factory from modelling the domain of enterprise application and service construction, using specific designers and specialised tools, that generate the solution and source for the layers and components in that solution.

EFx - An Architectural Guidance Package

jezzsa — Thu, 10 Nov 2005 00:10:00 GMT

Its flattering to hear from a well respected authority and author in the field of Software Factories, that EFx and its toolkit is in fact a Software Factory.

Now, I don't fully understand what it means to be a whole software factory, even though I understand the initiative very well.

(Yes - believe it or not, reading the book doesn't necessarily mean you get a full definition of one from it - its over 600 pages long. Full respect goes to the authors though, they have a whole industry to sway towards this thinking, and a simple 2 paragraph explanation ain't gonna do it for most people). The webcast is more concise about, it and to be fair.

Now from that video, the combination of EFx and the toolkit fits the definition very well. But then again the definition is pretty broad, so many people will be happy to know they have factories already too!

Most of my customers don't know much (more like - never heard of) Software Factories either, so it seems pointless to describe this framework and toolkit as one. Instead, after a little discussion with Wojtek today, and full thanks to him, we have decided that this is better termed a 'Architectural Guidance Package'. That is much easier to explain to customers in 60secs than what is a software factory. I am finding that describing a software factory sufficiently, often leads to a long monologue about the history of software and how its soon to be revolutionised. Its takes to long - no matter how interesting it actually is.

'Architectural Guidance Package' is much easier to explain since the words speak for themselves. So we are keeping it.

EFx - On the face of it

jezzsa — Wed, 09 Nov 2005 14:57:00 GMT

Its a pretty exciting time at present for the future of EFx and its toolkit.

We are developing a toolkit for the framework, the one piece that has been missing for some time. What's really exciting is the vision that this toolkit delivers. Its a bold one, but extremely powerful.

With this toolkit, we are able to create an entire Service Oriented Enterprise LOB solution that includes many client applications and/or services. We will be drawing the applications on diagrams and using recipes to create all the necessary project artefacts required by the solution. It is possible to draw a whole layered system from Service Interface through to the Data Access classes on a single diagram. The diagram models the interaction of the method calls between the layers and any programming requirements of the method calls (sequencing, transactions, authorisation, entity mapping, aggregation etc).

The diagram will generate the underlying code, and allow extension points for the developer to customise that code, to add finer granularity. Of course the code will be leveraging features of the underlying framework, and therefore all infrastructural services of Enterprise Library and extensions to that. The wizards will allow the developer to define the format of method calls without having to write the code. Its kind of like: coding by policy.