Published on MSDN here
The Enterprise Framework Factory (the EFx Factory) is an architectural-guidance software factory. It provides Microsoft partners with a software-factory implementation of the Microsoft Distributed Architecture for .NET Applications for building applications and services on the .NET platform.
The software factory is founded upon an application framework (Enterprise Framework) that combines Enterprise Library, a number of the Microsoft Application Blocks, and best practices in .NET solution development, and supplements those with specialized class libraries for creating service-oriented, enterprise-level applications.
EFx has had a long and interesting evolution from its beginnings as an Architectural Application Framework to a Software Factory today. During this evolution, it has encountered a number of key architectural and implementation issues that have been systematically addressed and resolved using a software-factories approach. Most of the issues faced represent challenges that most enterprise-level projects still face today.
This article describes the evolution of this factory, the challenges it has faced, and the features that this factory has developed to address them.
IntroductionEarly BeginningsEFx Factory EmergesConclusion: A Factory Foundation
Solution architects and solution developers face numerous challenges when defining and constructing enterprise solutions today—for example, how to:
The EFx Factory has addressed these and many other concerns of constructing enterprise solutions. The EFx Factory has been developed by Microsoft Consulting Services and provides a comprehensive architectural framework, visual designers, guidance, tooling, reusable assets, and run-time environment for creating enterprise-strength service-oriented applications and services.
The factory has harvested and encapsulated within itself a wealth of solution-domain experience, best practices, and proven patterns of implementation from many real-world enterprise-level solutions using the application framework. It provides a powerful, real-world software factory for creating service-oriented applications and services.
EFx began as an architectural application framework built specifically for Microsoft solution integrators more than three years ago on version 1.0 of the .NET Framework in New Zealand.
The original intention of this framework was to provide solution integrators with a structured architectural framework for constructing baseline enterprise applications and services using the best .NET Framework practices, patterns, and reusable application blocks available at the time from Microsoft. This framework was to enable rapid and consistent development of such enterprise-level solutions. The framework primarily addressed the architectural, implementation, and deployment concerns of structured, layered, service-oriented applications.
The challenge still faced by Microsoft solution integrators was that, while Microsoft provided a number of Application Blocks that solved specific problems in enterprise-application design (such as Data Access, Configuration, Exception Management, UIP Design, and so on), these application blocks together did not form the basis for a complete, cohesive solution framework. The reason for this was that many of the components of the distributed solution architecture were not available as application blocks, and solution integrators were left to fend for themselves in these areas. Also, understanding the capabilities of the available blocks that were provided, and how to leverage and combine them successfully, proved to be a huge challenge for most development teams, and many overlooked these blocks in favor of home-built solutions.
The guidance for implementing such frameworks was present in written form in numerous valued best-practice guides from the Microsoft Patterns & Practices team. This rich guidance instructs the reader how to implement these patterns and solutions. However, at that time, no practical implementation of such a solution framework was provided by Microsoft. Many solution integrators faced the challenge of writing their own ad-hoc frameworks—many with very mixed results, and many with little sustainable success.
Enterprise Framework (and, later, the EFx Factory) was such an implementation that embodied the application blocks and written guidance into an Architectural Application Framework. The framework drives the user towards creating architecturally sound and structured applications and services that conform to established patterns, practices, and standards for Microsoft .NET solutions.
EFx has systematically evolved with time, knowledge, and technology to adapt to current best practices and principals, implementing new platform technologies and patterns as they develop.
EFx leverages Enterprise Library (Ent.Lib) and many of the common scenarios it addressed. It provides a thin abstraction layer over Ent.Lib to unify its interfaces, hide its dependencies, type its parameters, and extend and abstract its capabilities. It also extends Ent.Lib to include a number of more common real-world providers and extensions not shipped with Ent.Lib. It also supports the UIP block (currently not part of Ent.Lib). In addition to the class library that developed from this, EFx provided a set of infrastructural services (logging, security, configuration, and so forth) built upon the blocks that address the crosscutting concerns of the layers and components in the solution. Most all of these infrastructural services are driven by Ent.Lib configuration.
Figure 1. The components of the EFx application framework, illustrating the abstraction of Enterprise Library, extensions to it, and an additional class library
Each layer of the architecture has well-known concerns and responsibilities to fulfill for the solution as a whole. These were addressed by the framework and provisioned by the framework for the developer to utilize. The implementation details and variability of these concerns were identified and handled by the framework, so that the developer would only need to make simple calls to the framework to fulfill these concerns. The variability was provided by the framework in the form of typed parameters, attributes, and so on. The abstraction of the implementation details and technologies was provided by the application blocks and its run-time configuration. Therefore, as long as the developer made the right calls, in the right layers, and with the right parameters, the requirements from these concerns were fulfilled.
However, although facilitated by the framework, run-time configuration, and code templates, enforcing that these concerns were actually addressed was not directly enforceable without human intervention (in other words, code review, or policy), and it was left in the hands of the developers.
Connected systems communication was another challenging and evolving area, as the industry moved towards SOA. Originally (circa .NET 1.0), the framework implemented .NET Remoting as the communication channel between the client and server components. Later, this evolved into ASMX Web services and, again, to WSE secure Web services.
The solution architecture had to change, and the implementation of the framework class library had to evolve to accommodate more SOA patterns. The impact of this on the implemented solution was minimized because of the abstraction that the framework provided (but not eliminated). However, the technologies used in the implementations (ASMX, WSE) were still prescribed by the framework, and using other technologies fell outside the scope of the framework's usefulness.
Figure 2. Early EFx architecture illustrating a hybrid application and related service exposed using Web services, with prescriptive technology layers
At this stage, the framework was leveraging Enterprise Library version 1.1 and using the UIP Application Block version 2.0. Getting to grips with the framework from a developer's perspective required a rapid ramp-up on its implementation, learning its architecture, patterns, and variability points. For some, this familiarity took some time, and development teams found it hard to prioritize this over starting on their implementations.
Despite the evolution and changes in the architecture and implementation patterns, the development experience was rather similar. The framework demonstrated strong patterns of implementation using a Reference Implementation (RI) as an example, and providing written step-by-step documentation on how developers should create their own customized components at the various architectural layers to compose a cohesive solution framework.
At the time, on the available technologies (Microsoft Visual Studio 2002, 2003), the only automated code-development assistance that could be reasonably achieved was the use of solution and code templates provided by Microsoft Visual Studio Enterprise Templates. This technology proved to be very challenging to work with and get productive results, and was abandoned.
Many of the components in most of the architectural layers followed well-defined patterns with little variability (business-logic components representing the largest variability). However, most other components formed very similar implementation patterns with little variability to the configuration of the components. Because of this, many developers generated a significant amount of bugs inadvertently from copying and pasting boilerplate code from other components into new components, and tweaking certain parameters. The creative-design aspect for the developers had decreased dramatically, and they now spent most of their time arduously maintaining and configuring boilerplate code for many of the components.
Despite this, it was becoming apparent that this undesirable side effect of using a specialized structured architectural framework was that the code started looking very predictable, consistent, and hence tool-able. The potential for modeling and automation was apparent, and framework developers desperately needed to be more productive.
At this key period in the evolution of the EFx framework (circa early 2005), a number of key technologies and tools that looked to solve some of these problems were emerging from Microsoft.
Most notably among these were: the Visual Studio 2005 extensibility model, including the managed package framework; Visual Studio Modeling (Distributed Application and System Designers); the Guidance Automation Toolkit (GAT); and the Domain-Specific Language Toolkit (DSL). These tools filled the automation gap and lowered somewhat the extremely high bar for building custom modeling and templating solutions.
The EFx Factory evolution began and adopted these technologies to form the basis of an Architectural Guidance Package (AGP). This AGP used GAT recipes and template assets to create the structure for solutions that described the physical components and layers of the applications and services. GAT provided the code templates and recipes to implement and manage the artifacts. The DSL toolkit provided the visual designers to model the logical architectural components.
Initially, the objective was to model the implementation of the architectural layers, then analyze and constrain the variability of the implementation to a finite set, and define a meta-model to describe the variability points of the solution. (A meta-model can be thought of as a model of something that is described using metadata properties and behaviors.) Once this model was derived (by identifying the components of the product the factory was building), the artifacts that those components were composed of were identified. Finally, the assets (recipes, model, and tools) were assigned to create and configure those artifacts. A set of common user-interface elements (visual designers, tool-windows, wizards, editors, properties window) were provided to the factory users to configure the meta-model quickly and easily.
In addition to this, automation was used widely and systematically to handle changes to this configuration across the solution. For example, namespace changes in one part of the solution would cascade throughout all parts of the solution, as the factory (intimately familiar with the meta-model and the solution artifacts associated to it) would automate these changes for the developers.
The ultimate outcome of this approach was to automate the code generation of these low-variability components from this meta-configuration. The factory provides the code templates that would intelligently implement the components using the most appropriate coding patterns optimized for the current configuration. These coding templates were authored by harvesting common coding patterns from existing solutions where this variability had already been implemented. The high-variability components, such as business-logic components, could not be modeled appropriately to accommodate the variability they presented. Instead, these components were represented in the models and integrated into the layers of the solutions, but implementation of them was directed by the factory to the developer. The crosscutting concerns that these components must address, and their variability, was still to be modeled and implemented by the tooling.
In this way, the majority of the implemented source artifacts (source code, configuration, and so on) could be generated consistently, predictably, and reliably by the factory, while focusing the developers on the business requirements of the solution. The visual models provided a simplified (abstracted) view of the solution that removed the details of the technologies used to implement them, and focused the developers on the architecture and specification of the products they were building. This approach reinstated and emphasized the creative-design aspect for the solution developers.
Capturing and embedding the extensive, collective domain experience directly into these models and tooling removed the requirement for the factory user (or solution architect/developer) to possess the highly specialized skills required to implement the technologies used to implement the solution. That knowledge was baked into the tools—driven by configuration.
This approach enabled high productivity gains and consistent, quality output.
The factory also refined the solution architecture of the framework and unified the definitions of applications and services for SOA. Both applications and services are essentially fronted by an interface of a kind and are composed of business components (logic, rules, and entities) that use service agents and data accessors to connect to external services or data sources. This makes the design experience similar for both applications and services to developers and many of the components reusable among them.
The factory uses a subsystem concept (not a reference to UML) to represent a logical partition and grouping of a set of related components (from different layers of the architecture) into separate vertical "slices." Any single application or service is composed of one or more subsystems, each with its own interface. These are logical partitions and are intended to split the solution into manageable, versionable, deployable pieces. Typically this is done by feature set or scenario, but can also be done by product version, as examples. Components in one subsystem might take dependencies on components in another.
Figure 3. The unified definition of applications and services, illustrating how they are composed of subsystems
An e-procurement system could be, for example, divided into Products, Orders, Customers, Inventory, subsystems, and so on. Each subsystem would then have its own components at each of the layers, some of which could be shared by other subsystems, and some unique to the current subsystem. They could also be versioned independently.
It is often the case that a common or shared subsystem emerges that could be used by all subsystems.
With the modeling extensions introduced by Visual Studio 2005 (Team Edition for Software Architects) enabling System Architects to define and view a solution from a high-level view, the factory could visually define solutions in high-level terms from their logical components and provide a mechanism to drill down into their internal implementation.
EFx extended the Distributed Application and System Designers by providing additional shapes that represent the subsystems and applications and services created by the factory. From these diagrams, architects or developers are then able to compose their applications and services, and then implement their subsystems. These new shapes were now represented physically with the appropriate solution structure and source artifacts (projects, source files) that the factory created and maintained for the developers.
Figure 4. Extending the Distributed System Designers with additional shapes representing the subsystems of an application (Click on the picture for a larger image)
From these diagrams, the user is then able to drill down into each subsystem and navigate to the specification (inner details) of its component parts.
Figure 5. A drilldown into a subsystem specification, illustrating the model of a subsystem, its layers, the components at each layer, and the relationships between them (Click on the picture for a larger image)
The factory models the internals of each subsystem in a separate diagram (Subsystem Specification). It models the components of each layer and the relationships between them.
The architectural model enforces certain patterns, such as:
By embedding well-defined architectural rules in the model, the architectural concerns of each layer are preserved and enforced. This is then supported by a set of validation rules for the model, ensuring that regardless of how the model is configured it must always be valid before being implemented. By using a small set of shapes (and various abstractions), the model simplifies the possible set of interactions to a well-defined set of rules, which are indicated to the user by appropriate UI cues. This makes creating the "specification" of the subsystem a quick, easy, and well-guided exercise.
Figure 6. Configuring the subsystem specification, illustrating the various editors and tool windows to expose the meta-model of the selected shapes on the diagram (Click on the picture for a larger image)
The model also includes containers for representing and configuring Business Entities and Message/Data Contracts. These containers reduce the clutter on the diagram, while providing a means to define and configure these entities using several user-interface elements, such as wizards, the Properties window, the Model Explorer, and specific UI editors (tool windows).
The relationships between shapes on the diagram, the connectors, represent method calls from one component to another. When users connect the shapes together, they are prompted to define a method-call mapping, defining how the parameters and return values of each method/operation are mapped, as well as how different data types of the parameters and return values are mapped in the call. These mappings are also stored, so they only need to be defined once. In addition, the factory defines a simplified, extensible data-type model that is a subset of XSD types (string, integer, GUID, float, double, and so on). This allows the factory to automate the mapping between entity data types easily.
By default, the factory does not model the actual business-logic code, which is assumed to be too complex to model for all possible domains. Instead, the factory provides well-defined source-code placeholders (method stubs) where solution developers can implement their business logic, which is not interfered with by the factory. The model provides the navigation to this source code and keeps it synchronized with what methods have been customized. This source code is then "weaved" into the final source, which decorates it with the crosscutting concerns represented in the model.
The model used to represent a subsystem becomes the actual specification of the subsystem, representing the current documentation of the subsystem. It always represents the actual implementation of the solution, because the source artifacts are generated directly from this model. This is quite deliberate.
The configuration of the subsystem specification—the meta-model—is then transformed into source artifacts, together with the custom handwritten code, and then added to the appropriately assigned physical containers (Solution Folders, Project, Project Folders, Files) of the solution. Each one of these containers has an assigned role in the factory implementation, and is represented as such in the factory meta-model.
Figure 7. Snapshot of the solution structure of a service, showing the physical project container destinations for the generated source-code artifacts
Source artifacts are generated using customizable text-templating technologies (such as T4), which are written to generate source artifacts depending on the configuration of the meta-model of the subsystem. It is within these code templates that the variability of the meta-model is encoded into optimized patterns of implementation. These templates are easily customizable by the end user.
The source-code output of these templates is built—and runs—upon the underlying application framework libraries. The framework provides a useful abstraction of the .NET platform, reducing the quantity and complexity of the generated source code, and increasing the quality and configurability using the benefits of the application framework and underlying application blocks (Ent.Lib)—much the same as it did before the factory evolved.
Previously, the EFx application framework (prior to the factory) had always had to prescribe a specific technology implementation used at each of the layers of the architecture (Service Interface, Data Mapping, UI Process, and so on). With the factory now, the separation of the application or service specification from the technology (or patterns) is possible. The EFx Factory allows the factory user to postpone the decision on which technology to implement during the design phase. The user can choose and configure a different technology at any stage during the design phase.
One can then install and plug-in an appropriate technology provider to implement that layer. These plug-ins can themselves be other factories ("factorettes"), guidance packages, or designers.
Figure 8. EFx architecture, separation, and unified model of applications and services, offering a choice of implementation technology at each layer
By providing an extensibility mechanism at various points in the architecture, the factory delegates the configuration and implementation of the specific technology to an external plug-in. The factory then loads these plug-ins and integrates them seamlessly into the design experience for the factory user.
The ultimate purpose of these plug-ins is to provide the source artifacts that represent the implementation of a specific technology or pattern. For this reason, these plug-in assets are termed "Artifact Generators" (AGs).
AGs are authored by technology-domain specialists who have repeated experience in creating appropriate solutions with these technologies. These domain specialists are in the best position to provide these reusable assets by embedding their solution-domain experiences and knowledge within them (models, code templates, and so on). Ordinary factory users, who would not necessarily possess the specific technology skills or knowledge about them, can then apply these assets effectively.
The EFx Factory defines a number of extension points in which these AGs can be integrated, and each is designated a set of architectural responsibilities. These integration points give the factory users a chance to pick and choose the right technology for their current solution requirements.
Figure 9. Artifact Generators (AGs), the layers for which they are required in a service, and some possible common types of technologies or patterns that they would implement
Typically, an AG itself provides a relevant abstraction of a particular technology or pattern, and models the variability of that technology or pattern using its own meta-model. It then generates optimized source artifacts that are configured by this meta-model. The meta-model provided by the AG is seamlessly integrated into the factory's designers and presented to the factory user. This allows users to specialize their specifications to suit that chosen technology. Because the AG provides its own meta-model abstraction, this metadata is combined with the metadata provided by the factory on the shapes in the specification.
Figure 10. Illustration of the metadata configuration for the chosen technology—in this case, the "Simple ASMX" AG—and how that metadata is integrated into the specification of the selected element on the diagram (Click on the picture for a larger image)
An AG is also permitted to provide a unique design experience (models, designers, recipes, wizards, custom GUI) that can be integrated into the factory's designers. For example, from the shapes in the diagram, the user can "drill in" to display custom GUI provided by the AG to represent it abstraction of that shape. The factory also provides an extensibility mechanism that allows the AG to provide contextual commands on the design surfaces of the specification that can perform custom actions for the AG. For example, an AG providing the service-interface implementation could provide a contextual command on the diagram to import Service Operations and Data Contracts from a WSDL.
AGs also participate in specification validation. At various stages in the design of the solution, the models used to describe the specifications are validated to ensure architectural compliance and completeness. Because the AGs provide their own meta-model configuration, they must participate in this validation step. Not only are they expected to validate their own meta-model configuration, but in some cases they can have a dependence on the meta-model configuration provided by the factory. For this reason, during the validation and the code-generation phases, the AG is provided with the combined meta-model configuration of the specification (AG + factory configuration), so that the AG can walk the meta-model and validate its configuration in the context of the whole specification configuration.
Validation errors are presented immediately as either build warnings or errors that cannot be ignored by the developer. The value of this validation mechanism is that many of the manual errors of implementation that were rife in code-based development, prior to the factory, are now detected in the design phase of the solution. Using architectural models, the developer can be guided towards correct implementation patterns before committing design changes.
Keeping naming consistent and synchronized between related solution artifacts becomes very challenging. Changing key names and namespaces during development cycles has a significant impact right across a solution structure. Code namespaces, project namespaces, XSD schemas namespaces, assembly names, and source-artifact names change with time. Dependencies on these names must be updated to reflect a name change. On the scale of enterprise-level solutions, the impact of a name change is significant enough to prohibit it in some cases, if performed manually.
The EFx Factory addresses these challenges in a number of ways. Firstly, the factory provides a meta-model (or factory schema) of the solution in which all relevant source artifacts are represented and have appropriate metadata associated with them, such as their names and namespaces. This metadata is used to provide the default values for naming child or related source artifacts, as they are generated or edited. Secondly, the factory provides templates and recipes that create source artifacts from this model and its metadata. Thirdly, the factory manages changes to this metadata, and automates changes to it automatically on behalf of the user, and cascades that change to all child and related source artifacts in the model.
Another major challenge that solution developers face when constructing enterprise solutions is defining suitable solution structures, projects, and file-naming conventions. In enterprise-level solutions, a single application or service is normally composed of several projects, and many different components taking many different dependencies. Every organization or development team has a preferred structure and naming convention for its solutions.
A solutions' physical structure should define the deployment of its components. Often, development teams want to visualize a logical abstraction of the solution to ease understanding of the solutions' components. Often, they also attempt to do this in the physical-solution structure. These two driving forces (deployment model vs. logical model) oppose each other, and the development environments such as Visual Studio don't provide such abstractions natively. Consequently, many solution structures mismatch these two requirements, and neither is satisfied fully.
The meta-model that the EFx Factory provides for the solution is a logical abstraction of the physical-source artifacts and solution structure on disk. This abstraction is powerful. It allows factory users to arrange or name the physical-solution structure in any way that makes sense in their deployment model, perhaps to suit an organizational guideline or deployment platform. The factory-provided logical meta-model of the product it is creating maintains a mapping to those physical artifacts. The logical model can be manipulated or viewed in any way the factory user sees fit, without rearranging any of the solution structure. The opposite is also true. Although the factory provides a recommended structure of its physical artifacts laid out on disk, the factory user is free to modify this physical structure, while not affecting the logical model of it.
Figure 11. Illustration of the relationship (mapping) between the physical-solution structure (Solution Explorer) and the logical product model (EFx Solution Explorer) (Click on the picture for a larger image)
The power gained here with the logical product model for users is that they can now focus upon the architectural elements of the product they are creating in any way they want (from any angle or viewpoint) without changing its physical structure. The architectural elements in the model can be filtered, sorted, rearranged, and zoomed in a number of dimensions (by user role, architectural role, deployment package, task-completion status, and so forth). This is possible due to a mapping between the physical-solution artifacts (represented by Solution Explorer) and the logical product model (represented by "EFx Solution Explorer"), so users can switch back and forth between the two views to suit their purposes at the current time.
The EFx Factory is, in essence, a horizontal factory that provides a means to model baseline applications and services, and to specialize those towards a specific set of business requirements. Additionally, it provides an extensibility mechanism to pick and choose the technology providers (AGs) used to implement these requirements. With this unique combination of abstraction, specialization, and extensibility, this factory enables the creation of variants of itself (horizontal or vertical) that can be tailored towards specific solution and technology domains.
The core factory (without the plug-ins) can be packaged separately and delivered into to a general solution-development environment. However, certain combinations of plug-in artifact generators packaged with the core factory will compose a factory variant tailored to a specific solution domain. Coupled with the fact that the core factory is itself customizable (recipes, templates, and so forth), it can be seen that this factory can be tailored to address a wide variety of horizontal or vertical solution domains.
The best software factories emerging today are those that capture and embed best-practices and implementation patterns developed by those who have repeated experience in the solution domain that the factory addresses. These factories then use this encapsulated knowledge baked into their automation and abstractions to achieve the highest levels of productivity for the factory users. Architectural factories do this by providing abstractions of the underlying .NET platform and defining architectural structure to products they create.
The EFx Factory is founded upon a specific, evolved application framework that is built upon best practices, Enterprise Library, and other application blocks, and thus makes best use of the guidance building blocks and patterns present from Microsoft in use in real-world solutions today. The EFx Factory focuses the solution architects and solution developers on implementing the things that make these types of enterprise solutions unique—the business requirements—instead of emphasizing the aspects that are the same across all solutions of this type. It does this by guiding its users—using high-level abstractions, automation, and technology separation—to focus their attention upon the business-requirement specification implementation.
By separating the implementation technologies from the specification of the solution, using a pluggable extensibility mechanism, the factory offers the solution architects an appropriate choice of implementation technology for the solution, at the appropriate time.
This freedom of technology choice and when to make that choice, and the ability to specialize the factory towards other specific domains, provide the factory owners the ability to compose a family of related factories that target specific vertical industry or technology solution types.
By providing unique views of the factory's product (logical meta-model), which abstracts its physical-solution artifacts and automates changes to them, the factory provides a high level of usability, productivity, automation, and contextual guidance in creating its products.
The unique features together of the EFx Factory—its use of abstraction, a model-driven approach, an architectural framework foundation, a powerful separation of technology capability, and its logical product model—all go to provide solution architects and developers the tools to quickly construct verifiable, predictable, and consistent applications and services.
Microsoft Consulting Services Application Platform and Development group helps organizations connect people, services, and software by using Microsoft .NET architecture, and establish collaborative processes for custom application development.
For further details, or if you are interested in the EFx Factory solving your enterprise-solution problems, please contact Microsoft Services in your region or email firstname.lastname@example.org.
Jezz Santos is a Senior Product Development Consultant in Microsoft Consulting Services, currently based in Western Europe. He has advised Microsoft partners and customers worldwide, as well as architected and implemented enterprise solutions on the .NET platform since its inception five years ago. He is the creator of the EFx Application Framework and the EFx Factory.
Jezz is deeply involved with several of the early incarnations of real-world Software Factories from Microsoft today, and contributes to Microsoft's Software Factory initiative.
You can find more information about the EFx Factory, its framework, and the principals behind its evolution, as well as other current thinking around software factories, from Jezz's blog.