I recently published an article on using software factories in supply chains for Methods and Tools. Also, many people have asked about the article I wrote about factories vs. MDA for the Perspectives of the IASA.

Irfan Idrees wrote with the observation that while performance analysis and prediction are of critical importance for software developers, the rigorous approaches prevalent in the marketplace have not been widely accepted in the community. Irfan then asked how software factories deal with this issue.

Software factories provide leverage by reducing the amount of variability in application development. With a software factory, one effectively builds a variant of a known application type. All features common to the product family are designed and implemented in known ways. The only variability in a given member of the family is due to the features that make it unique.

Unique features include both fully custom features and variations on known variable features. A fully custom feature is one that has not been predicted or taken into account in any way by the software factory. Variations on known variable features may be either fully constrained (e.g., picking one of five predefined transaction policies from a drop down), or partially constrained, (e.g., writing custom code to complete a template generated event handler designed to work within a predefined event architecture).

To the extent that the performance of that application type is known and understood, then the task of analyzing and predicting the performance of a given variant becomes a matter of analyzing and predicting the performance of its unique features, and of analyzing and predicting the impact of their performance on the known performance characteristics of the application type, such as overall performance, the performance of specific operations or scenarios, and the way those measurements may vary with variations on known variable features.

The CEO of 6th Sense Analytics recently published an article containing some inaccurate statements about Software Factories.

• The article asserts that SFs are about making developers into battery chickens who squeeze out code. On the contrary, SFs are designed to automate the rote and menial tasks that make developers feel like battery chickens, freeing up time for the creative work of software development.
• The article asserts that SFs attempt to solve software development problems by imposing process. On the contrary, SFs seek to enable the project team to be more agile by maintaining invariants to keep the product well formed and the project manageable, so that the overall process can follow any path that the team chooses to take.
• The article asserts that Software Factories are imposed by people outside the team that uses them. On the contrary, they are generally developed by their users and are most effective when they serve as a repository of shared experience.

These misconceptions suggest that the author of the article does not know much about Software Factories, and perhaps has not even read what we have written about them. By themselves, they would not be serious enough to merit the time required to post a reply, as most people who evaluate a technology look at it deeply enough to get past shallow misconceptions.

However, the article goes on to say that SFs “crush the flexibility and creativity needed to build software and meet constantly changing requirements”. This statement is profoundly misleading, and requires a response for the sake of the those who have read or may read the article.

We have published a book, as well as many articles, papers and pod casts describing Software Factories. Reading just a few of them would make it easy to see why this statement is off the mark. Therefore, rather than explain SFs yet again here, I will refer to them. You can find pointers to many of them in my previous blog postings.

I would also like to ask the author of the article to answer the following 3 questions:

1. Which of the following innovations crush flexibility and creativity?

• Compilers
• Class Libraries
• Frameworks
• Patterns
• Best Practices
• Databases
• Form Builders
• Transforms

2. Which of the following practices crush flexibility and creativity?

• Writing User Stories
• Defining Service Contracts
• Refactoring
• Testing
• Configuration Management
• Defect Tracking

SFs are a lot like the innovations and practices listed above in the way they organize and support software development, and in the way they help developers harvest experience and make it easy for others to reuse.

The author goes on to suggest that the key to success is visibility into project status, supported by metrics. I agree and recommend the article that I co-wrote with Marcel DeVries on using Software Factories with Visual Studio Team System, soon to be published on MSDN.

It explains how, by providing a simple but effective architectural framework for organizing a project, called a schema, Software Factories help their developers define better metrics, and how, by providing a framework for harvesting and reusing experience, Software Factories help their developers use metrics more effectively to improve future projects.

This is a response to a blog posting by Tad Anderson, and the pointer to it on the MSDN Architecture General forum.

First, I agree with your observation, Tad, about the lack of support for the Software Architect in VSTS 2005. We did indeed focus on the System Architect not the Software Architect. We had limited resources to allocate to architecture tools, and felt it was more important to support Microsoft’s drive toward connected systems for the 2005 release. As you suggest, however, we do intend to address the needs of the Software Architect in subsequent releases, and most of that support will come from our Software Factory strategy.

However, I disagree with your assertion that our Software Factory strategy, with its reliance on DSLs, will “make a huge mess”. From your comments, I gather that you haven’t read the Software Factories book, or been exposed to the many white papers, blog postings, conference proceedings, articles and podcasts that we’ve produced, since they address the issues you raise at length. I will summarize them here, but I'll also point you to the those other resources, where you learn a lot more about what we’re doing and why.

While it’s certainly true that unqualified practitioners might abuse our technology, there is nothing about our technology that makes it particularly susceptible to abuse. As we know, there is nothing to stop people from writing bad code in Java or from creating ineffective UML models.

However, we have admittedly not done a good enough job of making clear that we intend people to build DSLs primarily in the context of Software Factories. Yes, we do expect them to build standalone DSLs, but the primary goal of productizing the DSL technology was to support factory development. Why? Two reasons:

1)     Factories provide a methodology for determining what to model for a given product family. The key to using DSLs effectively is to understand how solutions are developed for a given problem domain. Factories are organized around a model of the development process for a given problem domain. This model, called a factory schema, is a graph of viewpoints. A viewpoint defines the perspective of a specific stakeholder or a specific role in the project team. As recommended by IEE 1471, and the book Documenting Software Architectures by Clements, et. al., we use the graph of viewpoints to define the product family architecture. We also use it to define the development process, or what Product Line Engineering calls the production plan. Each viewpoint describes the set of workflows used by its stakeholders, the work products acted upon by those workflows, and a set of assets, such as tools, templates, class libraries and guidelines, that support the enactment of those workflows. How does this relate to modeling? The schema shows us what to model. Each viewpoint defines a domain, such as logical database design, user interface process or business rule definition. These domains are aspects or subsets of the larger problem domain defined by the product family. We may only know enough from experience about such a domain to supply documentation assets, such as patterns or practices, to its stakeholders. However, if we know enough about it to model its key concepts, then we can supply more powerful assets, such as a DSL and tools that perform generation, validation, trace, analysis and other forms of automation using models based on that DSL. Of course, implicit in this approach to modeling is the assumption that DSLs are more effective than general purpose modeling languages like UML for model driven development because they are highly focused on specific problem domains. That point is discussed at length in many of our publications, including our Modeling Strategy paper. In summary, a factory contains a model that provides a basis for determining which aspects of a problem can and/or should be supported by DSLs, and how DSL based models should relate to each other and to the work products of non-model driven activities.

2)     Software development is about more than just models. While it’s nice to think that some day, we might develop every part of a system using models, just as we currently develop every part of an application using 3 GLs rather than assembly language, it’s just not feasible today. You can hear more on this topic in the recent ARCast on MSDN. Because we take this pragmatic view, we require factories to provide whatever assets are most appropriate to support the enactment of the workflows for a given viewpoint, and not just on modeling. A modeling tool is just one kind of asset that factory builders can provide to support factory users, but there are many others. This is one of the fundamental differences between Software Factories and MDA. You can read more about the differences between them in the article that I wrote for the newsletter of the International Association of Software Architects.

Regarding the effect of our strategy on industry standards, I think it was inevitable that people would demand more focused modeling technologies. Within the OMG, there is very strong momentum toward domain specific modeling. It just happens to be expressed using UML profiles, which are awkward at best. A large block of OMG members would prefer to see domain specific languages defined using the Meta Object Facility (MOF). In effect, that’s what we’re doing, too. However, we don’t use the standard MOF. Neither does IBM, Sun, or anyone else. Why? Because the MOF maps to the internal architecture of the modeling tool. Experiences like IBM AD/Cycle show that tool internals cannot be standardized effectively, since vendors must be free to design and implement tools in response to market changes, not standards committee proceedings. Based on these experiences, we prefer the standardization of interachange formats – not interchange formats like XMI, which depend on MOF alignment to achieve interoperability, but interchange formats like the ones defined by UN/CEFACT, RosettaNet and HL7 to name a few.

As for learning a new language with every gig, you have to do that anyway, whether you’re using UML profiles, or no modeling language, at all. While the UML defines a set of widely understood notational styles, the meaning of any given profile must be learned. This should come as no surprise. What you really have to learn with every gig is the problem domain, or rather the many subsets or aspects of the overall problem domain, and how they all relate to each other. The goal of software factories is to help solve that problem. The software factory schema provides a way to model all of the aspects and their interrelationships. While the DSLs that people build using our tools are unique, they are usually based on the notational styles supplied by the templates that we provide. They happen to look a lot like UML notational styles, of course, and that’s intentional. We have no interest in reinventing the wheel when it comes to notational styles that have proven effective. By the same token, we have not followed any of UML notational styles that have not proven effective, such as the deployment diagram.

It’s true that that we have not yet aligned Software Factories and MSF. That is currently work in progress. When the alignment is published, I think you will find that the result quite different from RUP and XP. For one thing, the development process in a factory is organized around viewpoints, as I’ve just described. For another, the workflow in a factory is driven not by sequential execution, by pre- and post- conditions associated with each activity. Activities come into scope when their pre- conditions are satisfied and go out of scope when their post- conditions are satisfied. Work can be done in any order subject to these constraints.

It’s also quite true that we need to provide a more complete set of tools and templates to support the Software Architect, as you suggest. That’s what we’re busy doing.

Finally, I should point out that a critical mass of industry experts supports our work.

For example, if you read the Acknowledgements in the Software Factories book, you’ll find that we leaned heavily on the work of Paul Clements and Linda Northrop, and met with them multiple times in person. You’ll also learn that Paul was a reviewer, and you’ll find his endorsement of our work inside the front cover.

If you look at our web site, you’ll discover that Linda Northrop invited me to give the keynote at the SEI sponsored Software Product Lines Conference 2004, giving an overview of Software Factories.

If you look at our blogs, you’ll find that Martin Fowler had some complimentary things to say about our strategy in his paper on Language Workbenches.

It may be worth mentioning, while we’re talking about industry expert response to our work, that we have had and continue to have critical interaction with some of the other people on your list, including Mr. Jacobson, Mr. Ambler and Mr. Evans.

Inside the front cover of the book, you will also find endorsements from Doug Schmidt, lead author of Pattern Oriented Software Architecture, Volume 2 (POSA2), David Frankel, author of Model Driven Architecture, Krzysztof Czarnecki, author of Generative Programming and leading voice in the industry on feature modeling, and John Crupi, lead author of Core J2EE Patterns and the writer of our foreword.

There are many other industry experts who continue to review our work, share their thinking with us, and generally advise us in our efforts to craft a pragmatic approach to software development that is firmly rooted in proven practices from software product lines, model driven development and service oriented architecture. You can read what many of them have written in the papers they submitted to the International Conference On Software Factories at OOPSLA 2005.

Keith Short, Mauro Regio and I have agreed to write a new book called Software Factories Applied. Our goal is to write a short book for practitioners that complements the theoretical foundation established in the original Software Factories book by showing how to build a software factory using currently available technologies, namely the DSL tools, the Guidance Automation Toolkit and Visual Studio 2005 Team System. A key topic covered in the book will be how to build a software factory schema, which is the model at the heart of a software factory. Our plan is to walk through a complete worked example from end to end, showing what we did at each step. It will probably be based on a Business Collaboration Factory that generalizes the HL7 factory that Mauro and I developed. You can read about the lessons learned in the development of that factory in this paper that we contributed to the International Workshop on Software Factories at OOPSLA 2005.

In this posting, I continue addressing issues in the recent blog posting by Grady Booch. As I noted in my previous posting, the debate up to this point has been about the suitability of the Unified Modeling Language (UML) as a medium for model driven development.

However, that is not really the question we should be debating.

Instead of asking whether or not we should use our existing hammer (i.e., UML) to hit everything in sight, even things that don't look like nails, we should be asking what it is that we want to accomplish (i.e., model driven development), and what it looks like when done properly. Only after those questions have been answered is it appropriate to ask what languages, techniques, tools or other resources we should use.

Why? Because we cannot evaluate the suitability of UML or any other technology apart from understanding what we hope to gain by using it and how we will use it to realize those gains. To make this point more bluntly, requirements should drive technology decisions, not the opposite.

While this should be patently obvious, the debate about UML vs. DSLs continues to put the cart (i.e., modeling language technology) before the horse (i.e., requirements for model driven development). The debate should instead be about the methodologies that are proposing to use these modeling language technologies for model driven development, since they are both key sources and key repositories of those requirements.

This connection between methodology and modeling language technology is no accident. There is a finite set of practices for which every technology is well suited. Conversely, no technology is well suited to all possible practices. If this debate is going to address the practical concerns of those who will use modeling languages to develop, test, deploy, operate and maintain software systems, it must therefore be broader than a debate about whether to use UML or domain specific languages (DSLs).

I am therefore refocusing the debate, or at least the part of it that weaves through my blog, to the issue of Software Factories vs. Model Driven Architecture.

Grady took up this focus in his posting with his comments on Software Factories before turning to the discussion of UML. Despite the inaccuracies in his characterization of Software Factories, which I answer below, I see his decision to bring methodology into the debate as an important step in the right direction. It was an acknowledgement that UML does not stand alone as a modeling language, but comes wrapped in an approach known as Object Oriented Analysis and Design (OOA&D).

Indeed, as noted in my previous posting, the UML is a direct outgrowth of OOA&D, and is used primarily to support the practice of OOA&D, despite its apparent aspirations to parlay its undisputed success in that arena into a new career as the lingua franca for model driven development.

Grady does a good job of explaining that software factories are *not* at all about stamping out endless copies of the same thing, as some have mistakenly claimed, but rather about automating rote and menial tasks, so that developers can get on with the abstract reasoning that humans do so much more effectively than tools.

That said, he also makes some misstatements about software factories. For example, he quotes Tom Demarco’s writing about early experiments with software factories twenty years ago in Japan, asserting that software is a development activity, not a production activity, and that trying to make it a production activity produces bad software. With due respect for both Grady and Tom, whose collective works I greatly admire, I offer the following points in contradiction to Tom's initial assertion and to Grady's invocation of it.

·      First,  production activities can yield good software, as well as bad software. Why? Because software quality is not a function of production. Production in any industry is about copying masters. Apart from gross mistakes or media errors, copying software does not affect its quality. Copying a good master produces good replicas and copying a bad master produces bad replicas. The quality of software is therefore determined almost entirely by its design and implementation.

·      Second, I find it surprising when people assert that production has no place in the software industry. It seems to me that producing CDs, DVDs, downloads and other copies from masters is actually quite essential to the survival of the software industry. We certainly care deeply about production here at Microsoft. From this observation, I conclude that software is both a development activity and a production activity.

·      Third, as should be clear by now, production is a red herring in this debate. The development process by which the masters are produced is the real concern. Indeed, in many situations, we might never produce any copies of a given master, or perhaps only a handful of copies.

·      Fourth, those who have studied industrialization know that in addition to economies of scale in the production process, mature industries seek economies of scope in the development process, as explained in the Software Factories book, and in my articles on MSDN. Software factories, as we define them, are about economies of scope in the development process, not about economies of scale in the production process.

Instead of repeating the lesson on economies of scale and scope from the book here, I will simply point out that software factories are about reusing patterns, frameworks, tools, templates, requirements, tests, instrumentation, process guidance, and many other assets used throughout the software life cycle, in a systematic, rather than ad hoc fashion, not about stamping out endless copies of the same thing.

It is worth noting, as we do in the book, that this is not really new. We already reuse some assets systematically, such as operating systems, web servers, class libraries and user interface designers. The goal of software factories is to radically increase the amount of systematic reuse in software development, so that much more of the application architecture can be implemented in the same way that UIs are implemented with UI builders, and data access is implemented using SQL engines. We are also interested in making it easier to work at higher levels of abstraction than we can with general purpose languages. For an interesting apologetic on this topic, see the recent article by Sergey Dmitriev. I'll have more to say about what Sergey calls Language Oriented Programming in subsequent postings.

The first volley on the topic of Software Factories vs. Model Driven Architecture, which Paul Ballard brought to my attention, has already been fired by Wim Bast in a news posting on The Server Side .NET, which I have answered there.