Jack Greenfield's Blog : Domain Specific Languages

Articles on Software Factories

Jack Greenfield — Wed, 10 Oct 2007 23:26:00 GMT

I recently published an article on using software factories in supply chains for Methods and Tools. Also, many people have asked about the article I wrote about factories vs. MDA for the Perspectives of the IASA.

Performance Analysis and Prediction

Jack Greenfield — Sun, 29 Oct 2006 23:49:00 GMT

Irfan Idrees wrote with the observation that while performance analysis and prediction are of critical importance for software developers, the rigorous approaches prevalent in the marketplace have not been widely accepted in the community. Irfan then asked how software factories deal with this issue.

Software factories provide leverage by reducing the amount of variability in application development. With a software factory, one effectively builds a variant of a known application type. All features common to the product family are designed and implemented in known ways. The only variability in a given member of the family is due to the features that make it unique.

Unique features include both fully custom features and variations on known variable features. A fully custom feature is one that has not been predicted or taken into account in any way by the software factory. Variations on known variable features may be either fully constrained (e.g., picking one of five predefined transaction policies from a drop down), or partially constrained, (e.g., writing custom code to complete a template generated event handler designed to work within a predefined event architecture).

To the extent that the performance of that application type is known and understood, then the task of analyzing and predicting the performance of a given variant becomes a matter of analyzing and predicting the performance of its unique features, and of analyzing and predicting the impact of their performance on the known performance characteristics of the application type, such as overall performance, the performance of specific operations or scenarios, and the way those measurements may vary with variations on known variable features.

Ad Hoc and Systematic Reuse

Jack Greenfield — Wed, 18 Oct 2006 01:19:00 GMT

In the software factories book, we explain that systematic reuse is effective, but ad hoc reuse is not.

Ad hoc reuse is the “Field of Dreams” approach… “If we build it, they will come”. Great line for a movie, but it doesn’t work well in the real world of software development.

Systematic reuse relies on the recognition that every component embodies many assumptions about functional requirements, technology choices, operational requirements, architecture, deployment topology, testing strategy, development process, and many other aspects of the software development life cycle.

Failing to recognize and plan around these assumptions leads to the well known problem called “architectural mismatch”. This problem is described in detail in a series of papers from David Garlan at Carnegie Mellon. The patterns and practices team arrived at this same conclusion “bottom up”, by listening to customer feedback regarding the guidance offerings they were developing. Customers were asking how to use those offerings to build a specific type of application.

Recognizing and planning around these assumptions leads to defining families of similar solutions, and separating their common features, which make them similar, from their variable features, which make one member of the family different from the next. The common features are used to develop a custom process for building the members of the family, and of a set of reusable assets supporting the process. The variable features are used to drive parameterization, configuration, assembly and other techniques for accommodating the differences between the family members.

This is the thinking underlying the development of the components that are truly reusable, such as RDBMSs and GUI frameworks, although it may not have been recognized at the time by all of the people involved in building those components.

Work at the Software Engineering Institute (SEI) formalized this thinking in the concept of Software Product Lines. Software Product Line practices have been well established through many books, case studies and experience reports, and are widely accepted in the software engineering community.

Software Factories is a methodology for using domain specific languages and other technologies to automate Software Product Lines. They were developed by leveraging the work of David Garlan, as described in the book, in consultation with the authors of Software Product Lines at the SEI, and with other experts in related disciplines, such as pattern languages, generative programming and feature based development.

Architecture Interview and Podcasts

Jack Greenfield — Mon, 23 Jan 2006 04:44:16 GMT

Here’s an update about Software Factories on ARCast, the podcast series on the Architecture Resource Center. Last month, five panelists (counting myself) recorded three shows on model driven development. It's been quite successful, with the largest number of downloads to date in the series. You can find the first show here, the second one here and the third one here.

Following up the panel discussion, Martin Danner and I did another podcast on MDA vs. Software Factories.

I also joined Mauro Regio recently for an ArcTalk interview with Ron Jacobs about the HL7 Software Factory described in my recent post, and Steve Cook also did an interesting one on one with Ron Jacobs about DSLs and Software Factories.

RE: VSTS 2005, DSL, and Software Architecture

Jack Greenfield — Fri, 06 Jan 2006 21:45:00 GMT

This is a response to a blog posting by Tad Anderson, and the pointer to it on the MSDN Architecture General forum.

First, I agree with your observation, Tad, about the lack of support for the Software Architect in VSTS 2005. We did indeed focus on the System Architect not the Software Architect. We had limited resources to allocate to architecture tools, and felt it was more important to support Microsoft’s drive toward connected systems for the 2005 release. As you suggest, however, we do intend to address the needs of the Software Architect in subsequent releases, and most of that support will come from our Software Factory strategy.

However, I disagree with your assertion that our Software Factory strategy, with its reliance on DSLs, will “make a huge mess”. From your comments, I gather that you haven’t read the Software Factories book, or been exposed to the many white papers, blog postings, conference proceedings, articles and podcasts that we’ve produced, since they address the issues you raise at length. I will summarize them here, but I'll also point you to the those other resources, where you learn a lot more about what we’re doing and why.

While it’s certainly true that unqualified practitioners might abuse our technology, there is nothing about our technology that makes it particularly susceptible to abuse. As we know, there is nothing to stop people from writing bad code in Java or from creating ineffective UML models.

However, we have admittedly not done a good enough job of making clear that we intend people to build DSLs primarily in the context of Software Factories. Yes, we do expect them to build standalone DSLs, but the primary goal of productizing the DSL technology was to support factory development. Why? Two reasons:

1) Factories provide a methodology for determining what to model for a given product family. The key to using DSLs effectively is to understand how solutions are developed for a given problem domain. Factories are organized around a model of the development process for a given problem domain. This model, called a factory schema, is a graph of viewpoints. A viewpoint defines the perspective of a specific stakeholder or a specific role in the project team. As recommended by IEE 1471, and the book Documenting Software Architectures by Clements, et. al., we use the graph of viewpoints to define the product family architecture. We also use it to define the development process, or what Product Line Engineering calls the production plan. Each viewpoint describes the set of workflows used by its stakeholders, the work products acted upon by those workflows, and a set of assets, such as tools, templates, class libraries and guidelines, that support the enactment of those workflows. How does this relate to modeling? The schema shows us what to model. Each viewpoint defines a domain, such as logical database design, user interface process or business rule definition. These domains are aspects or subsets of the larger problem domain defined by the product family. We may only know enough from experience about such a domain to supply documentation assets, such as patterns or practices, to its stakeholders. However, if we know enough about it to model its key concepts, then we can supply more powerful assets, such as a DSL and tools that perform generation, validation, trace, analysis and other forms of automation using models based on that DSL. Of course, implicit in this approach to modeling is the assumption that DSLs are more effective than general purpose modeling languages like UML for model driven development because they are highly focused on specific problem domains. That point is discussed at length in many of our publications, including our Modeling Strategy paper. In summary, a factory contains a model that provides a basis for determining which aspects of a problem can and/or should be supported by DSLs, and how DSL based models should relate to each other and to the work products of non-model driven activities.

2) Software development is about more than just models. While it’s nice to think that some day, we might develop every part of a system using models, just as we currently develop every part of an application using 3 GLs rather than assembly language, it’s just not feasible today. You can hear more on this topic in the recent ARCast on MSDN. Because we take this pragmatic view, we require factories to provide whatever assets are most appropriate to support the enactment of the workflows for a given viewpoint, and not just on modeling. A modeling tool is just one kind of asset that factory builders can provide to support factory users, but there are many others. This is one of the fundamental differences between Software Factories and MDA. You can read more about the differences between them in the article that I wrote for the newsletter of the International Association of Software Architects.

Regarding the effect of our strategy on industry standards, I think it was inevitable that people would demand more focused modeling technologies. Within the OMG, there is very strong momentum toward domain specific modeling. It just happens to be expressed using UML profiles, which are awkward at best. A large block of OMG members would prefer to see domain specific languages defined using the Meta Object Facility (MOF). In effect, that’s what we’re doing, too. However, we don’t use the standard MOF. Neither does IBM, Sun, or anyone else. Why? Because the MOF maps to the internal architecture of the modeling tool. Experiences like IBM AD/Cycle show that tool internals cannot be standardized effectively, since vendors must be free to design and implement tools in response to market changes, not standards committee proceedings. Based on these experiences, we prefer the standardization of interachange formats – not interchange formats like XMI, which depend on MOF alignment to achieve interoperability, but interchange formats like the ones defined by UN/CEFACT, RosettaNet and HL7 to name a few.

As for learning a new language with every gig, you have to do that anyway, whether you’re using UML profiles, or no modeling language, at all. While the UML defines a set of widely understood notational styles, the meaning of any given profile must be learned. This should come as no surprise. What you really have to learn with every gig is the problem domain, or rather the many subsets or aspects of the overall problem domain, and how they all relate to each other. The goal of software factories is to help solve that problem. The software factory schema provides a way to model all of the aspects and their interrelationships. While the DSLs that people build using our tools are unique, they are usually based on the notational styles supplied by the templates that we provide. They happen to look a lot like UML notational styles, of course, and that’s intentional. We have no interest in reinventing the wheel when it comes to notational styles that have proven effective. By the same token, we have not followed any of UML notational styles that have not proven effective, such as the deployment diagram.

It’s true that that we have not yet aligned Software Factories and MSF. That is currently work in progress. When the alignment is published, I think you will find that the result quite different from RUP and XP. For one thing, the development process in a factory is organized around viewpoints, as I’ve just described. For another, the workflow in a factory is driven not by sequential execution, by pre- and post- conditions associated with each activity. Activities come into scope when their pre- conditions are satisfied and go out of scope when their post- conditions are satisfied. Work can be done in any order subject to these constraints.

It’s also quite true that we need to provide a more complete set of tools and templates to support the Software Architect, as you suggest. That’s what we’re busy doing.

Finally, I should point out that a critical mass of industry experts supports our work.

For example, if you read the Acknowledgements in the Software Factories book, you’ll find that we leaned heavily on the work of Paul Clements and Linda Northrop, and met with them multiple times in person. You’ll also learn that Paul was a reviewer, and you’ll find his endorsement of our work inside the front cover.

If you look at our web site, you’ll discover that Linda Northrop invited me to give the keynote at the SEI sponsored Software Product Lines Conference 2004, giving an overview of Software Factories.

If you look at our blogs, you’ll find that Martin Fowler had some complimentary things to say about our strategy in his paper on Language Workbenches.

It may be worth mentioning, while we’re talking about industry expert response to our work, that we have had and continue to have critical interaction with some of the other people on your list, including Mr. Jacobson, Mr. Ambler and Mr. Evans.

Inside the front cover of the book, you will also find endorsements from Doug Schmidt, lead author of Pattern Oriented Software Architecture, Volume 2 (POSA2), David Frankel, author of Model Driven Architecture, Krzysztof Czarnecki, author of Generative Programming and leading voice in the industry on feature modeling, and John Crupi, lead author of Core J2EE Patterns and the writer of our foreword.

There are many other industry experts who continue to review our work, share their thinking with us, and generally advise us in our efforts to craft a pragmatic approach to software development that is firmly rooted in proven practices from software product lines, model driven development and service oriented architecture. You can read what many of them have written in the papers they submitted to the International Conference On Software Factories at OOPSLA 2005.

Software Factories Applied

Jack Greenfield — Fri, 09 Dec 2005 20:24:00 GMT

Keith Short, Mauro Regio and I have agreed to write a new book called Software Factories Applied. Our goal is to write a short book for practitioners that complements the theoretical foundation established in the original Software Factories book by showing how to build a software factory using currently available technologies, namely the DSL tools, the Guidance Automation Toolkit and Visual Studio 2005 Team System. A key topic covered in the book will be how to build a software factory schema, which is the model at the heart of a software factory. Our plan is to walk through a complete worked example from end to end, showing what we did at each step. It will probably be based on a Business Collaboration Factory that generalizes the HL7 factory that Mauro and I developed. You can read about the lessons learned in the development of that factory in this paper that we contributed to the International Workshop on Software Factories at OOPSLA 2005.

OOPSLA 2005 Recap

Jack Greenfield — Fri, 09 Dec 2005 05:10:00 GMT

Keith Short has already written a post that provides an overview our activities at OOPSLA 2005. Here is a compendium of my activities at the conference.

On Sunday, 10/16, I hosted a full day tutorial on Software Factories with Steve Cook.
On Monday, 10/17, Steve and I hosted a full day workshop on Software Factories. The workshop announcement and proceedings are posted here.
On Tuesday, 10/18, I spoke on a panel called Aspects – Passing Fad or New Foundation with Dave Thomas, Gregor Kiczales, Ivar Jacobson and Matthew Webster hosted by Steve Berczuk.
On Wednesday, 10/19, I wrote an article for the OOPSLA Conference Newsletter called A Not So Random Walk.
On Wednesday, 10/19, I attended a half day tutorial on Agile Software Development in the Large.
On Wednesday, 10/19, I met with Brian Henderson-Sellers and Don Firesmith to discuss the Open Process Framework.

Putting The Cart Where It Belongs

Jack Greenfield — Tue, 04 Jan 2005 01:41:00 GMT

In this posting, I continue addressing issues in the recent blog posting by Grady Booch. As I noted in my previous posting, the debate up to this point has been about the suitability of the Unified Modeling Language (UML) as a medium for model driven development.

However, that is not really the question we should be debating.

Instead of asking whether or not we should use our existing hammer (i.e., UML) to hit everything in sight, even things that don't look like nails, we should be asking what it is that we want to accomplish (i.e., model driven development), and what it looks like when done properly. Only after those questions have been answered is it appropriate to ask what languages, techniques, tools or other resources we should use.

Why? Because we cannot evaluate the suitability of UML or any other technology apart from understanding what we hope to gain by using it and how we will use it to realize those gains. To make this point more bluntly, requirements should drive technology decisions, not the opposite.

While this should be patently obvious, the debate about UML vs. DSLs continues to put the cart (i.e., modeling language technology) before the horse (i.e., requirements for model driven development). The debate should instead be about the methodologies that are proposing to use these modeling language technologies for model driven development, since they are both key sources and key repositories of those requirements.

This connection between methodology and modeling language technology is no accident. There is a finite set of practices for which every technology is well suited. Conversely, no technology is well suited to all possible practices. If this debate is going to address the practical concerns of those who will use modeling languages to develop, test, deploy, operate and maintain software systems, it must therefore be broader than a debate about whether to use UML or domain specific languages (DSLs).

I am therefore refocusing the debate, or at least the part of it that weaves through my blog, to the issue of Software Factories vs. Model Driven Architecture.

Grady took up this focus in his posting with his comments on Software Factories before turning to the discussion of UML. Despite the inaccuracies in his characterization of Software Factories, which I answer below, I see his decision to bring methodology into the debate as an important step in the right direction. It was an acknowledgement that UML does not stand alone as a modeling language, but comes wrapped in an approach known as Object Oriented Analysis and Design (OOA&D).

Indeed, as noted in my previous posting, the UML is a direct outgrowth of OOA&D, and is used primarily to support the practice of OOA&D, despite its apparent aspirations to parlay its undisputed success in that arena into a new career as the lingua franca for model driven development.

Grady does a good job of explaining that software factories are *not* at all about stamping out endless copies of the same thing, as some have mistakenly claimed, but rather about automating rote and menial tasks, so that developers can get on with the abstract reasoning that humans do so much more effectively than tools.

That said, he also makes some misstatements about software factories. For example, he quotes Tom Demarco’s writing about early experiments with software factories twenty years ago in Japan, asserting that software is a development activity, not a production activity, and that trying to make it a production activity produces bad software. With due respect for both Grady and Tom, whose collective works I greatly admire, I offer the following points in contradiction to Tom's initial assertion and to Grady's invocation of it.

· First, production activities can yield good software, as well as bad software. Why? Because software quality is not a function of production. Production in any industry is about copying masters. Apart from gross mistakes or media errors, copying software does not affect its quality. Copying a good master produces good replicas and copying a bad master produces bad replicas. The quality of software is therefore determined almost entirely by its design and implementation.

· Second, I find it surprising when people assert that production has no place in the software industry. It seems to me that producing CDs, DVDs, downloads and other copies from masters is actually quite essential to the survival of the software industry. We certainly care deeply about production here at Microsoft. From this observation, I conclude that software is both a development activity and a production activity.

· Third, as should be clear by now, production is a red herring in this debate. The development process by which the masters are produced is the real concern. Indeed, in many situations, we might never produce any copies of a given master, or perhaps only a handful of copies.

· Fourth, those who have studied industrialization know that in addition to economies of scale in the production process, mature industries seek economies of scope in the development process, as explained in the Software Factories book, and in my articles on MSDN. Software factories, as we define them, are about economies of scope in the development process, not about economies of scale in the production process.

Instead of repeating the lesson on economies of scale and scope from the book here, I will simply point out that software factories are about reusing patterns, frameworks, tools, templates, requirements, tests, instrumentation, process guidance, and many other assets used throughout the software life cycle, in a systematic, rather than ad hoc fashion, not about stamping out endless copies of the same thing.

It is worth noting, as we do in the book, that this is not really new. We already reuse some assets systematically, such as operating systems, web servers, class libraries and user interface designers. The goal of software factories is to radically increase the amount of systematic reuse in software development, so that much more of the application architecture can be implemented in the same way that UIs are implemented with UI builders, and data access is implemented using SQL engines. We are also interested in making it easier to work at higher levels of abstraction than we can with general purpose languages. For an interesting apologetic on this topic, see the recent article by Sergey Dmitriev. I'll have more to say about what Sergey calls Language Oriented Programming in subsequent postings.

The first volley on the topic of Software Factories vs. Model Driven Architecture, which Paul Ballard brought to my attention, has already been fired by Wim Bast in a news posting on The Server Side .NET, which I have answered there.

Microsoft and Domain Specific Languages, Reprise

Jack Greenfield — Tue, 21 Dec 2004 05:08:00 GMT

Two weeks ago, Grady Booch posted a blog entry titled “Microsoft and Domain Specific Languages”. The posting is part of a running debate between Grady, and my colleagues at Microsoft Steve Cook and Alan Wills.

While there are several points in the posting on which Grady and I disagree, which I will address over the course of the next few postings, there are also several points on which we agree. In particular, we share the conviction that packaging knowledge for reuse in patterns, languages, frameworks, tools and other form factors is “the right next stage in cutting the Gordian knot of software”. You can read about what we think is required to accelerate this type of knowledge reuse in the Software Factories book, and about Microsoft’s work on patterns, led by David Trowbridge and Ward Cunningham, here and here.

That said, the devil is in the details of how to automate more of the software life cycle using metadata captured by models, a goal often described as model driven development. At least that’s the goal here at Microsoft. Of course, that is not the goal for which UML was developed. It was developed to provide a common notation for the practice of object oriented analysis and design (OOA&D), as described here.

So, is it fair to judge the UML on how well it measures up as a medium for model driven development? Well, one might not think so, except for the fact that it is being promoted by a standards organization (i.e., the Object Management Group), not only as A medium, but as THE (official, sanctioned, approved, acceptable and appropriate) medium. The implication is that the UML has already been demonstrated to be the most appropriate medium for model driven development, and that the burden of proof falls not to the UML, nor to its promoters, but to those who would use some other medium.

This debate, then, is about the suitability of the Unified Modeling Language (UML) as a medium for developing source artifacts from which running systems are created, and for capturing information about those systems (i.e., metadata) that can be used to automate activities across the software life cycle.

The first point in Grady’s posting on which we disagree is his claim that Microsoft has rejected the UML in favor of proprietary domain-specific languages. Of course, this is not an accurate description of our position, which we have stated publicly on many occasions. We have not rejected the UML. Rather, we agree with Grady that UML has indeed proven to be very useful as a repository for notational conventions that support a range of modeling styles. There is no question that this was a valuable contribution that significantly advanced the practice of modeling, as traditionally understood, where the focus is primarily on creating documentation that describes system structure or behavior.

What we have rejected is the claim that the UML is the most appropriate medium for model driven development, where models are used as source artifacts that are compiled, interpreted, or otherwise used to automate software life cycle activities. On this point, we agree with Grady that UML is not appropriate for all purposes. After all, it is a unified modeling language, so called because it was developed to integrate notational styles used by multiple methodologists, not a universal language that can solve all problems.

Clearly, developing, testing, deploying, managing and maintaining software are quite different than creating documentation. One obvious difference is that models used for development must have semantics defined precisely enough to support computation, not just human interpretation. A second is that models used for automation must fit well into the software development process, and the file oriented world in which it is practiced.

Perhaps, instead of trying to bend an existing language designed for designed for creating documentation to the duties of this new and promising approach to software development, we would do better to come at the question from the opposite direction, and ask what the most appropriate language or languages would look like. On this point, we agree with Martin Fowler, who expresses skepticism about attempts to use the UML as a language for model driven development here, and with Dave Thomas, who wrote here about issues with UML and its use in the OMG branded methodology called Model Driven Architecture.

I’ll have a lot more to say about this topic, and about the broader question of how to practice model driven development effectively in subsequent postings.