Wojtek's Blog

Code Generators: when can you live with them?

Wojtek Kozaczynski — Sun, 18 Nov 2007 22:04:32 GMT

Tom Hollander just posted a note Code Generators: Can't live with them, can't live without them. His question is really about the ROI on developing code generators. Most of the comments, however, are about the specific case he is trying to address with generators; creation of database mapping code.

Let me try to address Tom's key question: When it is worth creating a code generator as opposed to either (a) writing the code by hand or (b) trying to bake the logic (in this case db mapping) into a framework?

My experience with writing code generators is that their complexity grows very fast. Simple code generators, like the ones based on T4 templates used in GAX and DSL designers, are relatively inexpensive. However they suffer from the issues Tom outlines like, how to separate the generated code and the hand-written code, or how to generate and regenerated only the code that you need?

As soon as you attempt to address those issue, the complexity of the generator sky rockets. Take the latest version of the Web Service Software Factory as an example. The code is selectively generated from models. You can generate code only for selected model elements, the entire model, or even connected models. However to do that, we had to create a code generation framework that extends the out-of-the-box T4 template-based code generation of the DSL toolkit. We also had to add a model validation framework to ensure that the models are consistent before we attempt to generate the code. Developing those mechanisms was a significant portion of the cost of building the factory. Our estimate, which we presented at the last p&p Summit, was that we spent over 50% of the effort on building the factory infrastructure, which did not include writing templates, code generation strategies, etc.

As another example take the LINQ-to-SQL modeling tool. It selectively generates LINQ code, but to do that it introduces a model that sits between the database and the code generators.

Bottom line, unless simple code generator addresses your needs, think very hard about the cost of building a more sophisticated generator. Almost always you end up introducing an explicit or implicit model of what you are generating or the generation mappings. However if you really need to do that, take a look at what we have done in the Web Service Software Factory and see if you can reuse the mechanisms we developed.

Wojtek

Inputs to Web Service Software Factory: Take 2

Wojtek Kozaczynski — Sun, 18 Nov 2007 21:24:21 GMT

After my post "Web Service Software Factory; the Modeling Edition" Jason Hogg reminded me that I missed two significant inputs: EDRA and Web Service Security Patterns. I have added them to the top of the diagram below.

EDRA, which stands for Enterprise Development Reference Architecture, goes almost four years back. This guidance has been since phased out, but it had significant influence in two areas. One, it got us thinking about code generation from architecture templates is an alternative to "baking" things into frameworks (EDRA was a framework with no generated code). That led to GAX and then BATs. Second, it taught us a lot about the desired architecture of Web Services, and those lessons were carried forward by Jason and Don into the WSSF.

Web Service Security Patterns have been implemented as a separate guidance package, which moved from the Web Service Software Factory v2 to the Modeling Edition with almost no change. This show the durability of well defined patterns and their automation.

Wojtek

Application Management, Here I Come

Wojtek Kozaczynski — Sun, 18 Nov 2007 07:27:42 GMT

After almost 5 years in patterns & practices, after participating in building EDRA, architecting GAX, working on three software factories, helping define the software factory platform, working on pattern languages, and doing other tings, I am moving on. I am not moving far, however. I will be an architect in the Dynamic Systems Foundation group, which is a part of the Microsoft Management & Services Division (MSD).

For the foreseeable future, I tend not to plan too far ahead though, I will be working on tools for modeling managed systems and applications. In order to manage anything, and in particular a complex hardware and software systems, one needs to

know what the managed reality should be
what the managed reality actual is
what the differences between the is and the should are, and
how to change the actual reality to meet its intended should be state.

You cannot do any of the above effectively without describing (or modeling) the "should be" and the rules for the "change". Problem is, that there is no one agreed upon way of doing that and hence there are not tools to effectively model managed systems and applications. This, the shared models and tools, is what I will be working on.

Wojtek

Web Service Software Factory; the Modeling Edition

Wojtek Kozaczynski — Sun, 18 Nov 2007 07:03:00 GMT

Last week we (the Microsoft patterns & practices team ) released the Web Service Software Factory (or WSSF), the Modeling Edition (WSSF MD). When I looked back I realized that this factory is a cumulative result of an interesting chain of developments. I don't know how soon patterns & practices will build another factory like this one and when it does, I probably will not be participating in that work. So I thought it would be nice to recount those developments before we all forget about them. The diagram below should help.

The story starts almost 3 years ago when David Trowbridge and his team were working on the Integration Patterns (Integration Patterns) and Daniel Cazullino (Kzu), Oscar Calvo (Oscar), Tom Hollander (Tom) and I were working of GAX and GET. I was arguing with David about how to best package guidance in Visual Studio. He finally got my point, looked at me and said: "So you want to build toolkits for architecture baselines." This is how the term Baseline Architecture Toolkit (or BAT for short) was born and this is the term Eugenio Pace (Eugenio) and I used when we started developing what later became the Smart Client Software Factory. We changed the name when Tom and I agreed that there were more similarities between BATs and Software Factories described by Jack Greenfield (Jack) than differences. We understood that there were differences, but our thinking was that it would be better to have one term, not two.

At the same time Stuart Kent (Stuart), Steve Cook (Steve) and Gareth Jones (Gareth) were busy developing the Domain Specific Languages (DSL) SDK (DSL Tools). We cooperated with them very closely because GAX and DSL SDK share the code templating engine.

The Smart Client Software Factory is what we call a wizard-driven factory, that uses only GAX. It was followed by two versions of GAX-based Web Service Software Factory. This effort was led by Jason Hogg (Jason) and Don Smith (Don).

At about the the same time two factory efforts started that used both GAX and DLSs. The first one was Mauro Regio's (Mauro) HL7 Factory (HL7 SF) and the second was Jezz Santos' (Jezz) EFx Software Factory (EFx SF). Since Mauro's factory uses a number of interacting DSL designers, he was the first one who was really faced with the model integration problem. Daniel Cazzulino was one of the HL7 factory developers and from what I recall, he and Victore Aprea (vga) developed the first version of the Designer Integration Service, which is a Visual Studio service that can create, resolve and navigate cross-model links. Jezz, on the other hand, was the first who separated technology-independent models from technology-specific extensions. We reused his experiences when creating the Web Service factory model extension framework.

In the meantime the second version of Web Service Software Factory was very well received, but the overwhelming community demand was to simplify its user experience by introducing models. So in February we started the development of the modeling addition, and it was finished in October. The factory uses three types of models to generate service code

Service Contract models
Data Contract models, and
Hosting models.

The factory uses GAX, DSL SDK and a modified version of the DIS. It also introduces model, code generation, and validation extensions of the DSL toolkit. It is the first fully articulated, model-driven factory (automation, reference implementations, documentation, etc.) released by patterns & practices.

Wojtek

Patterns vs. Tooling

Wojtek Kozaczynski — Tue, 13 Nov 2007 21:45:00 GMT

At the p&p Summit last week Jason (The Hogg Blog), Ade (#2872), Dragos (Dragos Manolescu's blog) and I staged a discussion that was designed to polarize and divide the audience along "patterns vs. tools" line. Jason describes the event (The Hogg Blog)and J.D. adds interesting remark in (Patterns vs. Tooling).

On stage we intestinally took inflexible, all-or-nothing positions. In reality we all believe, at least I do, that tools and patterns complement each other. For example, on the following day I gave a presentation on "Patterns of Software Factories" in which I showed how the p&p factories (smart client, web client, web service, and repository factories) use patterns to build what they build. I have identified over 20 major patterns and showed how the factories put them together and implement them.

My key point in the discussion, and the thing I believe in, is that most useful are instrumented (encoded in tools) collections of carefully selected patterns that together build parts of applications: composite clients, WCF services, data mapping layers, etc.

Libraries of pattens, like Pattern Share, suffer from two major problems:

it is difficult to map a large design problem onto a collection of patterns that if put together will address is, and
the distance between the pattern description and its running implementation can be large.

Even the current, initial versions of factories address both of these problems; they come with pre-selected patterns and they come with tooling to commit those patterns to code. The major shortcoming of the current factories is that if the selected patterns don't exactly fit your needs, they (the factories) are difficult to change. But this is a subject of a different discussion.

Bottom line, I don't believe that this is "either patterns or tools" situation. On the other hand, today I will put my money on instrumented collections of pattens that do something, over an open-ended library of patterns.

Wojtek

I have been tegged

Wojtek Kozaczynski — Thu, 18 Jan 2007 05:21:00 GMT

I have been “tagged” by Jezz Santos, Daniel Cazzulino and J.D. Meier. There aren’t that many things about me that are either interesting or worth knowing J. So after searching my memory banks for a while, here are a few possible candidates

1. I almost ended up being a mechanical engineer. As a teen I was fascinated with cars and motorcycles, which I was disassembling and putting beck together. I got accepted to a mechanical engineering school and spent a year trying to convince myself that I liked learning about the physical properties of different materials. Eventually I gave up, and switched to EECS.

2. At about the time I was accepted to graduate school, I also got an offer to work as a professional skiing instructor in Canada. After many sleepless nights I decided to continue my education. To this day I am not positive that I made the right choice.

3. However I never managed to get rid of the skiing bug. My dream is to spend a year traveling around the world skiing every patch of snow I could find (well, many be every patch of snow that I could get on the top of using some means of transportation other than climbing).

4. I love dogs; there was always at least one dog in my house.

5. I programmed in assemblers.

I will tag Peter Provost, Tom Hollander, Jason Hogg, Don Smith and Dragos Manolescu.

Wojtek

The First Three Software Factories From p&p

Wojtek Kozaczynski — Sun, 06 Aug 2006 21:48:00 GMT

You may have noticed that in the span on one month the patterns&practices group (p&p) released three software factories.

The first out was the Smart Client Software Factory (SCSF). It is a factory for developing WinForm-based, composable client applications.
The second out of the gate was the Mobile Client Software Factory (MCSF) for developing applications where the client is running on mobile devices such as PDAs and smart phones.
The third was Web Service Software Factory (WSSF) for developing distributed applications communicating via Web services.

If you take a look at the three factories you will find that they are quire similar in the way they are structured, documented and built. But upon closer inspection you will also find that they are not strictly derived from the same software factory template (called software factory schema, see Jezz’s blog for a summary).

This is going to change for the better moving forward. A few of us at Microsoft have joined forces to work on a common schema and the Visual Studio factory authoring tools. The group now includes Jack Greenfield, who started it all :-), Tom Hollander, who had his fingers in the two out of three p&p factories, Mauro Regio, who led the development of the HL7 software factory , Jezz Santos mentioned already, Steve Cook and Stuart Kent who gave us the DSL tools, and your’s truly, an accomplice to many of the above crimes.

I am hoping we will be able to share the early results of our work with the community soon. If you are interested in contributing please ping us.

-- Wojtek

New Version of GAX

Wojtek Kozaczynski — Tue, 06 Jun 2006 21:00:00 GMT

It has been a while since I posted a message, but this occasion is hard to pass; we will be releasing a new version of GAX.

Since the last release in December we have been using GAX and GAT extensively on a number of projects including the Smart Client Software Factory project (practices.gotdotnet.com/scbat) and the Web Services Software Factory project (practices.gotdotnet.com/svcfactory). Needless to say, we found some things that needed fixing and some opportunities for improvements. Here is the list of major fixes and improvements that are coming up

- GAX and GAT will now work in non-English versions of Windows

- We have fixed the annoying bug that prevented text from appearing in wizard fields if you changed the font size on the desktop to Extra Large Fonts (this one caused me a very tense moment during a demo).

- We have moved the guidance package state from the .suo file to a separate file stored in the solution root folder (this one is actually interesting and may require a separate discussion).

- We have provided a mechanism to associate ms-help pages (or plain html pages) with wizards.

A brand new feature is the Guidance Navigator. The navigator is a new VS view (by default shown as a tab in the same workspace as the Properties view) showing for each guidance package

- applicable recipes and templates (the recipes and templates that cannot be executed are not show)

- history of the executed recipes with suggested applicable next steps

- help pages associated with each recipe.

The navigator addresses a common request of the community to make the recipes and templates easier to discover and apply.

So when will the new version be available? If nothing unexpected happens, it should be available to the community in the next two weeks.

-- Wojtek

Recipes Are Not Wizards

Wojtek Kozaczynski — Sun, 29 May 2005 19:43:00 GMT

I was helping another friend design a guidance package, and I realized that he was thinking about recipes as wizards. This is not such a good idea, and may lead to misusing GAT.

Recipes are simple animals; they get information from the user and/or their environment, and execute a sequence of actions. That's it.

You can make a recipe implement a conditional flow. To do that you have to make an earlier action in the sequence provide input to a later action (through an Output property), and program that later action to take a conditional flow depending on this input. This, however, breaks atomicity of actions and will be difficult to debug.

The GAT wizard has been designed to simplify obtaining recipe arguments values. If you wanted conditional flow, you would most likely wanted a wizard to show or not show (gray out) fields depending on values of other fields. This can be done in the GAT wizard framework extension, but only with custom wizard pages. Even if you do that, you end up with Null arguments (the grayed out ones) and have to make actions behave differently depending on if they have or have no argument values (back to the previous point about breaking action atomicity).

Recipes use simple wizard mechanism to collect information from the user, but they are not like wizards. They are atomic, in that they perform a sequences of chainable, yet independent actions. They don't perform actions between wizard pages and have not been designed to execute actions conditionally.

If you think you need a conditional recipe, try to break it up into different recipes, each performing one thing. Let the user decide which recipe is applicable in what situation. Also, remember that one recipe can attach (spawn) another recipe.

-- Wojtek

Nothing Shows Up Without a Reference

Wojtek Kozaczynski — Tue, 17 May 2005 16:47:00 GMT

I was helping Eugenio write his first guidance package. He asked me to show him a few things "so he did not have to read the documentation" J. He created a project template by copying one of the existing ones, rebuilt the guidance package solution (which registered the package), and tried it. His template did not show up?! Why? Because he did not create a reference. I gave him a quick overview of what reference are for, which I have summarized below .

GAX shows recipes and templates only if it has references for/to them (so in reality, it shows references to templates and recipes). If a recipe is defined as "bound" (an attribute of the Recipe element in the config file), you need to "attach" that recipes to a solution element before it shows up. The easiest way to attach a reference to an element is to make it a side effect of unfolding a VS template (by using the RecipeReference element in the WizardData). You can also do it the hard way, using GAX services.

If you define a recipe as unbound, you have to create a reference instance yourself. This is because the instance has to implement a method that GAX calls to ask the reference if it wants to be shown on the current selection. You can also create unbound references to templates (this is what Eugenio forgot to do).

We can go into details of creating references (if we need to) in other postings. Right now, however, I would like to get back to the main point I want to make; nothing shows up unless there is a reference. This is by design. One of the key design goals of GAT was to enable guidance (recipes and templates) "only where it applies and only when it applies". References are the mechanism to achieve this goal.

-- Wojtek

A Short History of the GAT Project

Wojtek Kozaczynski — Tue, 03 May 2005 00:51:00 GMT

I though you might be interested in how it happened that PAG developed a VS extensibility mechanism. So here is a short, unedited, history of the project.

The history starts before I joined PAG two years ago. When I came onboard Scott Densmore (yes, the EntLib Scott) was experimenting with using the VS Enterprise Templates to make the PAG code-based guidance appear in VS. It was sometimes painful to see Scott trying to put a square peg into a round hole, so to speak. The main problem was that Enterprise Templates were designed to constrain what you can do, not to help you do things in the context of what you have already done. Scott's project was called Daedalus, and it was part of what we called IPE, which stood for In Product Experience.

We concluded that Enterprise Templates did not have what we needed, but we were neither ready to embark on building something else or new what that would be (yet). But here came Shadowfax, and Ed Jezierski (yes, the EntLib and CAB Ed) and I realized, that the architecture of the framework would have been simpler and easier to use, have we had a VS extensibility mechanism that would allow us to manipulate VS solution artifacts at the design time. We had a few initial ides of what that mechanism would look like, and we asked Daniel Cazzulino to build a prototype. This was bit over a year ago. The prototype came out quite interesting and promising, so we decided it was time to give IPE the second chance.

Our initial design objectives where quite lofty (aren’t they always?), and we asked the Whitehorse team to help us investigate if we should use the MDF (Meta Data Framework). This is how Stuart Kent ended up working with us for a couple of months and greatly contributed to the articulation of the key GAT concept; the recipe. It was about May last year when we prototyped enough of the design to start a development project. In PAG-speak it means that you assemble a project team and you push the project acceleration pedal all the way to the floor. The project was named IPG, which stands for In Product Guidance, and the team members were Tom Hollander (product management), Bill Loeffler (program management), Daniel Cazzulino and Oscar Calvo (development) , Ashish Babbar (testing) and your truly (architecture). The rest is, as they say, history, but this history has a few interesting twists and turns worth mentioning.

Once we realized that we were building a VS extensibility machines, we was time to go to VS team and "confessed". Our VS brethren didn't like a number of the details (parts of user experience in particular), but they did liked our ideas in general, so they assigned a, Stephanie Saad (VS core program management), to be our "guardian angel". Although not an important project milestone, this was to me a very important moment. It was the first real indication that we were up to something interesting. Over the course of next few months Stephanie helped us tremendously. In particular, when we were integrating recipes with new VS templates and we needed a few changes in VS core (aka DCRs).

Another interesting twist was deciding what GAT really was. Was it a piece of guidance, was it a product, or was it something in-between? This is where the Whitehorse team helped again. Julia Liuson, the Whitehorse team Product Unit Manger, and Jack Greenfield (yes, the Software Factories Jack), offered a home fo GAT under the emerging tools initiative. This is why you can find GAT on the VS workbenches site, next to DSL SDK.

Coming up with the "official" name was surprisingly hard. Until late in the project we were working on what we called "Recipe Framework" and what is now called guidance packages, we called recipe clusters. When confronted with the unavoidable need to give "it" a proper name, we proposed VS-IGT, that stood for VS Integrated Guidance Toolkit. I liked the name quite a bit, but techies have little to say when it comes to selecting names of things we release to communities. The marketing folks have a lot to say, and then did not like VS-IGT. After running around a number of circles, Tom finally worked out a compromise, and GAT got its name.

The last interesting tidbit (one day I will ask Stuart to write his side of the story) is that about mid way through the project (it was fall by then) the DSL SDK team and us realized, that independently we crated very similar text templating engines. Daniel wrote the GAT's T3 (stands for Text Template Transformations) and Gareth Jones form the DLS team wrote their nameless engine. The two engines were remarkably similar, so we decided to merge them. The review (Beta2) version of GAT still has T3 inside, but the Whidbey version will have the shared T4 engine (Stuart added one more T to out three Ts, so unless you know this bit of the project history, the name makes little sense).

Well, this is my recollection of what was worth sharing about the project. If I missed anything or anybody, my team member will certainly catch the omissions and blog about it.

-- Wojtek

Introduction

Wojtek Kozaczynski — Tue, 03 May 2005 00:26:00 GMT

I am a Software Architect in the PAG (Platform Architecture Guidance) group, which is behind most of the developer and architect content posted on the Microsoft patterns & practices site. If you ever need to say my names, here is a few helpfull phonetic substitution rules: W sounds like V, J sounds like Y, and CZ sounds like TCH. So my names should sound something like: Voy-tek Koza-tchin-ski. Enough about pronouncing Polish names.

I have been with PAG for two years. I was the architect on the Shadowfax project (aka EDRA project), on the GAT project (this is the main reason why I decided that it is time to start bloging), and I am the architect on a new project called Smart Client Candidate Baseline Architectures (with time you should also see me posting about that).

I don't know how this bloging thing will go, but I intend to blog only when I have something to say (which will be probably infrequently).

-- Wojtek (or is it Voy-tek?)