Jezz Santos : Domain Modelling

New Software Factories Community Site

jezzsa — Sat, 05 Jan 2008 04:22:20 GMT

I am a bit tardy on the blogs about this, but if you haven't heard yet - we recently opened a new site dedicated to building and sharing a community of knowledge around Software Factories.

http://sf.devrevolution.com/

Huge thanks to Martin Danner who has hosted the site for us, and Edward Bakker who has championed the creation of the community. For some time we have aimed to provide a central place online we, as a community, can collect, discuss, rank, comment and order content about factories and Visual Studio Extensibility as this methodology gains momentum.

There is a mother site to this one (DevRevolution) which aims to address other industry topics such as Agile Development and Application Lifecycle Management as revolutions to the software development industry. However we are starting of humbly primarily focusing our attention on software factories at present. [Which just quietly, I see as an evolution not necessarily a full revolution to how we build some types of software - anyhow]

There are many sites and resources on the web about and around software factories from many of the great folks who have pioneered this space before, and heaps of information about its fundamental technologies and practices (i.e. domain modeling, automation, generative programming etc.) so we wanted to provide a central place to bring this knowledge together and have you guys - the community - view it all in one place. The idea is to then have the community collate, rate, comment and sort the content so that anyone visiting the site can find the most relevant content to consume based upon there understanding of this space. Clearly, no one wants to trawl hundreds of disparate links to sources of info to mine and find the best or the highest quality information on this topic. So having the community rank, comment and direct the visitors to the best content is an important attribute. So this is our primary goal - to enable that. We also want a place for people to come to and feel part of a larger industry group of like minded folks practicing this methodology developing its tools and practices.

We are currently a little centered on Visual Studio Extensibility as the platform for software factories at present simply because Microsoft is leading the Software Factories initiative at present, but we would like to think that the concepts and practices are not necessarily exclusive to Microsoft technologies, and would like to support ideas from other platforms and technologies in the industry.

Of course we realize that much of the relevant content out there for consumption is addressable only in the heads of many of the people involved in and around the factories initiative. We have linked some resources (as you can see), but we are relying on you to add whatever resources you are aware of.

We are focusing on refining the site and providing improved means (over what you see there now) to provide a better user experience and means to better add, rank, comment and sort the content best. Basically we are aiming to have each piece of content referenced on the site to support attributes, comments, ratings and categorization.

Currently the modules we have are a step in that direction, some parts allow this capability, others don't. So we have much work to do to reach that goal. It's a little ambitious for us, but we recognize that others are reaching out for such as resource.

However, we still need more content and feedback to help drive that.

Please have a look and let us know if you think this is useful, what is missing and how it should work to be the tool this evolving community will treasure.

[If you visit the site, and don't submit any content or offer any honest constructive feedback, then I hope you leave feeling guilty that you have not really participated in bettering the community as a whole - we especially need and value your comments and feedback.]

Some links:

Authoring XML using DSL's

jezzsa — Wed, 04 Jul 2007 17:20:10 GMT

Recently, a good colleague of mine (you know who you are Tim), asked me what steps he should take to build an XML authoring solution based upon existing XSD schemas they already have in their company. The objective was to create XML file instances (of a well known schema) to describe some business thing, but how to do that with a better authoring experience than writing angle brackets - since most of the folks who need do this in his company don't really have XML skills or sufficient knowledge of the XSD schemas to accurately construct one of these XML instances in its entirely. XML is just the chosen format the technical staff chose to use to represent this data for other technical reasons, and why should the people who need to create these business things have to care about XML or XSD? But they sure know what these business things are and how to define them in higher level terms.

Clearly, this is not an uncommon scenario, and, in principal at least, it should not warrant a massive development effort to create a simple XML authoring tool just dedicated just for this XSD schema. You might argue that they could just use XML notepad or other XML editor, but this is really not the kind of solution that non-developers business professionals would be very comfortable or productive with.

To be fair to Tim, he already knew what solution to go for of course, but wanted some reassurance of how quickly and what tasks need to be performed for his team to leverage some of the new tools to create this kind of solution for him. Providing him a nice dedicated 'smart' simplified graphical designer that he could put in front of his business oriented people to help guide them through the desired authoring process of these things; never having to touch (or care about) the underlying XML, how it is defined, or how it gets completed, with all its intricacies etc. He just wanted a simple GUI designer to define the main aspects of the business thing the XML is describing and spit out the fully compliant XML for other tools to process later.

Well, if you were confronted with this problem a few years back, you would have likely proposed creating a new dedicated web solution, convoluted XSLT type solution or some desktop application or tool (or perhaps even an InfoPath solution), or told Tim to go use XML notepad of some such generic XML authoring tool. But today, I am sure (I hope) most of you would now feel comfortable suggesting to go build a domain specific language with the DSL toolkit.

What makes building a DSL for this kind of 'brown-field' solution very interesting, is that you already effectively have a domain model defined in your XSD, which is a great starting point.

'Brown-fields' is a term used to describe scenarios where there are existing solutions already implemented and where you are trying to introduce a new technology or solution to help with the completion or enhancement of that solution. It's almost the opposite to 'Green-field' scenarios where you have freedom to build any solution with any means, and are unconstrained by existing tools, technology or solutions.

The XSD provides most of the information needed to describe the model and the types (domain classes, external types, inheritance etc) it needs. Once you have defined your domain model of course, what you show in a DSL designer does not have to reflect all domain classes and relationships from your XSD. After all, you may want to provide some abstraction of the thing defined in the XSD in your designer, and have the designer fill-in or imply parts of the XML for the user, based on either defaults, or other settings and constraints in the XSD. This ensures that the thing defined (in XML) is always valid and complete with respect to the XSD.

Once you have your domain model fully defined, and a designer defined to graphically represent your domain model, its simply a case of generating the XML from this DSL. You can do this in a text template, or apply an XSLT transform to the DSL file, both are common approaches. But there is a better technique for cases like this.

One little known fact about the DSL toolkit, is that you can extend it to control exactly how to persist the XML that is written out to describe the instance of your DSL.

In case you haven't noticed, the instances of your DSL files are persisted as XML in a logical XML format that reflects the domain model of your DSL. In fact, each DSL project creates for you an XSD that defines the format of this persisted XML as well. View your DSL instance files with a text editor, like notepad, and see for yourself this XML.

The DSL toolkit provides some simple built-in control to customize how the elements and attributes of the persisted XML in the domain model are serialized to your DSL file. This is limited to how to name the elements and attributes, and basic XML formatting. But the DSL tools also support another level of customization that enables you to change completely how these files are rendered in any XML format.

Through the use of custom serializers, you can render your XML in any format you wish, as long as that format is loss-less enough to be serialized and de-serialized without loss of data for your DSL. The DSL tools team have an example of this in chapter 6 their new book 'Domain-Specific Development with Visual Studio DSL Tools', and a bunch of guidelines to follow for custom serialization you should follow.

Knowing all this, it's now conceptually trivial to build a new DSL based upon an existing XSD, and provide a graphical designer to author new XML instances of this XSD. Then persist your DSL XML in a format that is defined by your XSD, and skip the additional transformation step - as the model file is the XML it represents. In this way, you are effectively creating a new dedicated graphical editor in Visual Studio specifically for editing your types of XML files, and these files are still valid XML instances that can be consumed by other tools and processes.

In practice though, it can be quite tedious to define all the domain classes and relationships from an existing XSD (using swivel-chair integration), especially if the XSD is large. It will also be a little technically challenging, since the DSL domain meta-model will not be able to support all constructs and relationship types defined in XSD - there will have to limited coverage of XSD that can be modelled, and graceful handling/workarounds of areas it can't.

It's a shame we don't have any tools to import XSD files as a starting point for a new DSL, or import parts of an existing XSD's into an existing domain model (imagine 'New DSL Solution from XSD' command). Which would then pre-populate a domain model reflecting the XSD, and generate the necessary custom serializers used to persist the DSL model to XML conforming to the XSD schema.

However, I am hoping at least someone eager in the DSL community will take it upon themselves to create an 'DSL Import XSD PowerToy' extension that would provide this for others to leverage - perhaps a new codeplex project? - it certainly would help a lot of people get started creating DSL's for existing solutions based upon XSD's, and provide a very quick and easy means to create graphical designers for XML authoring of any existing XSD schema.

DSL Editor PowerToy - New Release

jezzsa — Sun, 20 May 2007 10:17:00 GMT

We released the next milestone of the DSL Editor PowerToy today on CodePlex.

"A means to add multiple editors to your DSL, either hosted in a tool-window, or replace the graphical designer of your DSL"

The idea of this release is to provide the ability to add multiple DSL Editors to display different 'views' of your Domain Model with any windows form control. You write the control, we host it, provide the runtime UX and the integration to the domain model and runtime environment for you.

The idea with these 'views' is to allow you to display some aspect of your DSL (in an editor) for viewing and possibly editing. This 'aspect' could be for example, some parent-child relationship (potentially not even visible in your designer) or maybe some cross-cutting concern across whole domain model. The choice is yours. The 'view' is configured in the PowerToy (see below how), and the editor control displays that 'view' however you see fit.

[BTW: The editor is a windows control (forms-based) so one imagines this view will be displayed by some combination of windows controls, but its not limited to just forms controls. In fact any display technology you want.]

Ultimately, (in final release) the PowerToy will be providing an special editor control that has a tree-grid-like display, that can handle hierarchical relationships. But the idea is that you can provide your own control.

You can host these windows controls in either a dedicated tool-window or, (new in this release) as a replacement for the graphical designer of your DSL. [A little known secret of the DSL tools!] and is very handy in cases where a the box-line graphical designer is not optimal for displaying or interacting with your domain model.

The PowerToy now provides a small configuration language to configure your various editors, and control the hosting options (in toolwindow or as replacement designer).

This language will develop further in next release to help you define the 'views' mentioned above for display in the editors.

More Info

This release establishes a new infrastructure and design platform for the next milestones which will add the ability to graphically define the views (pick and chose various aspects of your domain classes/relationships) of your underlying domain model (expanding the configuration language above).

Those 'views' will be available to your editors to bind to to display and interact with. The next release will also deliver a simple hierarchical control to navigate these views.

Building DSL Enhancing PowerToys - Good Practices

jezzsa — Fri, 13 Apr 2007 10:50:00 GMT

If you are building DSL's and you haven't heard yet, we released the Editor PowerToy last month. The current release is only a small step towards the vision we have for the PowerToy that will deliver a very powerful capability to provide additional customized views in a tool-window (for your DSL diagram) that enhance the runtime usability of your DSL.

"All well and good" you might say, but this might not be that interesting to you anyway, since you have no use for the PowerToy presently.

Nonetheless, in this post I wanted to describe the design, techniques and patterns we used to create the PowerToy itself (which are going to be further developed in future releases of this PowerToy). Hoping that this discussion will help you and others in extending and adding further capabilities to DSL's (perhaps using automation), and we can develop a set of best practices for doing these kind of things.

I am certain this PowerToy won't be the last of its ilk, and hopefully some of you will find the discussion interesting enough that you might feel compelled to go ahead and innovate and package your own DSL enhancements, and build new PowerToys yourself.

I guess in one respect, what I am going to describe here is 'a means or a pattern for creating pluggable add-ins for DSL solutions.'

What we wanted to achieve

The current release of the DSL Editor PowerToy simply adds a new tool-window to the 'DSL Package' project of the chosen DSL solution. This tool-window hosts a simple control that responds to user selections. Next releases of the PowerToy will build upon this theme in various ways but the implementation patterns of the mechanics of the PowerToy itself will change little in how we basically achieved that. In this respect, this PowerToy provides a certain capability to enhance a DSL.

Our broad basic requirements for this tool were:

Make use of it as simple and clean as possible with the minimal amount of user interaction, and maximum amount of automation.
Make it trivial to remove/reinstall its capability should the developer decide not wish to go with it (try-it-out mode).
Provide an easy means to customize the capability so that a DSL developer can modify/enhance the moving parts of it with ease.

This last one is a major requirement, since we fully expect that in most cases many DSL developers will want to rip-and-replace certain parts of the capability for their own purposes - we want to facilitate that.

In order to satisfy these broad requirements we needed to come up with a basic pattern of how to provide additional capability to the DSL with as little impact on the existing DSL as possible, that is both highly customizable and integrated into the current developer-build process (Transform Templates->Compile->Run).

Basic Requirements

So to summarize, these are the basic things we wanted to achieve in the development of the PowerToy:

Add the new capability to the DSL project(s) using a simple automated, contextual, wizard based, configuration approach.
Minimise and simplify the amount of configuration information gathered from DSL developer.
Integrate components with existing generated DSL classes (making no assumptions about naming, namespacing etc. of DSL components).
Modularise the new capability into easy to understand 'add-in' architecture and physical structure.
Promote, and accommodate common scenarios for customization of the capability.
Integrate with standard development process for building DSL's. (i.e. no special build/configuration requirements)
Remove/re-install the capability entirely (as best we can), preserving any customizations to it.

How we achieved that

The general approach we took to designing and building the actual PowerToy was the same approach we take to build any Automated Guidance (or Software Factory) described in detail here.

Creating an RI

The very first thing we did before building the PowerToy itself was to create a prototype of the capability we wanted to build. Then we generalised and refactored that capability in a gradual and iterative process separating the common components and identifying the variability if the components that would have to be specific to the DSL. Fortunately for us (well, actually by design before we started), we had already prototyped this capability in a previous DSL solution, so we simply isolated this capability from that solution into a mini-solution for refinement. (Basically we built a reference implementation (RI) of it). Harvesting existing assets in existing solutions is by far the best means to explore the domain of the thing you want to automate.

Commonality/Variability

Without going into the gory details of the whole process of building guidance automation assets here, it's suffice to say that the outcome of that process was that we refactored out some common classes into some common assemblies, and teased out the specific (non-general) code that we needed to add to the DSL solution. (The common assemblies would be installed to the developers machine at install time of the PowerToy itself, and referenced by the specific code).

As part of the process, we quickly identified the variable parts of the components we needed to add, and all that was left to do was find a way to add the specific code and references the common libraries to the DSL solution, in such a way as to meet the other requirements stated above.

It was pretty clear from the RI, at the start, what actually needed to be physically done to add our tool-window editor capability to the existing DSL solution. In other words, what source and files we needed to add to the solution. Most of the components of this PowerToy's capability could be simply appended to the components already generated in the DSL solution.

Thanks to partial classes, and the DSL toolkit's provisioning of them, in most cases we could add our capability by just adding partial class source files to the DSL solution. However, in at least one area requires us to write code snippets into existing configuration files already present in the DSL solution. This presented challenges of its own, particularly to meet 'clean removal' requirements.

Templatizing Artefacts

Our RI identified the required source files, and code snippets that needed to be generated and added to integrate with the DSL solution. The next step was to templatize the RI source code and identify which parts of the source code (particularly identifiers) needed to be generated by the DSL solution itself. We wanted to make no assumptions about the naming of the DSL classes we were integrating with, and we didn't want to explicitly read configuration files (like the *.dsl file, or use reflection) to determine their names and namespaces either. We also wanted all our code to be part of the same namespace as the all other DSL code.

The DSL toolkit makes extensive use of the Text Templating Engine to generate its DSL classes in source files by transforming the domain model designed by the DSL developer. In fact if you look carefully, the *.dsl file is actually included in all the *.tt files that is read as input when you use the 'Transform Templates' command whilst developing your DSL, and this is processed by the text template at transformation time.

<#@ Dsl processor="DslDirectiveProcessor" requires="fileName='..\..\Dsl\DslDefinition.dsl'" #>
<#@ template inherits="Microsoft.VisualStudio.TextTemplating.VSHost.ModelingTextTransformation" #>
<#@ output extension=".cs" #>

Since one of our requirements is to leverage this transformation mechanism (i.e. not define a separate compile/build process for our PowerToy's capability) we needed to use the same mechanism to generate our specific code for the capability. It was pretty clear at this stage that we will be adding text template files (*.tt) to the DSL solution to generate our code.

To create these *.tt files, we simply work backwards from our RI code and replacing any DSL component identifiers with directives in the Text Templates. These text template files, will then include the *.dsl file as an input (as above), added to the DSL solution, and will participate in the normal text transforming process.

You can see an example of one of these text templates from this PowerToy.

Modifying Existing Artefacts

As mentioned earlier, most of the RI code can be added to the DSL solution by simply adding partial class source files, however we also needed to add code snippets to existing files in the DSL solution. This may also be a quite common means to integrate a capability into a DSL.

The *.ctc file is one example of this, where, in order to add VS menus and commands to the IDE, we need to write entries into the Command Table file at specific locations in the text. Other types of files may not be as structured and formatted as *.ctc, but in this case we could leverage that structure to append specific text one line at a time at specific locations in file. This of course had significant advantages when we needed to remove the appended text, if the developer wanted to remove/reinstall the tool-window editor.

[This remaining part of this section is kind of a sideline, but nonetheless might be interesting if you find yourself having to modify similar files.]

The CTC command table format is rather esoteric, but nonetheless it is predictable and assuming that the DSL developer maintains the standard format and layout in tact, additional commands can be easily added. The actual configuration for the commands is rather complex and can be variable (definitely customizable by the DSL developer), and to aid modularisation of this configuration the CTC file supports #define's, which allow us to #define the actual menu commands separately in another file, and only requires us to write in macros that will be expanded at CTC compilation time.

Firstly, we needed to add a #include statement at the start of the *.ctc file, which points to another *.h file that contains the #defines for the actual macros. This included *.h file is the one added by the PowerToy at a known location in the DSL solution, and contains the actual menu commands. Firstly, we add our #include statement after the other #include statements we expect in the *.ctc file by default. All that remains to do is to write in the various macros (i.e. 'DSLEDITORPOWERTOY_CMDPLACEMENT' and 'DSLEDITORPOWERTOY_BUTTONS') into the *.ctc file at the appropriate locations. We can do this by appending the macro text before the end of the relevant command table sections (such as: 'CMDPLACEMENT_END' and 'BUTTONS_END'). The resulting *.ctc file would look something like this:

#include "stdidcmd.h"
#include "vsshlids.h"
#include "msobtnid.h"
#include "virtkeys.h"
#include "DSLToolsCmdID.h"
#include "..\GeneratedCode\GeneratedCmd.h"
#include "..\GeneratedCode\DslEditors\CustomCmd.h"

CMDS_SECTION guidPkg
    MENUS_BEGIN
        GENERATED_MENUS
    MENUS_END
    NEWGROUPS_BEGIN
        GENERATED_GROUPS
    NEWGROUPS_END

    BUTTONS_BEGIN
        GENERATED_BUTTONS
        DSLEDITORPOWERTOY_BUTTONS
    BUTTONS_END
    BITMAPS_BEGIN
    BITMAPS_END
CMDS_END
CMDPLACEMENT_SECTION
    GENERATED_CMDPLACEMENT
    DSLEDITORPOWERTOY_CMDPLACEMENT
CMDPLACEMENT_END

The advantage of this approach with this kind of structured text file is that when it comes time to remove the capability, we only have to identify the lines on which the #include and macros exist, and simply delete those lines. (Again assuming the DSL developer maintains the clean format of this file).

Installing Artefacts

Now that we have our code templatized in text templates, and we know which code snippets need to be added to which existing files, the next challenge is to figure out where to add the text templates, and what else needs to be done to integrate the tool-window editor into the DSL solution.

At this point, from our RI, we have established that we need to do the following to integrate our tool-window editor:

Add a number of text templates to the DSL solution (which generate partial class source files, resource files, header files etc.)
Add static supporting files to the DSL solution (i.e. *.bmp files)
Modify existing files in DSL solution (i.e. *.ctc files)
Add assembly references to supporting common assemblies.

Modularisation

As with any add-in type functionality it is important to modularise your additional functionality. To facilitate add-in type add/remove functionality and clean separation of PowerToy artefacts from the rest of the DSL solution, it is necessary to group all PowerToy created source artefacts in the same folder.

DSL solutions, by default, separate their generated code from the source artefacts that the DSL developer works with at design time (i.e. the Domain Model designer *.dsl). They do this by placing all text template files within a folder called 'GeneratedCode' (by default). The DSL developer is not expected to modify these generation files, instead work with the Domain Model Designer (*.dsl) file to make any customizations to the DSL. It is the domain model design that acts as input to the generation files.

One of the objectives and requirements of the PowerToy is to provide a simple means to configure its capability and require little to no maintenance of the generated code from the DSL developer. For this reason, the PowerToy employs a similar strategy to the DSL toolkit for adding all source artefacts to a single folder. By default, this folder is located under the 'GeneratedCode' folder in the DSL solution.

In the first release of the PowerToy, the only difference between the semantics of the use of the PowerToy subfolder and the 'GeneratedCode' folder from the DSL solution is that there are source artefacts (such as *.resx files) that the DSL developer can customize the workings if the PowerToy's capability. This folder is also the intended location for additional source files from customization of the capability.

[In future releases of the PowerToy, there will be other designer artefacts with which the developer works with to make a cleaner separation between generated code files and design time artefacts.]

Supporting Install/Uninstall

One of the primary requirements of any add-in type mechanism is that it can be added or removed from the solution at will - it's pluggable. The objective should be - to have minimal impact on the existing DSL solution if a DSL developer added the capability and then removed it sometime later. In order to achieve this, for every installation type action listed above, we need to be able to reverse it in the 'removal' mode (and reset it in the 'reinstall' mode).

The basic philosophy here is that the PowerToy would undo any modifications it made to the DSL solution, whilst preserving any potential customizations the developer had made to its capability (within reason of course).

The tricky part is that, DSL solutions are innately flexible in their structure and naming, and there are little to no limitations on how a DSL developer can modify the projects of a DSL solution to suit their needs. The moving parts of DSL solutions make almost no assumptions about how the projects are named, and physical structure etc. It is feasible, for example, that a DSL developer could decide to combine all artefacts from both DSL projects into one project and completely restructure it, renaming its folders and files etc. This might be at the extreme end of the customization spectrum, but it is quite likely that every DSL developer will have their own strategies for customizing their DSL solution and how to structure customizations in their DSL projects. Certainly, for any customization that requires additional code artefacts, a developer may choose to restructure or enhance the structure of the projects. Also, its is feasible, especially in the software factory context, to have more than one DSL in a single VS solution.

This makes it reasonably challenging to perform uninstall type operations since the PowerToy can make very few assumptions about how the DSL developer could move files around and rename them to suit their needs prior to install, or between install and uninstall of the PowerToy's capability.

Grouping and modularisation of the installed artefacts (i.e at least placing all files in same folder) alleviates some of these issues, but does not eradicate them, or deal with all possibilities. Since there is no built in infrastructure in DSL solutions to aid us here, the PowerToy can only go so far to deal with this solution flexibility. (Perhaps a DSL PowerToy framework would be useful for this!).

Looking at the list of actions defined above, we simply need to be able to:

Remove/replace the files (and subfolder) we added.
Delete/replace the modifications to any modified files.
Remove/replace the assembly references.

However, most of those actions require us to reliably know (a) where the added files reside now, (b) where the modified files reside now. (assembly references are only located on project, so they are always predictable to modify). Furthermore, there are in fact, by default, 2 projects in a DSL solution, and some actions only apply to one or other of the DSL projects. (that also assumes the DSL developer didn't split or merge parts of these projects).

To resolve these issues with the minimum amount of assumptions being made (and without building a complex infrastructure around it all), the PowerToy employs the following strategy, to accommodate non default configurations.

On install:
- Prompt user for appropriate DSL solution project
- Create a subfolder in a user defined location on the DSL project
- Add all files to this subfolder
- Modify existing files, prompting user for location of specific file in DSL project
- Add assembly references to the DSL project
On reinstall:
- Same as on install, overwriting any existing files/modifications
On uninstall:
- Prompt user for added subfolder, in DSL project
- Delete all added files, leaving any additional files
- Delete subfolder only if empty of files
- Delete modifications to modified files, prompting user for location of modified file in DSL project
- Remove added assembly references from DSL project

The last remaining issue is how to determine which is the right DSL project (of the 2 default ones) to operate on. In this release of the PowerToy, we are only interested in the 'DslPackage' project (which contains the VSPackage class), and we have no modifications at this stage for the 'Dsl' designer project. In subsequent releases of the PowerToy we may need to perform different actions on different DSL projects.

The DSL toolkit does not provide any way to indicate which of the default 2 projects is the package project (for example using metadata). There are few, if any, reliable ways of determining the package project from the designer project, especially when you consider that the projects and their structure can be modified in any way the DSL developer chooses, and further, any given VS solution can contain any number of DSL's.

One could argue several techniques of determining the package project, you could reflect over the assembly of the project and determine if it contains a package class, but even this requires that the project has been built successfully at the time of deploying the PowerToy. Also, this does not deal with cases of multiple DSL's in a VS solution either.

Clearly, the only reliable means available to us in reliably selecting the right project that contains the package class, for the DSL of choice, is to ask the PowerToy user to select it from the list of projects in the VS solution. Once, we know this project, we can also ask the user to define the subfolder in which contain PowerToy files, and prompt them to locate the specific files to be modified by the PowerToy. Of course, we can assume certain defaults for these values to aid usability, based on what comes out of the box from the DSL toolkit, but we need to provide for the potential flexibility.

Now, when the PowerToy user comes to remove the PowerToy, it would be helpful to remember any settings we used on the install to act as the default values on the uninstall, particularly folder paths and specific modified files. We can also validate these folders and files exist currently.

It also comes in handy to mark the modified projects with specific PowerToy metadata, so we know which projects were modified by the PowerToy. Marking the projects and saving installation metadata is done by saving values to the projects global property bag on install.

As it turns out, this metadata 'marking' helps us drive further actions of the PowerToy and indicates the right context for the Add/Remove/Re-install actions.

Applying Guidance Automation

At this stage, we have resolved most of the challenges of installing and uninstalling the PowerToy capabilities. The final part to this PowerToy development is applying guidance automation and making that automation context aware so the user has the right actions available at the right times for the right purposes. We could of course get the DSL developer to perform the necessary actions to install the PowerToy's capability, but automation makes that trivial and reliable.

The primary areas to apply automation for such PowerToys are typically:

Installing the capability
Configuring the capability
Re-configuring the capability
Uninstalling the capability

In this simple release of the PowerToy the configuration of the capability was done during installation of the capability. The capability can be re-installed, at which time it will be re-configured. Later release of this PowerToy might vary with respect to when configuration/re-configuration takes place, since the configuration of later releases is expected to be vastly more complex.

However, each of these actions is a good candidate for full or partial automation.

Adding Recipes

Recipes are ideally suited to applying automation to the areas identified above. A recipe is a simple strategy for gathering information and executing a sequence of actions that provide the automation part. Recipes can be made context aware by only being active under certain conditions when the state and context is correct. This state and context can be derived from any source within the development environment, but typically it is based upon the users current actions. In this case, when the user is designing a DSL solution.

For this PowerToy, from the previous section we can make the recipes available at anytime during DSL solution development. The recipes can then prompt the DSL developer for the various settings they require to execute.

As mentioned before in the previous section we used metadata on the DSL projects where we installed our capability, and we also used metadata to save values used in the installation, to be used as default values at uninstallation time. This metadata provides us the context we need to determine where and when the various recipes are valid, and what recipes could be invoked at any particular time. The recipes themselves also manage the adding/removing of the metadata itself.

The installation recipe and uninstallation recipe for this PowerToy release are programmed with the sequence of actions already discussed in previous sections. You can see the parameters (arguments) that are collected from the user, the wizard that is used to gather the data from the user, and the sequence of actions that are performed.

Generating Text Templates

The last point that is noteworthy in this discussion is to do with the installation of the text templates that generate the code for the PowerToys capability.

Part of the configuration gathered by the recipes on installation (and configuration) needs to be written into the actual content of the text templates the PowerToy adds to the DSL solution. This is to customize the templates to the current DSL solution. Currently, this would include things like captions for resource files, and other parameters gathered from the user and recipe.

In order to do this in a recipe, the text templates themselves are rendered using the text templates themselves!, and the configuration gathered from the user and environment is used as inputs to the text template rendering.

Careful formatting of the text templates is required to support this 2-stage rendering, but this is an elegant pattern for customizing text templates for application to a specific DSL solution for generating source for a PowerToy capability.

You can see an example of one of these 2-stage rendered text templates from this release of this PowerToy

Facilitating Customization

So far we have seen how to build a PowerToy that installs a configured capability, but what about customization of that capability?

Since we are installing text templates that get transformed into source code during the compile time of the DSL, we are limited in how to customize this capability? Surely, that would involve editing source code in the text templates themselves wouldn't it? - Well, yes, yes it would. However in most cases, all you want to do is customize the code by providing your own methods for a class, to enhance an existing method or change the class's behaviour. The typical way you would want to do this is using partial classes, but that can't be done here since you can't rip and replace methods using the partial class machanism. You might then be able to use inheritance and subclassing (in some cases), but it's not your subclassed type that is going to be consumed by the existing executing code. To make that work, you would need to make changes to the existing text template code to explicitly use your subclassed type. None of this is elegant, and hampers seperation of the code the the capability and the code for customizing or configuring that capability.

The DSL Toolkit developers figured out a neat way around all this called the 'double-derived pattern' that gets around most all these issues very, very elegantly. Basically, what this amounts to is: instead of just defining a class and its' methods in a text template, you instead create an abstract base class containing all the properties/methods (which are now virtual) and you create a concrete derived class of the base class in the text template.

Now, as the DSL developer customizing the capability, you can create a partial class definition of the concrete class and modify its behaviour. It will be the concrete derived class that gets consumed by existing executing code, and this should require no changes to any text templates.

Together, text templates utilizing the double-derived pattern provide a strong foundation for facilitating customization of a PowerToy capability, and seperation of the PowerToy capability code from the DSL specific customization code of that capability.

Summary

In this post we discussed in detail the issues with creating a PowerToy for DSL solutions. In particular, we have described some of the general patterns and strategies that could be used to create DSL PowerToys. In this respect, these PowerToys to being modular plug-ins for DSL's, that support add/remove like capabilities and use automated guidance to provide easy operation.

To summarise we discussed the following patterns and strategies:

Building Reusable Assets - the process for developing guidance automation assets from existing solutions. (RI->CV analysis->Asset creation) Described in more detail here.
Creating contextual source artefacts - using text templates, and leveraging the incumbent DSL build process.
Adding modular capabilities - separating and modularising capabilities, packaging them for easy maintenance, configuration, and removal.
Applying Guidance Automation - automating the installation, configuration and uninstallation of the capability.
Facilitating Customization - designing text templates that employ the double-derived pattern for intentional customization.

I believe that these patterns and strategies are re-usable in the development of other PowerToys, which may lead to a future common framework, practices and guidance around DSL PowerToy development.

Resources

The DSL Editor PowerToy, used as an example for this article, is an open source community project hosted on CodePlex, you are free to download the releases of this PowerToy to examine the details of how it was built, and participate in the evolution of this PowerToy.

DSL Editor PowerToy - Just Released

jezzsa — Thu, 15 Mar 2007 02:24:00 GMT

You might be happy to know we have just released the first version of the ‘DSL Editor PowerToy’ on our codeplex project!

This initial release is slated as:

“A means to add a DSL specific tool-window, containing a simple editor, to an existing DSL”

You can download this release from the site.

It's not much really in terms of functionality at present, and the editor is really basic, but what we tried to accomplish in this release was setting up the infrastructure of what is to come.

We have a pretty decent plan of the features we want to work towards in later releases, and we have carefully designed the releases so that you can have something useful to take your own way at each step. The ultimate goal is demonstrated on the home page of the site, and this is what we are working our way towards providing everyone with a means to have multiple editors each exposing a customizable view of their DSL.

Some basic links for this release:

The home page of the release
How to install, and configure the PowerToy
How to customize the tool-window
Detailed design discussion

If you have any feedback about this release or the PowerToy in general, we would like to hear it.

DSL User Experience PowerToy!

jezzsa — Thu, 01 Mar 2007 20:53:00 GMT

If you are building DSL languages today, you may have experienced at least one of the following issues when figuring out how to represent your languages' domain model on a diagram surface:

Representing most of the domain classes as shapes makes the diagram really complex and cluttered, and therefore arduous to navigate around for the user
Box and lines are not always ideal for representing parent/child relationships
You can use compartment shapes to describe one level of hierarchy on the diagram, but no more
Representing some of the domain classes as connected shapes really does not provide much additional useful information to the user
Configuring multiple similar shapes, with unique values by individually clicking each one and then changing their domain properties via the 'Properties Window', is pretty mouse intensive and arduous for the user
It is hard to get a view of your language that focuses on one particular aspect of the domain

If any of the above are issues for your language, you might consider checking out the 'DSL Tree Grid Editor PowerToy' that we are building to resolve these and many other usability issues you may face designing your DSL today.

The PowerToy enhances your language by providing tree-grid-like editors (displayed in a tool-window exclusively for your language) that allows you to expose hierarchical domain relationships, and allow the user to navigate and configure the domain classes in the hierarchy with the keyboard.

The related domain classes are displayed in a grid which makes it easy - at a glance - to compare their important properties side-by-side, whilst configuring and creating new domain classes real easily within the editor.

The PowerToy allows you to add multiple of these editors; one for each 'aspect' that you wish to expose of your language. In effect, each editor acts as another view into your domain model.

As an example, see the DSL with editor below.

Multiple or Single Architectural Models/Views?

jezzsa — Wed, 22 Nov 2006 14:30:00 GMT

This was to be a FAQ, but due to the fact that the practices around this are unproven yet, there is no definitive answer to this question, rather a discussion of approaches, issues, guidelines and raising general awareness.

At the moment, there is quite a buzz about the correct usage of DSL's and modelling domains appropriately in the context of software factories.(within Microsoft at least, I would expect the rest of the DSM community have a more mature view on this).

We are starting to see an emergence of software factories of different varieties. Much of this innovation is growing from an absence of a formal framework for building factories and mechanisms to stitch them together, with navigation etc. (which may one day be available to us all).

A lot of the innovation is being put into how to describe a whole domain with multiple models, and when and how to relate those DSL's together. Services and practices are starting to emerge such as the integration service introduced by Mauro and Victor in their article on Integration Scenarios which allows you to reference elements from one model to another. Other factories, like the Ordina Factory, are using other techniques to achieve the same ends.

There appear to be two extreme views on how to model any part of a domain (sub-domain). On the one hand, some folks are using very few (or single) models to cover the sub domains. Others are decomposing the sub-domain into very many models.

Now, we need to be accurate here. There is a difference between the actual model and view (of the model). Ideally, you could have multiple models with multiple views, each view considers a whole model or part of a model. In some technologies a view can span multiple models also. Its the model that describes the logical architecture, and the view that provides the interface of that to the user. For the user, its the view that's important.

[It should be noted at this stage, that multiple views for one model is currently not supported in the Microsoft DSL toolkit. Views to span multiple models is also not supported currently. Only one view per model is presently supported.]

In the context of architectural guidance in a factory, imagine we are modelling a service, for example. Architecturally speaking, this is composed of several abstraction layers, and few deployment tiers. Lets consider the following standard .NET architecture for service orientation from patterns & practices. Since this is quite a popular architecture to create factories around at present.

Considering the abstraction layers in the logical architecture, the objective is to decompose the architecture into layers, with differing concerns, all in the scope of the whole service. i.e. Service Interface, Business Layer, Resource Access layers and their component layers.

Now undoubtedly each layer of the logical architecture is concerned with different aspects of the overall solution, and further, each layer has more than one concern to address, as well as the variability of that layer and its components.

The question is how best to apply DSL's to model these concerns and variability?

Jack addresses the same question is his article on bare naked languages.

There are a number of patterns that address these kinds of architecture, and its typically these patterns that determine the answer to this question.

However, I assert that basing your models and views upon these patterns alone, may not always be the best approach to exposing your DSL's to the users.

Multiple Models

To be a purest about separation of concerns, when analysing a problem domain, you would take the approach to define a single model for each layer which addressing a small number of concerns and the variability of that layer and its components. Or perhaps even, a model per concern!

Ideally, you would present several different views (if possible) of the model perhaps each one exposing each one of the concerns, or maybe just one view for all the concerns (which is the case today with using the DSL toolkit).

Then you would develop an appropriate mechanism to reference components from one model to another.

For example, a separate model for each of the layers:

A model for the Service Contract, and Service Adapter,
A model for the Business Logic implementation
Perhaps a separate model for the Business Entities
A model for the Data Access Logic
A model for the Service Agents

That's 6 models, but some could be combined. So in reality 4-6 models to cover the whole architecture.

All models address the concerns and variability of their layer.

Single Model

A more natural (perhaps less purest) approach, would be to define a single model for all layers which addresses all the concerns and variability.

This does not necessarily infer that appropriate emphasis has not been given to separation of concerns. It just means they are defined in the same model instead of separate models.

Ideally, you would present several different views (if possible) of the model perhaps each one exposing each one of the concerns or aspect of variability, or maybe just one view for all the concerns and variability (which is the case today with using the DSL toolkit).

For example, a single model for all layers. (example here from the EFx factory).

Considerations

Considering these two approaches (in this context).

These approaches aim to achieve the following:

Multiple Models

Maximum physical separation of concerns.
Maximum physical de-coupling of the layers.
Maximum flexibility in reuse of one model in another architecture.
Facilitate better source control in team environments

As developers can work on different parts of the architecture in isolation from others.

Single Model

Maximum holistic description of the architecture (big picture view).
Maximum usability.

The challenges with these approaches (in this context):

Multiple Models

No holistic view of the architecture. The inter-related pieces are presented in different views.
The interface to reference dependencies in other views is usually poorly visualised/represented.
Information to perform common user tasks maybe spread throughout several files.

Single Model

Complexity of diagram can be high, if the abstraction of the model is low.
Team development on same model can be challenging, as the view/model could be used by different roles addressing different aspects.

[If you have any other pros and cons to add I be glad to add them.]

Learnings?

So what can we learn from this? I believe neither of these extreme approaches is ideal. The answer lies somewhere in-between - a compromise of both approaches.

The emphasis of the views that you provide for your models, needs to be high usability. A software factory is a productivity tool for users that should not have to have the level of knowledge and experience that those building them have. So even though your model(s) employ the best architectural practices in separation of concerns, and pattern application etc, these should not necessarily be forced upon the users. Instead, these patterns and architectural practices should be abstracted in the views, in a way, that makes it easy for the users to get the job done in describing the solution domain. (Its the classic: Usability <-> Flexibility trade-off struggle)

Perhaps building the models and the views are two different roles of defining a model for your domain?

Here are some things to consider to make the modelling solutions (in a factory context) more usable:

Use fewer models when the abstraction is high and variability is low. Use many more loosely coupled models when the abstraction is low and variability is high.
For single models, use multiple instances of the model file to partition the solution into bite-sized chunks, and use cross-model references where applicable.

This will assist in team development.

For multiple models, provide holistic views of the other views that bring them together to form a composite 'zoomed out' view of how the models relate, so the user can get a feel of the whole picture rather than one aspect of it.

This will enable the users to stay focused on the objectives of the solution.

For any approach, analyse which layers are more tightly bound to others, and require less flexibility in reality - from the users perspective.

For example, in the above example, it is unlikely that the 'Resource Access Layer' is going to be used separately from the 'Business Layer', but it is highly coupled to it architecturally speaking. So it would make sense from a usability perspective to combine these layers into a single view.

For any approach, use views that expose particular concerns or aspects of the architecture.

A good use of these views is by role, so for example, a security expert can view the cross-cutting concerns of authorization across the whole architecture, without having to understand all other aspects or abstractions of the architecture.

[Again, if you have any other considerations to add, I be glad to add them here.]

Variability - What is it, and when to use it?

jezzsa — Sun, 19 Nov 2006 13:48:00 GMT

'FAQ - What is a Software Factory?' described what a 'product line' is, and discussed how the variants of the product line are basically defined. In this article, we want to dig a little deeper into what is variability, and how you define your product variants.

What is Variability?

So, each product that your factory creates will have certain aspects of it that can be variable, by changing these aspects you get a different 'product variant'. Maybe, these aspects are behavioural, maybe they are architectural. Either way, the factory needs to offer a means to specialise the product instance to suit certain end-requirements. Each aspect of the product that varies defines a point of variability of the product. So, if you configure this point of variability you end up with a different 'product variant' by definition.

So, you wouldn't go ahead and create two factories that created almost the same thing, if those two things if those two factories could define the same variability point for the same solution domain. Instead your factory would identify that its product instances share some common logical architecture and maybe only vary in appearance, behaviour, or even in physical architecture. Your factory would then define a shared logicalmodel of the product and offer a means to configure the variable aspects of it.

Now, a certain number of the variants will share the same set of configuration for a given set of variability points. You start to define the common product types your factory creates. You no longer require your factory user to define every single variable point, you can simply provide them a 'template' (a type) that fixes some of the variability points to known values, leaving the other variability points for specialisation.

I need to emphaise that at this point, we are not talking about the variance of the physical instance of the product (source code etc.), we are talking about the variance in the logical model of the product. The degree to how much the physical representation differs from the logical model will depend on the abstractions offered by the factory. It is feasible that there are many physical representations for the same logical model that differ by the configuration of the model.

Your factory's 'solution domain' is bounded by the product variants your factory makes. Although not precisely accurate, (depends on your factory) your logical product model defines your solution domain too.

How to Identify variability?

So, how do you identify the points of variability in your product?

There is plenty of information on the web about the practice of Commonality and Variability Analysis (CVA) which is basically how you identify what varies in your product and to what extend it varies.

The key pieces of variability to identify are logical architectural variance and variance in the components of logical architecture.

Interestingly enough, if you are building a factory from an existing framework, this practice has already been employed to a certain extent, as this is one of the steps taken to get from implementation to class library and then to framework.

How to Represent Variability

Once you have your points of variability, the next question is how to represent them in your product model.

In the simplest case, a point of variability could be represented in your model as a configuration setting: such as an integer for the number of cyclinders of the engine, or perhaps as a set of configuration settings, which may be defined for one or more elements in the model.

In more complex case you may need a slighly enhanced logical architectural model, to represent a different product type: such as, one model for the petrol engine, one for the diesel engine.

The art of defining the product model is identifying what is the minimal set of variability points, and hence their configurations or architectural representations, you need to define the types of the product variants in your solution domain.

How to Use/Abuse Variability

We said before that the best factories will be those with a highly specialised domain, and those with the best implementations for that domain. The question is how to define that domain and still create a successful factory?

I know - you just can't wait to break down your solutions into a product and types, and the possibilities are endless on the variability you can model - right?!?

Now here is the problem. Given any solution it is conceivable, in most cases, to decompose the solution, identify the product variants of ever finer granularity of variability, and define extensive configuration or models to address that variability. After a time, your model has so much variability it starts to look like a programming language. After all, high level programming languages are essentially a low level form of configuration – aren't they? Using this approach would yield a factory with a huge number of variability points, which results in potentially large product line, and therefore - a very generic factory!!!

You might initially think this is ideal? Well as it turns out - yes, it may appear to be ideal - except when you realise that this approach is not far from where we are today with a generic solution authoring tools like programming languages and class libraries.

The problem is that as the number of variability points increases, so do the number of possible variants, and therefore, so do the number of paths you can take to configure any given variant, and since its highly configurable how do you keep control of which variant you are working towards at any given time? And with all that configuration, it's very hard to validate and control the values of interdependent configuration. Not to mention, the amount of time it takes and the complexity to configure this model for the factory users.

It's exactly the same issue that we have with generic frameworks/libraries today – 'they create everything useful and nothing useful - all at the same time' and everyone has their own way to do it and much of it is unverifiable due to the large number of possibilities.

The other side-effect of this approach, on the solution side, is that it's very hard, if not impossible, for your factory to provide specific optimised solutions for you products if you have to cater for so many variants of them. (No single factory author nor tool can possess that many domain specific skills, experience, knowledge to create the most optimal solution to such a large set of variants).

Another way to look at this: - your factory is doing little to provide known specialised solutions to known specific problems, so why would anyone want to use it?

You want to beware of those factories that claim to create *anything* you want, or where the product line is very large.

It's all about being a specific and as highly specialised as you can, which means you are creating a small set of very high quality products. To be successful, your factory needs to provide highly specialised, proven solutions to known specific problems that accommodate some variability for customization.

The most successful, and hence most attractive factories, will be the ones that have a narrow well-defined domain (i.e. a small number of product variants). The physical solutions will be built by the best people, most experienced and skilled that solution domain.

I dislike to over-use automobile industry analogies, but they can be useful sometimes.

Have you ever wondered why the highest quality car manufacturers in the world only create a small number of high precision models tailored for certain markets?

The car manufacturers that make 'normal' cars, (for folks like you and me ), create the 'baseline' products (like the chassis with wheels and engines). Then other factories that tailor them to a given market, which only really add the bodywork, paint and interior, and ornament, depending on the size of your pay-check.