Jezz Santos : Guidance Automation

New Software Factories Community Site

jezzsa — Sat, 05 Jan 2008 04:22:20 GMT

I am a bit tardy on the blogs about this, but if you haven't heard yet - we recently opened a new site dedicated to building and sharing a community of knowledge around Software Factories.

http://sf.devrevolution.com/

Huge thanks to Martin Danner who has hosted the site for us, and Edward Bakker who has championed the creation of the community. For some time we have aimed to provide a central place online we, as a community, can collect, discuss, rank, comment and order content about factories and Visual Studio Extensibility as this methodology gains momentum.

There is a mother site to this one (DevRevolution) which aims to address other industry topics such as Agile Development and Application Lifecycle Management as revolutions to the software development industry. However we are starting of humbly primarily focusing our attention on software factories at present. [Which just quietly, I see as an evolution not necessarily a full revolution to how we build some types of software - anyhow]

There are many sites and resources on the web about and around software factories from many of the great folks who have pioneered this space before, and heaps of information about its fundamental technologies and practices (i.e. domain modeling, automation, generative programming etc.) so we wanted to provide a central place to bring this knowledge together and have you guys - the community - view it all in one place. The idea is to then have the community collate, rate, comment and sort the content so that anyone visiting the site can find the most relevant content to consume based upon there understanding of this space. Clearly, no one wants to trawl hundreds of disparate links to sources of info to mine and find the best or the highest quality information on this topic. So having the community rank, comment and direct the visitors to the best content is an important attribute. So this is our primary goal - to enable that. We also want a place for people to come to and feel part of a larger industry group of like minded folks practicing this methodology developing its tools and practices.

We are currently a little centered on Visual Studio Extensibility as the platform for software factories at present simply because Microsoft is leading the Software Factories initiative at present, but we would like to think that the concepts and practices are not necessarily exclusive to Microsoft technologies, and would like to support ideas from other platforms and technologies in the industry.

Of course we realize that much of the relevant content out there for consumption is addressable only in the heads of many of the people involved in and around the factories initiative. We have linked some resources (as you can see), but we are relying on you to add whatever resources you are aware of.

We are focusing on refining the site and providing improved means (over what you see there now) to provide a better user experience and means to better add, rank, comment and sort the content best. Basically we are aiming to have each piece of content referenced on the site to support attributes, comments, ratings and categorization.

Currently the modules we have are a step in that direction, some parts allow this capability, others don't. So we have much work to do to reach that goal. It's a little ambitious for us, but we recognize that others are reaching out for such as resource.

However, we still need more content and feedback to help drive that.

Please have a look and let us know if you think this is useful, what is missing and how it should work to be the tool this evolving community will treasure.

[If you visit the site, and don't submit any content or offer any honest constructive feedback, then I hope you leave feeling guilty that you have not really participated in bettering the community as a whole - we especially need and value your comments and feedback.]

Some links:

Building DSL Enhancing PowerToys - Good Practices

jezzsa — Fri, 13 Apr 2007 10:50:00 GMT

If you are building DSL's and you haven't heard yet, we released the Editor PowerToy last month. The current release is only a small step towards the vision we have for the PowerToy that will deliver a very powerful capability to provide additional customized views in a tool-window (for your DSL diagram) that enhance the runtime usability of your DSL.

"All well and good" you might say, but this might not be that interesting to you anyway, since you have no use for the PowerToy presently.

Nonetheless, in this post I wanted to describe the design, techniques and patterns we used to create the PowerToy itself (which are going to be further developed in future releases of this PowerToy). Hoping that this discussion will help you and others in extending and adding further capabilities to DSL's (perhaps using automation), and we can develop a set of best practices for doing these kind of things.

I am certain this PowerToy won't be the last of its ilk, and hopefully some of you will find the discussion interesting enough that you might feel compelled to go ahead and innovate and package your own DSL enhancements, and build new PowerToys yourself.

I guess in one respect, what I am going to describe here is 'a means or a pattern for creating pluggable add-ins for DSL solutions.'

What we wanted to achieve

The current release of the DSL Editor PowerToy simply adds a new tool-window to the 'DSL Package' project of the chosen DSL solution. This tool-window hosts a simple control that responds to user selections. Next releases of the PowerToy will build upon this theme in various ways but the implementation patterns of the mechanics of the PowerToy itself will change little in how we basically achieved that. In this respect, this PowerToy provides a certain capability to enhance a DSL.

Our broad basic requirements for this tool were:

Make use of it as simple and clean as possible with the minimal amount of user interaction, and maximum amount of automation.
Make it trivial to remove/reinstall its capability should the developer decide not wish to go with it (try-it-out mode).
Provide an easy means to customize the capability so that a DSL developer can modify/enhance the moving parts of it with ease.

This last one is a major requirement, since we fully expect that in most cases many DSL developers will want to rip-and-replace certain parts of the capability for their own purposes - we want to facilitate that.

In order to satisfy these broad requirements we needed to come up with a basic pattern of how to provide additional capability to the DSL with as little impact on the existing DSL as possible, that is both highly customizable and integrated into the current developer-build process (Transform Templates->Compile->Run).

Basic Requirements

So to summarize, these are the basic things we wanted to achieve in the development of the PowerToy:

Add the new capability to the DSL project(s) using a simple automated, contextual, wizard based, configuration approach.
Minimise and simplify the amount of configuration information gathered from DSL developer.
Integrate components with existing generated DSL classes (making no assumptions about naming, namespacing etc. of DSL components).
Modularise the new capability into easy to understand 'add-in' architecture and physical structure.
Promote, and accommodate common scenarios for customization of the capability.
Integrate with standard development process for building DSL's. (i.e. no special build/configuration requirements)
Remove/re-install the capability entirely (as best we can), preserving any customizations to it.

How we achieved that

The general approach we took to designing and building the actual PowerToy was the same approach we take to build any Automated Guidance (or Software Factory) described in detail here.

Creating an RI

The very first thing we did before building the PowerToy itself was to create a prototype of the capability we wanted to build. Then we generalised and refactored that capability in a gradual and iterative process separating the common components and identifying the variability if the components that would have to be specific to the DSL. Fortunately for us (well, actually by design before we started), we had already prototyped this capability in a previous DSL solution, so we simply isolated this capability from that solution into a mini-solution for refinement. (Basically we built a reference implementation (RI) of it). Harvesting existing assets in existing solutions is by far the best means to explore the domain of the thing you want to automate.

Commonality/Variability

Without going into the gory details of the whole process of building guidance automation assets here, it's suffice to say that the outcome of that process was that we refactored out some common classes into some common assemblies, and teased out the specific (non-general) code that we needed to add to the DSL solution. (The common assemblies would be installed to the developers machine at install time of the PowerToy itself, and referenced by the specific code).

As part of the process, we quickly identified the variable parts of the components we needed to add, and all that was left to do was find a way to add the specific code and references the common libraries to the DSL solution, in such a way as to meet the other requirements stated above.

It was pretty clear from the RI, at the start, what actually needed to be physically done to add our tool-window editor capability to the existing DSL solution. In other words, what source and files we needed to add to the solution. Most of the components of this PowerToy's capability could be simply appended to the components already generated in the DSL solution.

Thanks to partial classes, and the DSL toolkit's provisioning of them, in most cases we could add our capability by just adding partial class source files to the DSL solution. However, in at least one area requires us to write code snippets into existing configuration files already present in the DSL solution. This presented challenges of its own, particularly to meet 'clean removal' requirements.

Templatizing Artefacts

Our RI identified the required source files, and code snippets that needed to be generated and added to integrate with the DSL solution. The next step was to templatize the RI source code and identify which parts of the source code (particularly identifiers) needed to be generated by the DSL solution itself. We wanted to make no assumptions about the naming of the DSL classes we were integrating with, and we didn't want to explicitly read configuration files (like the *.dsl file, or use reflection) to determine their names and namespaces either. We also wanted all our code to be part of the same namespace as the all other DSL code.

The DSL toolkit makes extensive use of the Text Templating Engine to generate its DSL classes in source files by transforming the domain model designed by the DSL developer. In fact if you look carefully, the *.dsl file is actually included in all the *.tt files that is read as input when you use the 'Transform Templates' command whilst developing your DSL, and this is processed by the text template at transformation time.

<#@ Dsl processor="DslDirectiveProcessor" requires="fileName='..\..\Dsl\DslDefinition.dsl'" #>
<#@ template inherits="Microsoft.VisualStudio.TextTemplating.VSHost.ModelingTextTransformation" #>
<#@ output extension=".cs" #>

Since one of our requirements is to leverage this transformation mechanism (i.e. not define a separate compile/build process for our PowerToy's capability) we needed to use the same mechanism to generate our specific code for the capability. It was pretty clear at this stage that we will be adding text template files (*.tt) to the DSL solution to generate our code.

To create these *.tt files, we simply work backwards from our RI code and replacing any DSL component identifiers with directives in the Text Templates. These text template files, will then include the *.dsl file as an input (as above), added to the DSL solution, and will participate in the normal text transforming process.

You can see an example of one of these text templates from this PowerToy.

Modifying Existing Artefacts

As mentioned earlier, most of the RI code can be added to the DSL solution by simply adding partial class source files, however we also needed to add code snippets to existing files in the DSL solution. This may also be a quite common means to integrate a capability into a DSL.

The *.ctc file is one example of this, where, in order to add VS menus and commands to the IDE, we need to write entries into the Command Table file at specific locations in the text. Other types of files may not be as structured and formatted as *.ctc, but in this case we could leverage that structure to append specific text one line at a time at specific locations in file. This of course had significant advantages when we needed to remove the appended text, if the developer wanted to remove/reinstall the tool-window editor.

[This remaining part of this section is kind of a sideline, but nonetheless might be interesting if you find yourself having to modify similar files.]

The CTC command table format is rather esoteric, but nonetheless it is predictable and assuming that the DSL developer maintains the standard format and layout in tact, additional commands can be easily added. The actual configuration for the commands is rather complex and can be variable (definitely customizable by the DSL developer), and to aid modularisation of this configuration the CTC file supports #define's, which allow us to #define the actual menu commands separately in another file, and only requires us to write in macros that will be expanded at CTC compilation time.

Firstly, we needed to add a #include statement at the start of the *.ctc file, which points to another *.h file that contains the #defines for the actual macros. This included *.h file is the one added by the PowerToy at a known location in the DSL solution, and contains the actual menu commands. Firstly, we add our #include statement after the other #include statements we expect in the *.ctc file by default. All that remains to do is to write in the various macros (i.e. 'DSLEDITORPOWERTOY_CMDPLACEMENT' and 'DSLEDITORPOWERTOY_BUTTONS') into the *.ctc file at the appropriate locations. We can do this by appending the macro text before the end of the relevant command table sections (such as: 'CMDPLACEMENT_END' and 'BUTTONS_END'). The resulting *.ctc file would look something like this:

#include "stdidcmd.h"
#include "vsshlids.h"
#include "msobtnid.h"
#include "virtkeys.h"
#include "DSLToolsCmdID.h"
#include "..\GeneratedCode\GeneratedCmd.h"
#include "..\GeneratedCode\DslEditors\CustomCmd.h"

CMDS_SECTION guidPkg
    MENUS_BEGIN
        GENERATED_MENUS
    MENUS_END
    NEWGROUPS_BEGIN
        GENERATED_GROUPS
    NEWGROUPS_END

    BUTTONS_BEGIN
        GENERATED_BUTTONS
        DSLEDITORPOWERTOY_BUTTONS
    BUTTONS_END
    BITMAPS_BEGIN
    BITMAPS_END
CMDS_END
CMDPLACEMENT_SECTION
    GENERATED_CMDPLACEMENT
    DSLEDITORPOWERTOY_CMDPLACEMENT
CMDPLACEMENT_END

The advantage of this approach with this kind of structured text file is that when it comes time to remove the capability, we only have to identify the lines on which the #include and macros exist, and simply delete those lines. (Again assuming the DSL developer maintains the clean format of this file).

Installing Artefacts

Now that we have our code templatized in text templates, and we know which code snippets need to be added to which existing files, the next challenge is to figure out where to add the text templates, and what else needs to be done to integrate the tool-window editor into the DSL solution.

At this point, from our RI, we have established that we need to do the following to integrate our tool-window editor:

Add a number of text templates to the DSL solution (which generate partial class source files, resource files, header files etc.)
Add static supporting files to the DSL solution (i.e. *.bmp files)
Modify existing files in DSL solution (i.e. *.ctc files)
Add assembly references to supporting common assemblies.

Modularisation

As with any add-in type functionality it is important to modularise your additional functionality. To facilitate add-in type add/remove functionality and clean separation of PowerToy artefacts from the rest of the DSL solution, it is necessary to group all PowerToy created source artefacts in the same folder.

DSL solutions, by default, separate their generated code from the source artefacts that the DSL developer works with at design time (i.e. the Domain Model designer *.dsl). They do this by placing all text template files within a folder called 'GeneratedCode' (by default). The DSL developer is not expected to modify these generation files, instead work with the Domain Model Designer (*.dsl) file to make any customizations to the DSL. It is the domain model design that acts as input to the generation files.

One of the objectives and requirements of the PowerToy is to provide a simple means to configure its capability and require little to no maintenance of the generated code from the DSL developer. For this reason, the PowerToy employs a similar strategy to the DSL toolkit for adding all source artefacts to a single folder. By default, this folder is located under the 'GeneratedCode' folder in the DSL solution.

In the first release of the PowerToy, the only difference between the semantics of the use of the PowerToy subfolder and the 'GeneratedCode' folder from the DSL solution is that there are source artefacts (such as *.resx files) that the DSL developer can customize the workings if the PowerToy's capability. This folder is also the intended location for additional source files from customization of the capability.

[In future releases of the PowerToy, there will be other designer artefacts with which the developer works with to make a cleaner separation between generated code files and design time artefacts.]

Supporting Install/Uninstall

One of the primary requirements of any add-in type mechanism is that it can be added or removed from the solution at will - it's pluggable. The objective should be - to have minimal impact on the existing DSL solution if a DSL developer added the capability and then removed it sometime later. In order to achieve this, for every installation type action listed above, we need to be able to reverse it in the 'removal' mode (and reset it in the 'reinstall' mode).

The basic philosophy here is that the PowerToy would undo any modifications it made to the DSL solution, whilst preserving any potential customizations the developer had made to its capability (within reason of course).

The tricky part is that, DSL solutions are innately flexible in their structure and naming, and there are little to no limitations on how a DSL developer can modify the projects of a DSL solution to suit their needs. The moving parts of DSL solutions make almost no assumptions about how the projects are named, and physical structure etc. It is feasible, for example, that a DSL developer could decide to combine all artefacts from both DSL projects into one project and completely restructure it, renaming its folders and files etc. This might be at the extreme end of the customization spectrum, but it is quite likely that every DSL developer will have their own strategies for customizing their DSL solution and how to structure customizations in their DSL projects. Certainly, for any customization that requires additional code artefacts, a developer may choose to restructure or enhance the structure of the projects. Also, its is feasible, especially in the software factory context, to have more than one DSL in a single VS solution.

This makes it reasonably challenging to perform uninstall type operations since the PowerToy can make very few assumptions about how the DSL developer could move files around and rename them to suit their needs prior to install, or between install and uninstall of the PowerToy's capability.

Grouping and modularisation of the installed artefacts (i.e at least placing all files in same folder) alleviates some of these issues, but does not eradicate them, or deal with all possibilities. Since there is no built in infrastructure in DSL solutions to aid us here, the PowerToy can only go so far to deal with this solution flexibility. (Perhaps a DSL PowerToy framework would be useful for this!).

Looking at the list of actions defined above, we simply need to be able to:

Remove/replace the files (and subfolder) we added.
Delete/replace the modifications to any modified files.
Remove/replace the assembly references.

However, most of those actions require us to reliably know (a) where the added files reside now, (b) where the modified files reside now. (assembly references are only located on project, so they are always predictable to modify). Furthermore, there are in fact, by default, 2 projects in a DSL solution, and some actions only apply to one or other of the DSL projects. (that also assumes the DSL developer didn't split or merge parts of these projects).

To resolve these issues with the minimum amount of assumptions being made (and without building a complex infrastructure around it all), the PowerToy employs the following strategy, to accommodate non default configurations.

On install:
- Prompt user for appropriate DSL solution project
- Create a subfolder in a user defined location on the DSL project
- Add all files to this subfolder
- Modify existing files, prompting user for location of specific file in DSL project
- Add assembly references to the DSL project
On reinstall:
- Same as on install, overwriting any existing files/modifications
On uninstall:
- Prompt user for added subfolder, in DSL project
- Delete all added files, leaving any additional files
- Delete subfolder only if empty of files
- Delete modifications to modified files, prompting user for location of modified file in DSL project
- Remove added assembly references from DSL project

The last remaining issue is how to determine which is the right DSL project (of the 2 default ones) to operate on. In this release of the PowerToy, we are only interested in the 'DslPackage' project (which contains the VSPackage class), and we have no modifications at this stage for the 'Dsl' designer project. In subsequent releases of the PowerToy we may need to perform different actions on different DSL projects.

The DSL toolkit does not provide any way to indicate which of the default 2 projects is the package project (for example using metadata). There are few, if any, reliable ways of determining the package project from the designer project, especially when you consider that the projects and their structure can be modified in any way the DSL developer chooses, and further, any given VS solution can contain any number of DSL's.

One could argue several techniques of determining the package project, you could reflect over the assembly of the project and determine if it contains a package class, but even this requires that the project has been built successfully at the time of deploying the PowerToy. Also, this does not deal with cases of multiple DSL's in a VS solution either.

Clearly, the only reliable means available to us in reliably selecting the right project that contains the package class, for the DSL of choice, is to ask the PowerToy user to select it from the list of projects in the VS solution. Once, we know this project, we can also ask the user to define the subfolder in which contain PowerToy files, and prompt them to locate the specific files to be modified by the PowerToy. Of course, we can assume certain defaults for these values to aid usability, based on what comes out of the box from the DSL toolkit, but we need to provide for the potential flexibility.

Now, when the PowerToy user comes to remove the PowerToy, it would be helpful to remember any settings we used on the install to act as the default values on the uninstall, particularly folder paths and specific modified files. We can also validate these folders and files exist currently.

It also comes in handy to mark the modified projects with specific PowerToy metadata, so we know which projects were modified by the PowerToy. Marking the projects and saving installation metadata is done by saving values to the projects global property bag on install.

As it turns out, this metadata 'marking' helps us drive further actions of the PowerToy and indicates the right context for the Add/Remove/Re-install actions.

Applying Guidance Automation

At this stage, we have resolved most of the challenges of installing and uninstalling the PowerToy capabilities. The final part to this PowerToy development is applying guidance automation and making that automation context aware so the user has the right actions available at the right times for the right purposes. We could of course get the DSL developer to perform the necessary actions to install the PowerToy's capability, but automation makes that trivial and reliable.

The primary areas to apply automation for such PowerToys are typically:

Installing the capability
Configuring the capability
Re-configuring the capability
Uninstalling the capability

In this simple release of the PowerToy the configuration of the capability was done during installation of the capability. The capability can be re-installed, at which time it will be re-configured. Later release of this PowerToy might vary with respect to when configuration/re-configuration takes place, since the configuration of later releases is expected to be vastly more complex.

However, each of these actions is a good candidate for full or partial automation.

Adding Recipes

Recipes are ideally suited to applying automation to the areas identified above. A recipe is a simple strategy for gathering information and executing a sequence of actions that provide the automation part. Recipes can be made context aware by only being active under certain conditions when the state and context is correct. This state and context can be derived from any source within the development environment, but typically it is based upon the users current actions. In this case, when the user is designing a DSL solution.

For this PowerToy, from the previous section we can make the recipes available at anytime during DSL solution development. The recipes can then prompt the DSL developer for the various settings they require to execute.

As mentioned before in the previous section we used metadata on the DSL projects where we installed our capability, and we also used metadata to save values used in the installation, to be used as default values at uninstallation time. This metadata provides us the context we need to determine where and when the various recipes are valid, and what recipes could be invoked at any particular time. The recipes themselves also manage the adding/removing of the metadata itself.

The installation recipe and uninstallation recipe for this PowerToy release are programmed with the sequence of actions already discussed in previous sections. You can see the parameters (arguments) that are collected from the user, the wizard that is used to gather the data from the user, and the sequence of actions that are performed.

Generating Text Templates

The last point that is noteworthy in this discussion is to do with the installation of the text templates that generate the code for the PowerToys capability.

Part of the configuration gathered by the recipes on installation (and configuration) needs to be written into the actual content of the text templates the PowerToy adds to the DSL solution. This is to customize the templates to the current DSL solution. Currently, this would include things like captions for resource files, and other parameters gathered from the user and recipe.

In order to do this in a recipe, the text templates themselves are rendered using the text templates themselves!, and the configuration gathered from the user and environment is used as inputs to the text template rendering.

Careful formatting of the text templates is required to support this 2-stage rendering, but this is an elegant pattern for customizing text templates for application to a specific DSL solution for generating source for a PowerToy capability.

You can see an example of one of these 2-stage rendered text templates from this release of this PowerToy

Facilitating Customization

So far we have seen how to build a PowerToy that installs a configured capability, but what about customization of that capability?

Since we are installing text templates that get transformed into source code during the compile time of the DSL, we are limited in how to customize this capability? Surely, that would involve editing source code in the text templates themselves wouldn't it? - Well, yes, yes it would. However in most cases, all you want to do is customize the code by providing your own methods for a class, to enhance an existing method or change the class's behaviour. The typical way you would want to do this is using partial classes, but that can't be done here since you can't rip and replace methods using the partial class machanism. You might then be able to use inheritance and subclassing (in some cases), but it's not your subclassed type that is going to be consumed by the existing executing code. To make that work, you would need to make changes to the existing text template code to explicitly use your subclassed type. None of this is elegant, and hampers seperation of the code the the capability and the code for customizing or configuring that capability.

The DSL Toolkit developers figured out a neat way around all this called the 'double-derived pattern' that gets around most all these issues very, very elegantly. Basically, what this amounts to is: instead of just defining a class and its' methods in a text template, you instead create an abstract base class containing all the properties/methods (which are now virtual) and you create a concrete derived class of the base class in the text template.

Now, as the DSL developer customizing the capability, you can create a partial class definition of the concrete class and modify its behaviour. It will be the concrete derived class that gets consumed by existing executing code, and this should require no changes to any text templates.

Together, text templates utilizing the double-derived pattern provide a strong foundation for facilitating customization of a PowerToy capability, and seperation of the PowerToy capability code from the DSL specific customization code of that capability.

Summary

In this post we discussed in detail the issues with creating a PowerToy for DSL solutions. In particular, we have described some of the general patterns and strategies that could be used to create DSL PowerToys. In this respect, these PowerToys to being modular plug-ins for DSL's, that support add/remove like capabilities and use automated guidance to provide easy operation.

To summarise we discussed the following patterns and strategies:

Building Reusable Assets - the process for developing guidance automation assets from existing solutions. (RI->CV analysis->Asset creation) Described in more detail here.
Creating contextual source artefacts - using text templates, and leveraging the incumbent DSL build process.
Adding modular capabilities - separating and modularising capabilities, packaging them for easy maintenance, configuration, and removal.
Applying Guidance Automation - automating the installation, configuration and uninstallation of the capability.
Facilitating Customization - designing text templates that employ the double-derived pattern for intentional customization.

I believe that these patterns and strategies are re-usable in the development of other PowerToys, which may lead to a future common framework, practices and guidance around DSL PowerToy development.

Resources

The DSL Editor PowerToy, used as an example for this article, is an open source community project hosted on CodePlex, you are free to download the releases of this PowerToy to examine the details of how it was built, and participate in the evolution of this PowerToy.

DSL Editor PowerToy - Just Released

jezzsa — Thu, 15 Mar 2007 02:24:00 GMT

You might be happy to know we have just released the first version of the ‘DSL Editor PowerToy’ on our codeplex project!

This initial release is slated as:

“A means to add a DSL specific tool-window, containing a simple editor, to an existing DSL”

You can download this release from the site.

It's not much really in terms of functionality at present, and the editor is really basic, but what we tried to accomplish in this release was setting up the infrastructure of what is to come.

We have a pretty decent plan of the features we want to work towards in later releases, and we have carefully designed the releases so that you can have something useful to take your own way at each step. The ultimate goal is demonstrated on the home page of the site, and this is what we are working our way towards providing everyone with a means to have multiple editors each exposing a customizable view of their DSL.

Some basic links for this release:

The home page of the release
How to install, and configure the PowerToy
How to customize the tool-window
Detailed design discussion

If you have any feedback about this release or the PowerToy in general, we would like to hear it.

DSL User Experience PowerToy!

jezzsa — Thu, 01 Mar 2007 20:53:00 GMT

If you are building DSL languages today, you may have experienced at least one of the following issues when figuring out how to represent your languages' domain model on a diagram surface:

Representing most of the domain classes as shapes makes the diagram really complex and cluttered, and therefore arduous to navigate around for the user
Box and lines are not always ideal for representing parent/child relationships
You can use compartment shapes to describe one level of hierarchy on the diagram, but no more
Representing some of the domain classes as connected shapes really does not provide much additional useful information to the user
Configuring multiple similar shapes, with unique values by individually clicking each one and then changing their domain properties via the 'Properties Window', is pretty mouse intensive and arduous for the user
It is hard to get a view of your language that focuses on one particular aspect of the domain

If any of the above are issues for your language, you might consider checking out the 'DSL Tree Grid Editor PowerToy' that we are building to resolve these and many other usability issues you may face designing your DSL today.

The PowerToy enhances your language by providing tree-grid-like editors (displayed in a tool-window exclusively for your language) that allows you to expose hierarchical domain relationships, and allow the user to navigate and configure the domain classes in the hierarchy with the keyboard.

The related domain classes are displayed in a grid which makes it easy - at a glance - to compare their important properties side-by-side, whilst configuring and creating new domain classes real easily within the editor.

The PowerToy allows you to add multiple of these editors; one for each 'aspect' that you wish to expose of your language. In effect, each editor acts as another view into your domain model.

As an example, see the DSL with editor below.

Multiple or Single Architectural Models/Views?

jezzsa — Wed, 22 Nov 2006 14:30:00 GMT

This was to be a FAQ, but due to the fact that the practices around this are unproven yet, there is no definitive answer to this question, rather a discussion of approaches, issues, guidelines and raising general awareness.

At the moment, there is quite a buzz about the correct usage of DSL's and modelling domains appropriately in the context of software factories.(within Microsoft at least, I would expect the rest of the DSM community have a more mature view on this).

We are starting to see an emergence of software factories of different varieties. Much of this innovation is growing from an absence of a formal framework for building factories and mechanisms to stitch them together, with navigation etc. (which may one day be available to us all).

A lot of the innovation is being put into how to describe a whole domain with multiple models, and when and how to relate those DSL's together. Services and practices are starting to emerge such as the integration service introduced by Mauro and Victor in their article on Integration Scenarios which allows you to reference elements from one model to another. Other factories, like the Ordina Factory, are using other techniques to achieve the same ends.

There appear to be two extreme views on how to model any part of a domain (sub-domain). On the one hand, some folks are using very few (or single) models to cover the sub domains. Others are decomposing the sub-domain into very many models.

Now, we need to be accurate here. There is a difference between the actual model and view (of the model). Ideally, you could have multiple models with multiple views, each view considers a whole model or part of a model. In some technologies a view can span multiple models also. Its the model that describes the logical architecture, and the view that provides the interface of that to the user. For the user, its the view that's important.

[It should be noted at this stage, that multiple views for one model is currently not supported in the Microsoft DSL toolkit. Views to span multiple models is also not supported currently. Only one view per model is presently supported.]

In the context of architectural guidance in a factory, imagine we are modelling a service, for example. Architecturally speaking, this is composed of several abstraction layers, and few deployment tiers. Lets consider the following standard .NET architecture for service orientation from patterns & practices. Since this is quite a popular architecture to create factories around at present.

Considering the abstraction layers in the logical architecture, the objective is to decompose the architecture into layers, with differing concerns, all in the scope of the whole service. i.e. Service Interface, Business Layer, Resource Access layers and their component layers.

Now undoubtedly each layer of the logical architecture is concerned with different aspects of the overall solution, and further, each layer has more than one concern to address, as well as the variability of that layer and its components.

The question is how best to apply DSL's to model these concerns and variability?

Jack addresses the same question is his article on bare naked languages.

There are a number of patterns that address these kinds of architecture, and its typically these patterns that determine the answer to this question.

However, I assert that basing your models and views upon these patterns alone, may not always be the best approach to exposing your DSL's to the users.

Multiple Models

To be a purest about separation of concerns, when analysing a problem domain, you would take the approach to define a single model for each layer which addressing a small number of concerns and the variability of that layer and its components. Or perhaps even, a model per concern!

Ideally, you would present several different views (if possible) of the model perhaps each one exposing each one of the concerns, or maybe just one view for all the concerns (which is the case today with using the DSL toolkit).

Then you would develop an appropriate mechanism to reference components from one model to another.

For example, a separate model for each of the layers:

A model for the Service Contract, and Service Adapter,
A model for the Business Logic implementation
Perhaps a separate model for the Business Entities
A model for the Data Access Logic
A model for the Service Agents

That's 6 models, but some could be combined. So in reality 4-6 models to cover the whole architecture.

All models address the concerns and variability of their layer.

Single Model

A more natural (perhaps less purest) approach, would be to define a single model for all layers which addresses all the concerns and variability.

This does not necessarily infer that appropriate emphasis has not been given to separation of concerns. It just means they are defined in the same model instead of separate models.

Ideally, you would present several different views (if possible) of the model perhaps each one exposing each one of the concerns or aspect of variability, or maybe just one view for all the concerns and variability (which is the case today with using the DSL toolkit).

For example, a single model for all layers. (example here from the EFx factory).

Considerations

Considering these two approaches (in this context).

These approaches aim to achieve the following:

Multiple Models

Maximum physical separation of concerns.
Maximum physical de-coupling of the layers.
Maximum flexibility in reuse of one model in another architecture.
Facilitate better source control in team environments

As developers can work on different parts of the architecture in isolation from others.

Single Model

Maximum holistic description of the architecture (big picture view).
Maximum usability.

The challenges with these approaches (in this context):

Multiple Models

No holistic view of the architecture. The inter-related pieces are presented in different views.
The interface to reference dependencies in other views is usually poorly visualised/represented.
Information to perform common user tasks maybe spread throughout several files.

Single Model

Complexity of diagram can be high, if the abstraction of the model is low.
Team development on same model can be challenging, as the view/model could be used by different roles addressing different aspects.

[If you have any other pros and cons to add I be glad to add them.]

Learnings?

So what can we learn from this? I believe neither of these extreme approaches is ideal. The answer lies somewhere in-between - a compromise of both approaches.

The emphasis of the views that you provide for your models, needs to be high usability. A software factory is a productivity tool for users that should not have to have the level of knowledge and experience that those building them have. So even though your model(s) employ the best architectural practices in separation of concerns, and pattern application etc, these should not necessarily be forced upon the users. Instead, these patterns and architectural practices should be abstracted in the views, in a way, that makes it easy for the users to get the job done in describing the solution domain. (Its the classic: Usability <-> Flexibility trade-off struggle)

Perhaps building the models and the views are two different roles of defining a model for your domain?

Here are some things to consider to make the modelling solutions (in a factory context) more usable:

Use fewer models when the abstraction is high and variability is low. Use many more loosely coupled models when the abstraction is low and variability is high.
For single models, use multiple instances of the model file to partition the solution into bite-sized chunks, and use cross-model references where applicable.

This will assist in team development.

For multiple models, provide holistic views of the other views that bring them together to form a composite 'zoomed out' view of how the models relate, so the user can get a feel of the whole picture rather than one aspect of it.

This will enable the users to stay focused on the objectives of the solution.

For any approach, analyse which layers are more tightly bound to others, and require less flexibility in reality - from the users perspective.

For example, in the above example, it is unlikely that the 'Resource Access Layer' is going to be used separately from the 'Business Layer', but it is highly coupled to it architecturally speaking. So it would make sense from a usability perspective to combine these layers into a single view.

For any approach, use views that expose particular concerns or aspects of the architecture.

A good use of these views is by role, so for example, a security expert can view the cross-cutting concerns of authorization across the whole architecture, without having to understand all other aspects or abstractions of the architecture.

[Again, if you have any other considerations to add, I be glad to add them here.]

Factory Schemas

jezzsa — Sat, 27 May 2006 22:49:00 GMT

I recently decided to do a series of articles on factory schemas.

It has become obvious over the last while that the factory schema has enormous power both as a means to describe a factory and its content, a means to describe the factory itself and a means to offer huge benefits to the factory user, and also a means to solve larger problems we have in designing software.

Once you start to realise how these schemas are used, you really start to see the potential of factories in revolutionising the development of software in our industry today.

I wanted to write several articles to help you understand what a schema is, how to create one as a factory builder, and how it may be viewed by the users of your factory. I then wanted to explore an idea I had some time ago, that now becomes possible to realise with a factory schema in place.

My first article ‘What is a Factory Schema’ will address a common misunderstanding of the factory schema and aims to help you understand what it is in concrete terms.

The second article in the series ‘Defining a Factory Schema’ tackles how to define a schema for your factory by way of an example.

In the third article ‘A Use for the Factory Schema’ I describe a factory users view on the schema, and how it provides them with guidance through your factory.

In the final article ‘File | New | Blank Problem (Part II)’, (a continuation of a previous theme) I delve into how the factory schema will aid us, (some time down the road), in addressing how to locate and choose the right factory for the problem we are facing. This will be intended to show what could be possible with factories in the future once they are in common use.

File | New | 'Blank Problem' (Part II)

jezzsa — Sat, 27 May 2006 22:30:00 GMT

So, we have now had a pretty thorough look at what a factory schema is in a number of previous articles, and now how it can be used from both the factory builder’s perspective and also the factory user’s perspective.

In this article, I wanted to dig a little deeper in usage of the schema and demonstrate how it could provide the vital link between the defining of a problem and the finding of the right tool (factory) to implement the solution.

In the previous article ‘File | New | Blank Problem’ we explored a vision of the future of software development where we could start our development project by exploring the problem domain first, separate from the trappings of technology constraints (purely business requirement capturing). We use a set of relevant and related graphical design tools that interpret the expressions you make on design surfaces that interactively evolve into a definition of the problem domain you wish to address. Once we have that problem definition, after adequate refinement of these expressions, it was then only a matter of having the tools search for and suggest the right tools to help design the solution domain.

You may know by now that software factories are going a long way to address the concern of defining the solution domain in higher abstract terms and providing domain specific solutions. But we still don’t have either a mechanism to capture problem expressions and interpret them into a problem definition. That is currently up to the individual, and whether they have tools, the right tools to capture this rich information in the early stages of design. We also don’t have a way to match a problem definition with a toolset (factory) that could provide the solution implementation – or do we?

The Vital Link

One aspect of the factory schema that has not really been explored yet is the factory's ability to expose its description externally to prospective factory consumers. The factory schema is so far, only geared towards describing the internal workings of the factory. You still have to know about the workings of a factory and what it produces in order to use it effectively.

For example, it would be possible for a factory to define a set of pre-conditions that must be satisfied in order to start to use the factory, and a set of macroscopic Work Products it produces.

As an example, the 'EFx Factory' requires that you at least provide a ‘napkin description’ (in any media form) of the applications and services you which to construct in the factory, a product name, company name and the names of the applications or services you wish to construct. These are the pre-conditions of using this factory.

Without this basic information, the user will find it hard to get started using the factory. All factories will require some pre-conditions of use, (unless of course, the factory is not very specific, or maybe it can make these assumptions to begin with).

In order to encourage adoption of available factories, we are going to have to find a way to classify and advertise them in a meaningful way using a standard set of agreed upon semantics (taxonomy). Without this description, development processes will be slow on the uptake of factories for the simple issue of perceived unavailability of them. If people don’t know about them, how can they possibly use them? It’s a very common problem in software development today which hinders reuse, leading to all kinds of fruitless efforts and resource wastage. Ironically enough, factories were formulated to address part of this problem in the first place, by removing the requirement to own the knowledge and experience to implement solution domains.

So imagine we had a description of the factory which could be exposed to those interested in searching for and using this factory. This meta-description would contain a description of the factory’s solution domain, its’ capabilities perhaps even how it works, the technologies, versions and platforms it uses etc, certainly its main work products. Some of this information would of course have to be meaningful to the machines reading this description, as well as the human consuming it.

So, for now, side-stepping the challenging issues around how to classify this information and what semantics to use, this capability would become a powerful mechanism to track down and select a factory that suited a specific problem definition.

Connecting the Dots

Once we had this description, and an effective standard way to expose it to consumers, we only need to be able to pattern match this meta-description to someone searching for a factory to suit the right problem domain.

In the previous article on ‘Problem’ files, we saw that when we have tools good enough to decipher human expressions from a number of design surfaces into a meaningful problem definition, these problem definitions could start to yield a set of conditions to match those of a factory’s set of pre-conditions.

If we could do this, then we could suggest to the user a set of relevant factories to help them solve the problem. The user could then pick and choose, perhaps even try out, or even compose a larger more specific factory of their own with smaller factory pieces each addressing a part of the problem definition.

Today, we rely too much on the individual knowledge of the people designing the software to locate the correct tool, component, solution for a problem domain. Our stamina for finding and selecting the right tool for the job is very low, and often we resort instead to authoring our own tools convincing ourselves that no one could possibly have provided anything similar or useful enough for our problem. We attempt to use average generic tools to describe the problem domain, and very quickly converge to the ‘solution-first approach’.

Software factories can help us to a certain extent in abstracting the solution, but so far, they are not chartered with capturing problem expressions and definitions.

We need to start thinking about the separation of the two domains (problem and solution) if we are to succeed in this effort. Today, we are a fair way from ‘smart design surfaces’ that can interpret drawing and textual expressions into a problem definition sufficient to suggest a relevant solution. But, today’s technology is rapidly advancing. We already have inking and natural language technologies to recognise human expressions. What we now need is way to interpret those expressions into a definition of a standard problem or variant of that problem. Factories provide the obvious means of mapping from the problem definition to a solution definition, and the factory schema is the key to enable this link.

The 'Visual Studio Stare'

jezzsa — Sun, 19 Feb 2006 21:46:00 GMT

If you were following the previous post on File | New | 'Blank Problem' you will see we are a fair way from a development environment that provides us a starting point for defining the requirements for a solution, let alone a single environment for defining the problem. Until the day we can open Visual Studio, and create a 'New Blank Problem' file, where we can use a bunch of integrated tools to capture our problem and requirements as we think about them, the best thing we can do today is start to automate the solution implementation.

Of course initially, the key to getting started along the automation path is to either know about it or discover it, and hopefully avoid that phenomenon that most developers are familiar with - 'The Visual Studio Stare'.
You suffer the 'Visual Studio Stare' when you (the solution architect/developer) know you have something to build, you know the (only) tool you will ultimately build it with, but you just don't know how and where to get started with it....

...you suffer the familiar sinking feeling of the 'Visual Studio Stare'.

"I just know that I need to do something, just don't know how to start"

This phenomenon just illustrates the lack of tools we have today to conveniently and sufficiently express and capture a problem description. Subsequently, this forces us down the path of the 'Solution-First-Approach'.
Getting started today in Visual Studio typically begins at the 'Start Page', and more often than not begins by opening the familiar 'New Project' dialog box where the beginnings of the solution begin to be defined.