Alan Cameron Wills - Domain Specific Languages

my new blog about UML

Alan Cameron Wills — Sat, 31 Jan 2009 17:23:00 GMT

Since this blog has been all about my old job, I'm going to start a new one: UML and Agile Development in Visual Studio.

Writing about UML - my favorite pastime!

Alan Cameron Wills — Thu, 29 Jan 2009 15:15:00 GMT

In the days long ago before I joined Microsoft, I was an itinerant consultant in UML.

I'd wander the countryside with a slide deck, a projector and a cooking pot on my back, gathering small crowds under the shade of a tree to tell them the Good News: about how they could better understand and communicate about their software designs by drawing boxes, lines and stick people on their cave walls; and about how they would, if they did this properly, surely achieve their goals and feed their families. Well - they'd improve the chances a bit, anyway. I asked only a mug of tea and a biscuit in return. (And my standard fee, of course.) It was an ascetic life but a rewarding one, particularly when some youngster would show especial enthusiasm and I would introduce them, by the flickering light of my camp PC, to the arcane beauty of the Object Constraint Language and its virtues in the successful construction of tests; or the subtleties of covalent and contravalent composition among collaborations, and their uses in describing and applying design patterns.

But there were sorrows too. And the most poignant of these came whenever my followers gathered about me would ask, as they inevitably would, "OK chief, this UML stuff is cool. But what about tools?" And I would have to shake my head with a heavy sigh and reply that alas, there were apps that would help you draw the diagrams, but few that really delivered the goods in terms of what you'd want to do with the drawings once you'd drawn them. And that in consequence, they might as well stick to drawing on the cave walls. (Note: if you wish to do this, please make sure your walls are made of glass, and use dry-wipe pens.)

And one or two of them, perhaps those who had traveled beyond the bounds of their own village, would sometimes say "Yeah, but we heard you can get these tools where you can reverse engineer your code into diagrams, and then edit the diagrams and transform the diagrams back into code." And I would agree that these tools are indeed of value in showing you the complex interactions between the objects, and to help you visualize the links and dependencies between the classes. But after all, they only give you pictures of what is there in the program text. These tools have no power to abstract the higher design of your code, the grand vision that was in your mind when you wrote it (always assuming you had a grand vision, of course...); they cannot know what you meant when you wrote the code. That meaning has to come from you.

This is the real strength of UML, properly used: to help you think about and discuss and communicate the crucial aspects of your design in just a handful of skillfully-chosen lines. A bit like a cartoonist drawing the key features of a politician; and not like a camera slavishly recording pixels. Good UML tools don't just do the reverse engineering - they also help you express your big ideas, help you look at them from different angles, and help you turn them into plans, designs, and tests.

So I'm delighted, now that I'm working for a toolsmith, to be working on modeling tools. And yes, some of the tools are about reverse engineering from code - and they do in fact help you abstract the overall structure. But they are also about using UML to understand your users' needs better, and to help you meet those needs successfully.

My role is to write the user guides - not just at the mechanical level of what button to push, but also about how to make good use of UML in your project. So I'm going to start a new blog about UML - and, I hope, hear from you how you use UML - and what you want from our tools.

Move to MSF

Alan Cameron Wills — Thu, 07 Sep 2006 10:08:00 GMT

I've joined MSF - the Microsoft Solutions Framework. MSF is about providing process "guidance".

Software development processes are about who does what and in what order - role X writes a spec or a story, role Y writes the software, role Z tests it, etc. There are heaps of variations: different companies have widely differing sets of roles and responsibilities. In some outfits the workflow is simple: there's a list of tasks and people pick them off and do them; in some it's complex, involving multiple stages of work and signoffs.

I put the quotes round process "guidance" because it used to mean a load of narrative text describing the responsibilities of each role - so if you're a tester, you can look up what to do in each circumstance. But these days - particularly with the software factories movement - we're more interested in tooling up the guidance: "guidance" means everything in the user's environment (particularly Visual Studio) that supports the locally-adopted process. Work-tracking tools, tools for doing particular tasks like UI design, languages specific to the company's business, constraints that help you remember to run your tests before you check in, queries that enable you to find how the project is progressing - everything that helps the team work together effectively; and everything that helps them work in the way that's best tuned to their business.

Visual Studio already supports a lot of this customized process-oriented stuff. GAT provides a way to extend VS so that a developer working on a particular type of job - designing UI or databases or some specific aspect of your project - can invoke tools specific to that type of work. The DSL Tools provide ways to generate projectfuls of code from notations specific to your business. On a more process-oriented level, VS Team System lets you track progress of work items and set up a workflow that's suited to your project's way of working.

Customizing and tuning the process to your company's business is a very important aspect. So a good chunk of my job over the coming months is going to be improving the tools for doing the customization. We're shifting the emphasis from the narrative style of guidance (though not losing it entirely) to the tooled-up style, and making it easy for people to make the tools suit them.

Models conference - place to show off your DSLs

Alan Cameron Wills — Tue, 14 Feb 2006 11:58:00 GMT

The Models Conference is in Genova, Italy this year (October 1-6).

If you have developed some interesting stuff in DSL Tools, this is the place to share it and get useful feedback. If you're wondering about the best ways of using models to drive software, this is the place to hear about and discuss others' experience and ideas. It's where you'll meet the most innovative practitioners and tool makers in the field.

If you have interesting questions you want to discuss, run a workshop. If you've got some interesting bits running on DSL Tools, show them off at a Poster Session - you'll get lots of good ideas and make interesting contacts. Or if you've advice to give, run a Tutorial.

Models grew out of the old UML conference, and is the place to learn about model-driven software and development. Members of the DSL Tools team will be there; and have chaired and contributed to the conference on many occasions in the past.

Extending the DSL Tools

Alan Cameron Wills — Tue, 20 Dec 2005 12:58:00 GMT

The DSL Tools are extensible - as well as writing a specification of your own language, you can add your own code to extend the language and its tools in various ways. One of the best ways to learn the extensions is by reading the DSL Samples. They include detailed writeups on how to customize the tools, with sections on validation, deployment, templates, customized shapes, menu commands, and other goodies.

Models/UML conference

Alan Cameron Wills — Tue, 20 Dec 2005 12:54:00 GMT

The Models/UML conference has posted its call for papers http://www.modelsconference.org/. Steve, Stuart and I take it in turns on the programme committee for this. (Stuart was Programme Chair last year.)

"The MoDELS series of conferences are devoted to the topic of model-driven engineering, covering both languages and systems used to create complex systems. These conferences are both an expansion and a re-direction of previous Unified Modeling Language (UML) conferences, and replaced that series of conference since last year."

End-to-end DSL samples available

Alan Cameron Wills — Tue, 13 Dec 2005 11:48:00 GMT

We've just published an updated set of samples for the DSL Tools. As well as updates to the previous examples, which show how to customize your DSL, there's now a complete small end-to-end showing how a DSL can be used to generate the code of a project, and work as part of a larger solution.

The scenario in the end-to-end is in a team where they often have to design Windows wizards. The layout of the wizard pages is constant, with Next and Back buttons and so on; but the sequence of pages can change. So the team decide to create a Wizard Sequence Definition language. From the language, most of the code for the wizards is generated.

The developer assigned to author the language actually has two jobs: as well as creating the wizard sequence DSL itself, she has to create the generators that turn it into code. She begins by designing a typical wizard, and then gradually turns it into a template which reads the page sequence from a DSL instance. There are multiple code files in the wizard, so there is a complete project full code all generated from one DSL file.

When she is confident she has the language and the generated wizard project working, she packages it into a standard installer (MSI), with the help of the DSL Setup project. After her colleagues run the installer, they have a new project template available in theVisual Studio "new project" menu, which creates their own copy of the wizard project. They can now add a wizard to any application, and quickly generate the wizard code from a drawing of the page sequence.

The sample also makes good use of the new validation framework. Not all wizard sequences are valid - for example, you shouldn't have pages that can't be reached from the start page. The language author only has to create the code for the constraints themselves, and the validation framework applies them at the right times and manages the error list.

partial classes

Alan Cameron Wills — Thu, 03 Nov 2005 05:11:00 GMT

We use partial classes a lot in the DSL Tools. We need them to help compose code that's generated from different places, and generated with handwritten code.

If you're not familiar with it, a partial class definition is just a set of methods, properties etc that gets combined with another partial definition at link time, to form a complete class. The parts can be in different source files, which makes it easier to manage them separately: different generators can output to different files. So in the DSL Tools Use Case template, there are two generated files Designer.dsldm.cs and Shapes.cs that each contain aspects of every shape you have defined in your language. You can add a separate file with your own definitions.

There is a particularly useful pattern for providing a hook for optional manually defined features. Each generated class - let's take Container.cs in the Use Case template - has a framework superclass in which default values are defined as overridable properties. For example, HasBackgroundGradient is defined in the framework superclass OutlinedShape. This enables you to override HasBackGroundGradient if you wish in a partial Container class, but to leave it to default to the OutlinedShape value otherwise.

Generated:

public partial class ContainerShape : Microsoft.VisualStudio.Modeling.Utilities.OutlinedShape {...

Framework:

public class OutlinedShape { protected virtual bool HasBackgroundGradient { get { return true; } } ...

Optional manual override:

public class OutlinedShape { protected override bool HasBackgroundGradient { get { return false; } } ...

It isn't as sophisticated or flexible as delegation or aspect weaving, but does the trick.

Unfortunately, it only works for program code! If we're generating tables, configuration files, or reports, we need a way of merging them together. One way is to get the generator to read the file and only replace particular bits, corresponding to one view. Another, cleaner, way is to have each source generating to a separate file, and then weave them together in a final step.

Constraints and Restrictions in MS DSL Tools

Alan Cameron Wills — Mon, 31 Oct 2005 03:54:00 GMT

I do get irritated by over-officious tools. "You can't use that name, you've used it already over there." Well, I know, but I'm going to change that in a moment, and it'll be alright in the end, really.

In the DSL Tools Customization Samples & Guide, there is a section showing you how to prevent the user from making undesirable connections, such as loops in inheritance graphs. As the user moves the mouse to complete the last link in a loop, a tooltip pops up saying "that connection would create a loop" and the arrow changes to a not-allowed sign. This is quite effective, but of course you might prefer to let your user make the loop, and then cut it somewhere else later.

Also, the computation to find the loop - or whatever other restriction - has to be fast, to respond to the user's mouse movements in real time. In some cases, that might prohibit this method.

The solution is to use the new Validation framework, which is coming in the next version of DSL Tools. It allows you to specify constraints on any class (or superclass, or relation, or the whole model). These are applied to all instances of the type when the model is opened or saved, or on the user's explicit command. Error messages appear in the VS task list, and clicking on an error selects the offending objects. You write just the C# code for the constraint itself, and the framework applies it as required.

This means you have a choice of when to apply a constraint - at the moment it is about to be contravened, or at validation time.

Rules in the DSL Tools

Alan Cameron Wills — Thu, 27 Oct 2005 02:45:00 GMT

In the DSL Tools Customization Samples & Guide there's some examples that use MDF rules to spot a change and react to it. (1) In the class diagram sample, attributes are displayed as a string like "aName : SomeType", and in the properties grid, the name and type are in separate properties. All three forms are accessible to code, for example in any generator template. A rule fires when any of the properties changes, and propagates the change to the others. (2) In the activity diagram sample, rules are used to snap the Sync Bar shape always to be a constant width or height, no matter how the user pulls the corners. (3) There is a (rather primitive!) container shape in the activities and use-case samples, vaguely reminiscent of a swimlane and a system boundary respectively. The shape manages the trick of always remaining behind other shapes; and when it moves, so do any shapes within its boundaries.

In all these cases, you'll notice that if you undo a change, the whole thing undoes in one go: you don't, for example, have to first undo the readjustment to the shape, and then another undo for the original change. This is because the rule framework is part of the MDF transaction system. (MDF is the framework underlying the DSL Tools and other graphical designers in Visual Studio.) All the properties of an object - size and color of a shape, the value properties you've specified in your language, etc - are accessed through a framework that knows about the rules, and schedules firings as appropriate. Changes are always made within a transaction, and when the transaction is committed, the rules fire. All the firings are counted as being part of the transaction, which is the unit of undo/redo.

As you can see from the samples, rules are attached to specific types (such as SyncBarShape), using a class attribute. Each rule is a class inheriting from one of a few abstract parents. The most useful of these are ChangeRule, which monitors property changes, AddRule, RemovingRule (about to delete) and RemovedRule (after deletion). The main thing to be careful of is infinite reiterations: if a rule makes a change, it the appropriate rule will be scheduled on the firing list. However, ChangeRules only fire if there is an actual change of value - a set operation by itself doesn't fire. So in a rule that re-adjusts the same property that caused the firing, you just have to make sure things always converge rapidly on a stable value that gets adjusted to itself.

(If you're fascinated by scrolling through that long list you get after typing "override" in one of your language classes, you'll notice some methods called "On...Changed". You can override these to get an effect like the rules, but it doesn't work so well. The On... method is called after the end of the transaction, so you have to open another transaction to make your adjustments. Then of course, the user gets a horrible two-stage undo (which might sometimes be what you want, I suppose). And the biggest pitfall is that you mustn't make changes if the original change is part of an undo or redo - so if you decide you really must use this method, bracket the code with "if (!Store.InUndoRedoOrRollback)..." But it's really not recommended, and may not work in future versions.)

Code is Model?

Alan Cameron Wills — Sat, 22 Oct 2005 04:41:00 GMT

Interesting post and comments on Harry Pierson's blog:

http://devhawk.net/2005/10/05/Code+Is+Model.aspx

Customizing the MS Domain Specific Language Tools

Alan Cameron Wills — Sat, 22 Oct 2005 01:58:00 GMT

One of the features of the DSL Tools is extensibility. With the language definition files, you can define a wide range of thing-and-link style graphical notations. The current version of the Tools comes with a set of templates for creating class, use case and activities diagrams; but the idea is that you adapt the language definitions to suit your own purposes - for example to add new sorts of classes, or cut out some sorts of association.

That kind of change can be made in the language definition files. But by plugging in your own custom code, you can extend the features of your language to obtain even more interesting effects.

For example, in the Activity Diagrams template in the latest version of the Tools, there's a shape called Synchronisation Bar. It comes onto the diagram as a long thin black rectangle; but of course, the user can pull the corners so that it doesn't look like a Sync Bar any more. But with a bit of custom code, you can constrain the height to be constant - so that the user can only make it longer or shorter. (Actually, a neat trick is to snap it either to a thin horizontal bar or a thin vertical one, depending on how the user tries to reshape it - so they can reorient it and change the length, but it's always a bar, one way or the other.)

Another useful extension is connectivity restrictions. With the Class Diagrams template, you can make a language with classes, associations and inheritance etc. Again, your first step would probably be to adapt the language definition to do what your team needs; but pretty soon, you'll notice that it's perfectly possible to draw diagrams in which Class A inherits from Class B inherits from Class A - round in a loop. If your application needs to exclude this, you can write code that stops the user making a loop -- popping up a warning as soon as the mouse hovers over the target of the last link.

There are also a few features you can obtain with custom code in the current version of the Tools, that will probably be controlled through the language definition files in the Release version -- for example, whether connectors are straight or rectangular, background color is graduated, and other basic stuff. Other features require fairly complex code at present, but will be simplified in the future - for example, if you want to add your own menu commands to the designer.

We've recently released a Customization Samples & How-To Guide package that explains these techniques, and include code for all these features. Follow the link, download, and enjoy!

DSLs at JAOO

Alan Cameron Wills — Fri, 30 Sep 2005 05:56:00 GMT

JAOO in Aarhus, Denmark, is getting big - almost 900 participants, I heard.

There was a Domain Specific Languages track on Wednesday - DSLs is really gathering steam! The day kicked off with an excellent plenary on Software Factories and DSLs by Microsoft architects Arvindra Sehmi and Beat Schwegler. Markus Voelter gave a very interesting talk on how to design DSLs, and Juha-Pekka Tolvanen - who has been doing it for a decade - about how to put them to use.

MetaCase (usually represented by Juha-Pekka or Steve Kelly) always emphasise how you can build whole systems from DSLs, without any (or much) extra 'glue' code. I guess this is because in general they're working with mature product lines - often in embedded systems like phones. There, the generic framework is sufficiently evolved with sufficiently well-defined bounds. In the MS work, we tend to talk more about cases where the DSL is just covering some part of it.

Frequently Asked Questions

Alan Cameron Wills — Mon, 08 Aug 2005 11:37:00 GMT

There's now a DSL Tools FAQ, which I've pulled together from the multifariouse Q&A on the DSL Tools Forum over the past few months.

Thanks to all the questioners and answerers.

Course on DSLs

Alan Cameron Wills — Tue, 02 Aug 2005 10:27:00 GMT

Interesting progress - there's an independent company offering courses on DSLs, including DSL Tools as an exemplar.

http://www.clipcode.biz/workshops/Clipcode_DSL_Workshop_Data_Sheet.pdf