Model Citizen

Data + XML + “Oslo” = “It’s All Data” (the new Data Developer Center)

2009-11-17T19:04:00Z

For those who have been frequenting http://msdn.com/oslo, you'll find as of this morning that you'll end up now on http://msdn.com/data. That's because we've just finished merging the Data and "Oslo" Developer Centers (along with the XML DevCenter, which was brought into Data in October). All the details are in my post on the Data blog.

While we've merged the DevCenters, note that we'll still be maintaining this blog, Model Citizen, for modeling-related posts.

Announcing the SQL Server Modeling (née “Oslo”) CTP for November 2009

2009-11-17T18:12:00Z

One certainty about product teams at Microsoft is that they never stop working to improve and refine their technologies. This is true even through organizational rearrangements and little things like, you know, finding a ship vehicle and landing a few real names for products!

Now as you may have read in Douglas Purdy's recent blog post (to whom we shamelessly give top billing as we're all directly reporting to him now), what has been called code name "Oslo" has become SQL Server Modeling. This should come as no surprise, because many of the formerly-"Oslo"-now-SQL-Server-Modeling technologies, such as "Quadrant", are directly tied to SQL Server. And even those that aren't so specific, such as the "M" language, have some natural synergy, as Doug explains in his follow-up post.

Of course, there are details aplenty to work out with such things, and many questions (for which there are some answers in our Q&A articles). But maybe the easiest answer is that our commitment to these technologies stays the same regardless of their collective moniker. Witness the fact that the SQL Server Modeling (née "Oslo") team has just released the November 2009 CTP of "M", "Quadrant", and what is now also called SQL Server Modeling Services (formerly the "Repository" and its included domains).

So now that we got that covered, we can once again highlight some of the many changes in this CTP, as described in the Release Notes, and give a big curtain call to our endeavoring writers in the documentation team who have published their refreshed set of topics and tutorials in the MSDN Library.

Overall Changes

Upon installing the November 2009 CTP, you'll see that the "Oslo" name no longer appears in the setup program or in the handful of Start menu items that it creates for you. To be perfectly honest, though, that's all we had time to change when the SQL Server Modeling name was handed down! In other words, you'll still see Oslo show up in your Program Files folder, and you'll still end up with a Repository database (free of charge!). In short, while the name has changed, the CTP is structurally the same as the previous May 09 CTP.

It's also important to note that this CTP requires Visual Studio 2010/.NET Framework 4 Beta 2. If you haven't installed this package yet, now is a great time.

Changes in SQL Server Modeling Services

As Modeling Services got its turn this round to be first in the Release Notes, we can start there ourselves. For one, the UML domain (System.Uml2) model has changed significantly such that code written against the old model no longer works. It's also now possible to export UML data from models in the repository database for interop with other UML tools.

The CLR domain has also seen some changes, especially in the behavior of the LoadAssembly.exe tool. If you like thinking and speaking in terms of set theory, you might enjoy hearing, for instance, that LoadAssembly "no longer calculates and loads the transitive closure of all assemblies referenced by the assemblies specified on the command line" which prevents, among other things, multiple copies of the .NET Framework metadata being loaded. A good thing, for certain.

The other highlight we want to point out is a new sample called PatternApplication that helps you easily apply standard repository patterns to a set of models, rather than applying patterns to each model individually. You can find that sample on the MSDN code gallery.

Speaking of which, the all-up SQL Server Modeling samples page, is found on http://code.msdn.microsoft.com/SQLServerModelingCTP.

The latest information on SQL Server Modeling Services can now be found on the Data Developer Center at http://msdn.microsoft.com/en-us/data/ee720189.aspx.

"Quadrant" Changes

Now to "Quadrant". This tool continues to get more exciting. For some of us, we're most thrilled that the default color scheme has finally landed on something less sea-sick green and more my-eyes-won't-hurt blue. For the other 98.54%, what's most exciting is how much more deeply the "M" language has been integrated into "Quadrant". For example:

You can write "M" directly in a workpad (see File | New | M File) and deploy a database directly from that.
You can convert an existing database to "M", modify it, and redeploy.
You can view the template configuration (as it's called) in "M" for any viewer, make changes, and apply those changes to customize a viewer (try the Workpad | View Source menu).
"M" is now the default query language in the query bar of workpads (within which you get full syntax-checking)

The overall "Quadrant" UI has also been greatly refined, with full context menus all around, nuances like automatic workspace panning for newly-opened workpads, and a set of features around change tracking, conflict resolution, and error handling.

Something else you can really start playing with now are custom commands and custom viewers. An interesting sample in this regard is the "Media Library for ‘Quadrant'" that you can see listed along with all the other CTP samples on http://code.msdn.microsoft.com/SQLServerModelingCTP.

A few other notables:

The new T-SQL Console in which you can carry out queries and other operations with direct T-SQL statements (File | New | T-SQL Console).
Many performance improvements including table virtualization.
Support for SQL Server authentication.
The ability to just open a fresh, empty workpad and write a query to populate it (File | New | Workpad).

We also want to thank the "Quadrant" Program Management team for producing a full set of videos that will walk you through many of the core scenarios of "Quadrant". One of our favorites is the "Quadrant" UI Overview in which UI designer Stephen Danton really shows the fluidity with which "Quadrant" can be used to visualize data in ways that no other tool can match.

The latest information on "Quadrant" can now be found on the Data Developer Center at http://msdn.microsoft.com/en-us/data/ee720190.aspx.

The "M" Language and Tools

And now we come to "M". As you can see from its integration in "Quadrant", the "M" language is finding powerful roles beyond the command line and "Intellipad" (it's worth remembering that the name "Intellipad" comes from "IntelliSense Workpad" in the context of "Quadrant".)

Some notable new language features:

The ability to reference labeled entity instances in expressions and across compilation episodes.
Entities and collections now support equality evaluations.
One-to-many relationships no longer create a separate join table.
Logical operators
String concatenation with the + operator
Member access in expressions
Projectors
.Count support
Type conversions
Right-hand side expressions in "M" grammars

In other words, the "M" team has been busy!

There are a number of breaking changes you should know about, that are again highlighted in the Release Notes. One such change will affect quite of bit of existing "M" code: the syntax for collection types. Generally speaking, collections now get wrapped with curly braces:

T* becomes { T* } T#n becomes { T#n } T#n.. becomes { T#n.. } T#..x becomes { T#..x } T#n..x becomes { T#n..x }

As a concrete example, a common form of an extent declaration looks like this, where Person is a previously-declared type:

People : {Person*};

If the collection type (like Person) is declared directly inline, then you'll see a syntax like this:

People : {{
Name : Text;
Age : Integer32;
}*};

In short, curly braces are needed around whatever type is used in the extent, and the multiplicity operators like * appear with the type inside the braces. (A good place to see some of these differences highlighted, the way, is in the transcript for the Modeling in Text video series where we point out code that's changed.)

There are some changes also to the "M" tools, and many to "Intellipad" such as block commenting, syntax coloring, and the ability to reverse engineer "M" from a database. And then there's much to do with Visual Studio 2010 projects. It's really quite a long list!

What Do You Mean, "Data Developer Center"?

No, those aren't typos in the text nor the URLs. The whole business of naming "Oslo" and finding its rightful home has brought us into an intimate partnership with many other existing data development technologies. To make a long story (and a whole lot of work) short, the front-side slogan of our PDC t-shirts, "It's All Data", applies to the MSDN Developer Centers: you'll find that http://msdn.com/oslo now takes you directly to the highly-covetedURL http://msdn.com/data.

But we’ll save all the details for another blog post, as this Letter has become long enough already. We’ll just close then to say, as always, that we’d love to hear your feedback on this CTP through the SQL Server Modeling Forum as well as the SQL Server Modeling Connect site. You can also just send email to dpfback (at) microsoft.com.

Enjoy--
Kraig (Brockschmidt), Elisa (Flasko), Kent (Sharkey), and Chris (Sells) for the “It’s All Data” Developer Center Team

Next "Oslo" CTP to be released in conjunction with PDC

2009-11-02T19:56:00Z

Many of you are surely asking when the next CTP for "Oslo" will be released, since we haven't had one since May and also because the May CTP has some incompatibilities with the recently-released Beta 2 of Visual Studio 2010 and .NET Framework 4.

With the PDC now only two weeks away, we're putting the finishing touches on the "Oslo" CTP that will be available for download that that time (November 17th). We're also working to update many other pieces of content (such as samples, videos, articles, etc.). Note that the new CTP and samples will then require Visual Studio 2010/.NET 4 Beta 2--so if you haven't installed that Beta, now is a good time.

There will be much more to talk about and share when PDC arrives, so stay tuned.

Kraig Brockschmidt

First Modeling Sessions Posted for PDC'09

2009-08-19T02:54:00Z

Data Programming and Modeling for the Microsoft .NET Developer, by Don Box and Chris Anderson. Watch the Modeling sessions page for more as they're posted. PDC home page is http://www.microsoftpdc.com.

What's So Compelling about "Quadrant" Anyway?

2009-07-31T19:26:00Z

By Michael Murray, Senior Program Manager
(Edited November 17, 2009, to reflect the renaming of "Oslo" to SQL Server Modeling)

Microsoft code name "Quadrant" is a ‘tool for viewing and editing SQL data,' but... so what? There are many tools for viewing and editing SQL including Microsoft Access, Excel, and SQL Server Management Studio as well as custom-created applications using .NET. But "Quadrant" is different in certain respects:

"Quadrant" automatically ‘lights up' when you connect to SQL data. Without writing any code or queries, "Quadrant" drives the viewing and editing experience by interpreting the relational model at runtime. Views are automatically created for individual records and collections. Foreign keys are displayed and can be dragged onto the workspace to follow relationships. "Quadrant" uses the structure of the data to enable a good editing experience that encourages exploration by following foreign key references.
"Quadrant" also lights up when you connect to a relational database, such as the Repository in SQL Server Modeling Services. For example, if you model Repository folders, "Quadrant" will use your folders to provide a Windows Explorer-like navigation experience.
"Quadrant" provides a built-in configuration experience that lets users customize the generated data views, switch viewers, and compose the built-in views to create new views.

Here are some examples of how "Quadrant" is different. To see these examples yourself, install the "Quadrant" Sample Data. Once you've followed the instructions on DevCenter and installed the "Quadrant" Sample Data, you can verify that the samples are installed by opening the Repository Explorer (using View/Explorer/Repository from the "Quadrant" menus) and you should see the Explorer like the following:

Folders

The first thing you can see is that the example uses Repository folders. Organization is the top-level organizing folder, with subfolders for Events, People, and more. If you use Repository folders, "Quadrant" will use them for navigation in the Explorer. Content not organized using folders is displayed under the Catalog node in the Explorer.

Exploration

Consider this view (by clicking on OrganizationGroups in the Repository Explorer, then drag the icon next to ‘Business Analysts' onto the workspace):

Not only can you see the (simple) properties associated with Business Analysts, as you'd expect, but you can also see a References section. The References section is in the default properties view. Here, you can see that there are five PersonToOrganization instances; or in other words, there are five people who are business analysts.

This view was generated by "Quadrant" based on the underlying structure and the multiplicity of the related instances. No code or query was necessary for creating this view, only the model defined using SQL or the "M" language.

If you drag the icon to the right of PersonToOrganization you can see a table view of the persons who are business analysts...again without writing code or queries:

The query bar in the PersonToOrganizationGroups workpad shows the query that "Quadrant" used to retrieve the contents displayed, but you didn't have to write it: You merely dragged something onto the workpad.

Likewise, you can follow the relationship in the opposite direction: If you drag the icon next to Business Analysts from a row in the PersonToOrganizationGroups table view, you'll see the properties for Business Analysts again. "Quadrant" encourages exploration by putting these relationships front-and-center in built-in views.

Built-In Customization

As a refinement to the above example, you can again view Business Analysts in a workpad. But this time, first click once on the icon to the right of PersonToOrganization:

Then select View -> View Item As -> Table from the "Quadrant" menu. You should then see something like this (you may need to widen the workpad):

This is a custom, composed view created using the "Quadrant" ‘building blocks'. Here, the table viewer is hosted inside the ‘References' section of the properties view for Business Analysts. You can now save this view as a new, named view (like Business Analysts with Persons Table) by selecting View -> Save View As and specifying a name in the dialog.

[Note: for more customization capabilities now available in the SQL Server Modeling November 2009 CTP, see the "Quadrant" technology page on the DAta Developer center, http://msdn.microsoft.com/en-us/data/ee720190.aspx, especially the videos and additional samples. All of the CTP samples can be found on http://code.msdn.microsoft.com/SQLServerModelingCTP.]

Summary

"Quadrant" provides an out-of-box viewing, editing, and navigation experience that ‘lights up' with SQL data. In this way "Quadrant" is different than traditional tools for accessing SQL in that relationships and the underlying models drive the experience, without writing custom code or queries. Instead of looking only at the raw data, as database tools generally provide for, "Quadrant" surfaces much more of the meaningful structures that were in the minds of those who originally designed the models. And that makes for a much richer, efficient, and accurate means of communication between designer, developer, and database administrator, and experience that will continue to expand as "Quadrant" is developed further.

SQL Script to Uninstall an "M" Image

2009-07-23T20:52:00Z

by Oz Evren, Software Design Engineer for Microsoft code name "Oslo"

One feature that is commonly requested for the “M” command-line utility, mx.exe, is being able to uninstall images. This is very useful when you’re developing a set of models and want to make sure you’ve cleaned up prior installations in the repository before the next test iteration. Or perhaps you’re just following through a sample but screwed something up.

In the May CTP bits, we started working on parts of uninstall support for schema created by an image, but didn’t expose it as a user-visible feature as we will in future CTPs. So if you want to get your hands on this functionality right now, give this SQL script a try (below and attached). This creates a stored procedure creatively named “Uninstall,” which takes one argument, the id of the image to uninstall which can be obtained from the [Catalog.Runtime].[Images] table.

Enjoy.J

Note: this script is corrected from an earlier version.

create procedure [Catalog.Runtime].[Uninstall]

@image int

begin

declare @schemas as table ([schema_id] int not null primary key);

with [AllModuleNames](ModuleName) as (

select M.Module from [Catalog.Runtime].[ImageModules] as M

where M.[Image] = @image

)

insert into @schemas

select S.[schema_id] from sys.schemas as S

inner join [AllModuleNames] as M

on M.[ModuleName] collate Latin1_General_100_CS_AS_KS_WS = S.[name] collate Latin1_General_100_CS_AS_KS_WS or

'$MRuntime.'+M.[ModuleName] collate Latin1_General_100_CS_AS_KS_WS = S.[name] collate Latin1_General_100_CS_AS_KS_WS

declare @items as table ([ordinal] int not null, [type] sysname not null, [object_id] int null, [schema_id] int not null, [constraint_name] sysname null);

insert into @items select 1, N'trigger', O.object_id, O.schema_id, null from sys.triggers as T

inner join sys.objects as O on O.object_id = T.object_id

where O.schema_id in (select [schema_id] from @schemas);

insert into @items select 2, N'procedure', O.object_id, O.schema_id, null from sys.objects as O

where O.type = 'P' and O.schema_id in (select [schema_id] from @schemas);

with [Constraints]([name], [schema_id], [object_id], [parent_object_id]) as (

select F.name, F.schema_id, F.object_id, F.parent_object_id from sys.foreign_keys as F

union all

select D.name, D.schema_id, D.object_id, D.parent_object_id from sys.default_constraints as D

union all

select C.name, C.schema_id, C.object_id, C.parent_object_id from sys.check_constraints as C

)

insert into @items select 3, N'constraint', C.[parent_object_id], C.[schema_id], C.[name] from Constraints as C

inner join sys.tables as T on C.parent_object_id = T.object_id

where T.schema_id in (select [schema_id] from @schemas);

insert into @items select 4, N'view', V.object_id, V.schema_id, null from sys.views as V

where V.schema_id in (select [schema_id] from @schemas);

insert into @items select 5, N'function', O.object_id, O.schema_id, null from sys.objects as O

where (O.type = 'FN' or O.type = 'TF' or O.type = 'IF') and O.schema_id in (select [schema_id] from @schemas);

insert into @items select 6, N'table', T.object_id, T.schema_id, null from sys.tables as T

where T.schema_id in (select [schema_id] from @schemas);

insert into @items select 7, N'type', T.user_type_id, T.schema_id, null from sys.types as T

where T.schema_id in (select [schema_id] from @schemas);

insert into @items select 8, N'schema', null, S.[schema_id], null from @schemas as S;

declare @script nvarchar(max) = N'

set xact_abort on;

begin transaction;

declare @type sysname;

declare @item_schema_id int;

declare @item_object_id int;

declare @constraint_name sysname;

declare itemCursor cursor local fast_forward for

select [type], [schema_id], [object_id], [constraint_name] from @items order by [ordinal] asc;

open itemCursor;

fetch itemCursor into @type, @item_schema_id, @item_object_id, @constraint_name;

while @@fetch_status = 0

begin

declare @item_name nvarchar(max) = quotename(schema_name(@item_schema_id));

if (@item_object_id is not null)

begin

if (@type = N'type')

begin

set @item_name += N'.' + quotename(type_name(@item_object_id));

end

else

begin

set @item_name += N'.' + quotename(object_name(@item_object_id));

end

if (@constraint_name is not null)

begin

set @script += N'alter table ' + @item_name + N' drop ' + @type + N' ' + quotename(@constraint_name) + N';';

end

else

begin

set @script += N'drop ' + @type + N' ' + @item_name + N';';

end

fetch itemCursor into @type, @item_schema_id, @item_object_id, @constraint_name;

end

close itemCursor;

deallocate itemCursor;

set @script += N'

delete from [Catalog.Runtime].[ImageResources] where [Image] = ' + convert(nvarchar,@image) + N';

delete from [Catalog.Runtime].[ImageModules] where [Image] = ' + convert(nvarchar,@image) + N';

delete from [Catalog.Runtime].[ImageDependencies] where [DependentImage] = ' + convert(nvarchar,@image) + N';

delete from [Catalog.Runtime].[Images] where [Id] = ' + convert(nvarchar,@image) + N';';

set @script += N'commit transaction;';

begin try

exec(@script);

end try

begin catch

declare @ErrorMessage nvarchar(max);

declare @ErrorSeverity int;

declare @ErrorState int;

select

@ErrorMessage = error_message(),

@ErrorSeverity = error_severity(),

@ErrorState = error_state();

RAISERROR (@ErrorMessage, @ErrorSeverity, @ErrorState);

end catch

end

Recent DevCenter Additions, Updates, and Content Levels

2009-07-16T01:18:00Z

Since we released the May CTP of the code name "Oslo" technologies more than a month ago now, we've been continuing to add new content and make some additional changes to the Oslo DevCenter site itself. As often happens, you put these things out into the world, then listen to the silence! So we wanted to take a moment to highlight those changes and additions to make sure y'all are aware of them. This also gives you a chance to think about what you'd like to see as well.

Recent Additions

First, we've added a couple of new samples:

MRepl: ("M" read-evaluate-print loop): source code for what was originally built into the "samples enabled" mode of Intellipad.
WMI: Models for Windows Management Instrumentation that serves as reference material for hierarchical models in "M".

We also added a healthy paper on the MGraph Object Model.

Here's the list of new videos so far (with more in the works-they're not the easiest of things to produce, but we know they're valuable...summertime is also peppered with vacations):

Using the CLR Domain Part 1 | Part 2 | Part 3
Exploring the "Oslo" Repository and Connecting to Data in "Quadrant"
Using "M" to Create Custom Icons and Display Names in "Quadrant"
Installing and Uninstalling "Oslo"

We've also been working to refresh a number of videos that were originally made for the October 2008 CTP and are getting quite stale. Like chomping on an old half-bag of Fritos, they do the job but just don't deliver on the ideal experience! So here are some we've gotten refreshed already:

We also refreshed Modeling in Text which is now in five parts. On some of the individual video page (as well as the main Video page) you'll see that we've been adding some slide decks as well as some transcripts. In fact, there is now also a transcript index page that we snuck in there.

Again, more is in the pipeline, and we're also finishing up May CTP-compatible versions of existing samples (like Spork, PhotoStore, Event Pattern Mesh, and MGrammar XAML, for which there are some videos to refresh).

We've also been working over the videos index page itself (kindly, you know, not with a meat tenderizer or anything). For starters, we're making sure that more current and significant videos are in the rotators on the home page and the videos page.

Content Levels

Something else that's made its appearance on the videos page are content "level" tags to give you some idea where the piece falls between the introductory end of things ("100" level) and the "really deep dive Ph.D.'s-only" technical end ("400" level). Over time we intend to tag most of our content this way, with the videos being a great place to start.

These levels are intended to be a guideline to help you self-select from our all-you-can-eat buffet (to continue the pattern of food analogies here...) based on your interest and experience. Here's how we're generally using them:

100: These are introductory materials for any given technological area ("M", "Quadrant", repository, domains, "Intellipad", overall "Oslo", etc.) in which you can expect to see more diagrams and basic demos. We like to think of these are your freshman-level courses in college that try to get everyone in a degree program on some common foundation. Part of the intention here is to identify materials that might serve developers as well as technical decision-makers.

200: These materials go more into usage scenarios, architectural discussions, and "what things do" without necessarily getting into details so that can again serve technical decision-makers and developers both. These are like the sophomore-level classes that covered how computers are designed and how their components relate, but you aren't yet getting into actually building one (of course, you probably took a few apart in your spare time-and found that if your tool of choice was a ball-peen hammer you were probably more destined for testing than development).

300: You'll often see this number with the words "core competency," meaning that we consider these materials-again in any given technological sub-area-to convey essential information for all developers using that particular technology. These are like the degree-specific but non-specialized junior-year courses where you do the real work of your chosen discipline (like melting diodes as I clearly remember in my first EE lab).

400: At the high end, these are materials we consider specialized in some way that will probably only be of interest to a small percentage of developers, but very important to that small percentage. Like senior-level college courses, these would hone your particular expertise above and beyond the core competencies of level 300.

The boundaries between these levels can be somewhat fuzzy at times, of course, and their specific definitions are evolving as we're starting to apply them. Feedback is always welcome.

It's also worth noting that the distribution of content is not flat across these levels. We're looking to have 10-15% of our total content to be 100-level, 10-15% at the 200 level, 55-60% at the 300 level, and 15-20% at the 400 level. At least we can absolutely guarantee that it'll all add up to 100% in the end!

The last bits we wanted to point out are the key blogging activities of members of the "Oslo" product team. These have been appearing in the "What's New" section of DevCenter alongside articles written by other members of the "Oslo" community at large, so you may have already seen them. Some have also been specifically written to add to the overall body of our DevCenter content, so we wanted to provide a list of these posts (in no particular order):

M Interop with XML (Jeff Pinkston)
What is the Purpose of Types? (Jeff Pinkston)
Structural vs. Nominal Typing (Jeff Pinkston)
From DSLs to Models to "Quadrant" using "Oslo" May CTP Part 1 | Part 2 | Part 3 (Dana Kaufman)
Top Six Common MGrammar Integration Errors (David Matson)
(Transcript) Don Box and Chris Anderson Explain "Oslo" in 5 Minutes [actually 6 minutes and 32 seconds]
The Microsoft "Oslo" Base Domain Library (the BDL) -- now not completely in T-SQL! (Ralph Squillace)
"Oslo" Repository in the May CTP (Aim Mangkorntong)
Modeling Inheritance in "M" (Dave Langer)
"Oslo" and UML (Keith Short)

As always, there will be more.

What Content Would You Like to See?

Speaking of more: if there is any content you'd like to see, or anything that you'd change about the "Oslo" DevCenter, don't hesitate in dropping a line to oslodc@microsoft.com or creating a bug on the Connect site. This DevCenter is here to support you in your development efforts, so any suggestions (whether they be nasty or nice) about how we can improve that support are welcome.

Kraig Brockschmidt (for myself, Chris Sells, Kent Sharkey, and the "Oslo" team)

(Transcript) Don Box and Chris Anderson Explain "Oslo" in 5 Minutes [actually 6 minutes and 32 seconds]

2009-07-06T22:00:00Z

This post is an edited transcript of the short video on http://channel9.msdn.com/posts/Dan/C9-Bytes-Don-and-Chris-explain-Oslo-in-5-minutes/. Even though the video is from October 2008 and slightly dated, it contains some helpful insights about the motivations behind “Oslo” and some of the thinking of “Oslo” architects such as Don and Chris. For that reason we wanted to make the discussion accessible in (mildly edited) text.

OK, “Oslo”. Assuming that I know nothing about “Oslo”, what is it? What’s the explanation for average developer?

Don: “Oslo” is three things: a tool for interacting with information [“Quadrant”], a language for creating and manipulating information [“M”], and it’s a repository for storing that information.

Why are we building it? Because we believe there is great value in writing software that’s more and more declarative, more and more data-driven, where the data that you’re writing down is more closely aligned with the problem domain that you’re trying to deal with rather than MSIL or x86 assembler.

Does this replace other technologies like WCF and WPF, that you both worked on?

Chris: No, it’s not a replacement solution. We see ourselves as being 15 years into a 30-year journey toward being more and more declarative, more and more data-driven. Think of WCF with its configuration system, WPF with XAML, WF with the ability to describe whole program flows in structured data—we’ve been moving down this path for quite a long time. We really view “Oslo” as the next step in the path where you go from a suite of data solutions where some people bet on config, some people bet on XAML, some people do database things, to a holistic solution that lets people build tools and suites and systems on top of a common data runtime.

Don: Like when we did XAML, you didn’t have to rip out all of your code. Yes, it is possible to write a complete program in XAML, in appropriate scenarios, but that isn’t necessarily what you want to do all the time.

So what are the killer scenarios for “Oslo”? When is “Oslo” the right thing? What do you see as the key scenarios that you’re really targeting?

Don: Many of us on the team came from middle-tier, server-side technologies, and if you look at a typical WF/WCF application, there’s a sea of data that’s used to create that. There’s a contract footprint, a fair amount of config footprint, and then—oh yes—there’s also this code you write which is some of the logic. When I throw WF into that mix, a lot of that logic also gets expressed as data. But it’s smeared across a bunch of different ways to write it down. So I think the value proposition of what we’re doing in “Oslo” is immediately visible if you take a look at an existing WF/WCF app that’s written in the current technologies. That’s one scenario.

Is “Oslo” just about the middle-tier or are you looking to where you can code-gen XAML?

Chris: People confuse what we’re doing in “Oslo” with code-generation which is very traditionally how we did modeling systems: you wrote some description or drew some pictures and that generation thousands of code artifacts. In fact, if you look at early versions of WPF, we use to do code-gen with tons and tons of C# files. We eventually found out that going with a data-driven approach and having the WPF runtime itself understand a model-driven approach improved performance in many ways.

Don: And made the engineering hygiene better.

Chris: Way better. And so I think there are many interesting spaces to also go into on the client. I imagine a set of scenarios with parsing user input, where they type a string in and I can [in the program] understand what they wrote. I can use portion of “Oslo” to do just that. And if you start with the tiny micro-scenarios I think there’s an immediate value [with CTP bits today] to an incredible number of applications from client to server to middle-tier, etc. As the technology developers you’ll be able to incrementally add value from there.

Top Six Common MGrammar Integration Errors

2009-06-17T03:16:00Z

by David Matson, Test Engineer for Microsoft code name "Oslo"

The examples in this article work with the May 2009 CTP.

Summary

Using the M languages and types features together can be tricky for the uninitiated. I've collected some of the most common issues together with the best current solutions or workarounds.

Overall Scenarios

You are creating your own Domain Specific Language (DSL) that you want to parse against a custom grammar. You want to take the parsed output of your grammar and load it into a Repository. You may also want to have this parsed output conform to a specific schema that you have defined in an external .m file.

Common Error #1: Getting Non-Text Output from MGrammar

For this scenario, assume you have a list of names and birthdays that you want to parse and load into a database.

Example DSL

John Smith 1/1

Jane Doe 5/15

MGrammar

module Samples

{

language Birthdays

{

token Name = ("a".."z" | "A".."Z")+;

token Month = "1".."9" | "10" | "11" | "12";

token Day = "1".."9" | "10" | "11" | "12" | "13" | "14" | "15" | "16" | "17"

| "18" | "19" | "20" | "21" | "22" | "23" | "24" | "25" | "26" | "27" | "28"

| "29" | "30" | "31";

token FullName = Name " " Name;

syntax BirthdayLine = n:FullName " " m:Month "/" d:Day "\r\n"

=> {Name => n, Month => m, Day => d};

syntax Main = l:BirthdayLine* => Birthdays{valuesof(l)};

}

MSchema

module Samples

{

Birthdays : {

Name : Text;

Month: Integer8;

Day : Integer8;

}*;

}

Problem

1. Save the above files as Sample.birthdays, Birthdays.mg, and BirthdaysSchema.m, respectively.

2. Compile the grammar.

m.exe Birthdays.mg

3. Execute the DSL text against the grammar to produce Sample.m.
mgx.exe Sample.birthdays -reference:Birthdays.mx -mmodulename:Samples

4. Try to compile the DSL output together with your schema and produce a SQL script.
m.exe BirthdaysSchema.m Sample.m -package:Script

5. You get the following errors:
BirthdaysSchema.m(3,3): error M0144: The initial value 'Birthdays { { Name = "John Smith", ... }, ... }' of the extent 'Birthdays' is not compatible with the extent's type 'Collection(anonymous entity { Name; Month; Day; })'.
Sample.m(5,22): error M0152: Literal '"1"' cannot be converted to the type 'Integer8'.
Sample.m(6,20): error M0152: Literal '"1"' cannot be converted to the type 'Integer8'.
Sample.m(10,22): error M0152: Literal '"5"' cannot be converted to the type 'Integer8'.
Sample.m(11,20): error M0152: Literal '"15"' cannot be converted to the type 'Integer8'.

Issue

MGrammar's parsed output currently only supports the text data type.

Workaround 1

Change all schema types to text and leave them that way in the database.

module Samples

{

Birthdays : {

Name : Text;

Month: ~~Integer8~~Text;

Day : ~~Integer8~~Text;

}*;

}

Workaround 2

Use two schemas, one designed for importing data from your DSL and another designed for querying your data in the form you expect. These tables could live in the same database or even two different databases (for example SamplesImport and SamplesQuery). Use your favorite database API (T-SQL scripts, SSIS, ADO.NET via C# or VB, etc.) to write logic to move/transform data from the import format to the query-friendly format.

Workaround 3

Create a custom tool that reads your parsed DSL output and changes the data types as necessary. In this example, that would mean removing the quotes after Month and Day in Sample.m. The revised set of commands would be as follows. (For brevity, the remaining examples use the abbreviated aliases for command-line parameters.)

1. m.exe Birthdays.mg

2. mgx.exe Sample.birthdays -r:Birthdays.mx -m:Samples

3. FixMyDataTypes.exe Sample.m (your custom tool that updates Sample.m in place)

4. m.exe BirthdaysSchema.m Sample.m -p:Script

Workaround 4

Create a custom tool that replaces mgx.exe and fixes the data types before writing Sample.m. The revised set of commands would be:

1. m.exe Birthdays.mg

2. MyMgxWithDataTypeConversion.exe Sample.birthdays Birthdays.mx Samples

3. m.exe BirthdaysSchema.m Sample.m -p:Script

Below is a skeletal example of how this program could be implemented (for brevity and clarity, no checks are made for errors such as null reference exceptions).

using System;

using System.Collections.Generic;

using System.Dataflow;

using System.IO;

using System.Text;

namespace MyMgxWithDataTypeConversion

{

class Program

{

static int Main(string[] args)

{

if (args.Length != 3)

{

Console.Error.WriteLine(

"Usage: MyMgxWithDataTypeConversion.exe FileToParse CompiledGrammarMx ModuleName");

return 1;

}

string fileToParse = args[0];

string compiledGrammarMx = args[1];

string moduleName = args[2];

DynamicParser parser = DynamicParser.LoadFromFile(compiledGrammarMx, null);

parser.GraphBuilder = new NodeGraphBuilder();

Node root = (Node)parser.Parse(fileToParse, null);

GraphStore store = root.Store;

foreach (Edge recordEdge in root.Edges)

{

Node record = recordEdge.Node;

foreach (Edge field in record.Edges)

{

string label = field.Label.Text;

if (label == "Month" || label == "Day")

{

store.SetConstantValue(field.Node, int.Parse((string)field.Node.AtomicValue));

}

string destinationFile = Path.ChangeExtension(fileToParse, "m");

StringBuilder fullMFile = new StringBuilder();

fullMFile.AppendFormat("module {0} ", moduleName);

fullMFile.AppendLine("{");

fullMFile.Append(root.WriteToString());

fullMFile.AppendLine("}");

File.WriteAllText(destinationFile, fullMFile.ToString());

return 0;

}

Sample Compilation

The source code is also available in the download that accompanies this post.

The code above references objects in the System.Dataflow assembly. To compile, you must add a reference to this assembly (located at C:\Program Files\Microsoft Oslo\1.0\bin\System.Dataflow.dll or a similar location).

Sample Overview

The basic cookbook for writing an mgx.exe replacement is as follows:

1. Load a parser for the grammar (using DynamicParser.LoadFromFile).

2. Run the parser against an input file (using parser.Parse).

3. Perform some custom actions on the output graph from the parser.

4. Serialize the modified output graph to an .m file.

In this example, the custom step is finding any fields named Month or Day and replacing the contents of each field with a parsed integer instead of the original string. For more information on working with the M graph object model, see the paper by Clemens Szyperski (http://msdn.microsoft.com/en-us/library/dd878360.aspx).

Common Error #2: Module Name Mismatch

For this scenario, you use mgx.exe to create a .m file with your parsed DSL data. You try to match the DSL data with a schema in another .m file. When you run m.exe with these two files, you get strange error messages or incorrect results.

Solution

When you run mgx.exe, make sure you specify the same module name that your schema file uses. For example (when using BirthdaysSchema.m from Common Error #1 above):

mgx.exe Sample.birthdays -r:Birthdays.mx -m:Samples

If you leave off -m, mgx.exe uses the name of the DSL text file as the name of the schema, which usually is not what you want. In this example, m.exe would still run without errors, but the resulting SQL script will contain two schemas: Samples.Birthdays (with your schema from BirthdaysSchema.m) and Sample.Birthdays (with your data from Sample.m). This is an easy mistake to make, so the best practice is always to specify the module name when calling mgx.exe, and double-check that the module name here matches any .m schema files.

Common Error #3: Duplicate Labeled Instances

New MGrammar users often assign labels to everything. Sometimes, however, a label can do more harm than good. Record entities parsed through MGrammar are usually better left unlabeled, and definitely should not have duplicate labels. The following is a common error based on the Birthdays grammar above:

syntax BirthdayLine = n:FullName " " m:Month "/" d:Day "\r\n"

=> Birthday{Name => n, Month => m, Day => d};

In this case, the output M would be:

module Samples {

Birthdays {

Birthday {

Name => "John Smith",

Month => "1",

Day => "1"

Birthday {

Name => "Jane Doe",

Month => "5",

Day => "15"

}

The highlighted portions result in duplicate labels, which will cause compilation problems. In this case, the specific errors would be as follows:

BirthdaysSchema.m(3,3): error M0144: The initial value 'Birthdays { Birthday { Name = "John Smith", ..., ... } }' of the extent 'Birthdays' is not compatible with the extent's type 'Collection(anonymous entity{ Name; Month; Day; })'.

Sample.m(4,13): error M0187: Cannot initialize a node more than once using '='.

Sample.m(5,13): error M0187: Cannot initialize a node more than once using '='.

Sample.m(6,13): error M0187: Cannot initialize a node more than once using '='.

Solution

Do not label record entities (recommended). Or, if you do label record entities, make sure each record has a unique label, for example:

syntax BirthdayLine = n:FullName " " m:Month "/" d:Day "\r\n"

=> id(n){Name => n, Month => m, Day => d};

Common Error #4: Populating Parent/Child Data

For this scenario, assume you have a list of managers and their direct reports that you want to parse and load into a database.

Example DSL

Steve manages Bob and Craig

Lisa manages Carol and Cindy

Ray manages Amy

MGrammar

module Samples

{

language Managers

{

token Name = ("a".."z" | "A".."Z")+;

syntax ManagerLine = m:Name " manages " n1:(ns1:Name => {Name => ns1}) n2:(" and " ns2:Name => {Name => ns2})* "\r\n" => {Name{m}, DirectReports{n1, valuesof(n2)}};

syntax Main = l:ManagerLine* => Managers{valuesof(l)};

}

MSchema

module Samples

{

type Manager

{

Id : Integer32 => AutoNumber();

Name : Text where value.Count <= 100;

DirectReports : Report*;

} where identity Id && value.DirectReports <= Reports;

Managers : Manager*;

type Report

{

Id : Integer32 => AutoNumber();

Name : Text where value.Count <= 100;

} where identity Id;

Reports : Report*;

}

Problem

1. Save the above files as Sample.managers, Managers.mg, and ManagersSchema.m, respectively.

2. Compile the grammar.

m.exe Managers.mg

3. Execute the DSL text against the grammar to produce Sample.m.
mgx.exe Sample.managers -r:Managers.mx -m:Samples

4. Try to compile the DSL output together with your schema and produce a SQL script.
m.exe ManagersSchema.m Sample.m -p:Script

5. You get the following errors:
ManagersSchema.m(10,5): error M0144: The initial value 'Managers { { Name { "Steve" }, ... }, ... }' of the extent 'Managers' is not compatible with the extent's type 'Collection(anonymous entity { Id; Name; DirectReports; FieldNames; })'.

Sample.m(3,10): error M0195: To be an entity field, 'DirectReports' must have one and only one successor. (Did you forget an '='?).

Sample.m(16,10): error M0195: To be an entity field, 'DirectReports' must have one and only one successor. (Did you forget an '='?).

Sample.m(32,17): error M0206: To be an entity field, '"Amy"' must have a unique label.

Issue

When your DSL text file is parsed to M, the results are as follows:

module Samples {

Managers {

{

Name {

"Steve"

DirectReports {

{

Name => "Bob"

{

Name => "Craig"

}

{

Name {

"Lisa"

DirectReports {

{

Name => "Carol"

{

Name => "Cindy"

}

{

Name => "Ray",

DirectReports => {

Name => "Amy"

}

Notice that after DirectReports there is no => operator (or, equivalently, there is no extra set of braces around each record). The correct version of this file would be as follows:

module Samples {

Managers {

{

Name => "Steve",

DirectReports => {

{

Name => "Bob"

{

Name => "Craig"

}

{

Name => "Lisa",

DirectReports => {

{

Name => "Carol"

{

Name => "Cindy"

}

{

Name => "Ray",

DirectReports => {

{

Name => "Amy"

}

Solution

In order to produce this output, change the following line in your grammar:

syntax ManagerLine = m:Name " manages " n1:(ns1:Name => {Name => ns1}) n2:(" and " ns2:Name => {Name => ns2})* "\r\n" => {Name => m, DirectReports => {n1, valuesof(n2)}};

Note that the grammar must produce a total of three sets of curly braces (or => operators) between DirectReports and the first label/value pair inside it: 1 set of braces (or, in this case, the => operator) to denote the value of DirectReports, 1 set of braces to denote that the value of DirectReports is a collection, and 1 set of braces to denote an entity inside that collection. Also note that each record is unlabeled (see "Common Error #3: Duplicate Labeled Instances" above). Finally, in cases where the output is a valid entity, note that the .m file contains the => operator instead of one set of braces.

Limitation

In the schema above, the parent (manager) contains the child collection (direct reports). The child (direct report) does not directly reference the parent (manager) via a foreign key. Instead, a many-to-many table is automatically generated to store the links between parents (managers) and children (direct reports).

Workaround 1

Live with the many-to-many table in the database.

Workaround 2

Use two schemas, one designed for importing data from your DSL (with the many-to-many table) and another designed for querying your data in the form you expect (with a foreign key from direct report to manager). These tables could live in the same database or even two different databases (for example SamplesImport and SamplesQuery). Use your favorite database API (T-SQL scripts, SSIS, ADO.NET via C# or VB, etc.) to write logic to move/transform data from the import format to the query-friendly format.

Workaround 3

Create a custom tool that reads your parsed DSL output and changes the shape as necessary to conform to the a schema you prefer, such as:

module Samples

{

type Manager

{

Id : Integer32 => AutoNumber();

Name : Text where value.Count <= 100;

} where identity Id;

Managers : Manager*;

type Report

{

Id : Integer32 => AutoNumber();

Name : Text where value.Count <= 100;

Manager : Manager;

} where identity Id && value.Manager in Managers;

Reports : Report*;

}

In this example, your custom tool would need to add labels to each Manager, move the DirectReports to a separate collection off of the root, and add links from each DirectReport back to the correct manager. For the input above, the correct revised M would be:

module Samples {

Managers {

Steve {

Name => "Steve",

Lisa {

Name => "Lisa",

Ray {

Name => "Ray",

}

Reports {

{

Name => "Bob",

Manager => Managers.Steve

{

Name => "Craig",

Manager => Managers.Steve

{

Name => "Carol",

Manager => Managers.Lisa

{

Name => "Cindy",

Manager => Managers.Lisa

{

Name => "Amy",

Manager => Managers.Ray

}

Note that, when using this approach, your sequence of commands would change as follows.

1. m.exe Managers.mg

2. mgx.exe Sample.managers -r:Managers.mx -m:Samples

3. FixMyDataShape.exe Sample.m (your custom tool that updates Sample.m in place)

4. m.exe ManagersSchema2.m Sample.m -p:Script

Workaround 4

Create a custom tool that replaces mgx.exe and fixes the data types before writing Sample.m. The revised set of commands would be:

1. m.exe Managers.mg

2. MyMgxWithDataShapeConversion.exe Sample.managers Managers.mx Samples

3. m.exe ManagersSchema2.m Sample.m -p:Script

Below is a skeletal example of how this program could be implemented (for brevity and clarity, no checks are made for errors such as null reference exceptions).

using System;

using System.Collections.Generic;

using System.Dataflow;

using System.IO;

using System.Text;

namespace MyMgxWithDataShapeConversion

{

class Program

{

static int Main(string[] args)

{

if (args.Length != 3)

{

Console.Error.WriteLine(

"Usage: MyMgxWithDataShapeConversion.exe FileToParse CompiledGrammarMx ModuleName");

return 1;

}

string fileToParse = args[0];

string compiledGrammarMx = args[1];

string moduleName = args[2];

DynamicParser parser = DynamicParser.LoadFromFile(compiledGrammarMx, null);

parser.GraphBuilder = new NodeGraphBuilder();

Node root = (Node)parser.Parse(fileToParse, null);

Dictionary<string, List<string>> directReports = new Dictionary<string, List<string>>();

// Make the root modifiable (specifically, its edge collection).

root = new Node(root.Store, root.Brand, root.Edges);

foreach (Edge recordEdge in root.Edges)

{

string manager = null;

List<string> manages = new List<string>();

Edge? directReportsField = null;

foreach (Edge field in recordEdge.Node.Edges)

{

string label = field.Label.Text;

if (label == "Name")

{

// Get the name of the manager.

manager = (string)field.Node.AtomicValue;

}

else if (label == "DirectReports")

{

directReportsField = field;

foreach (Edge childRecord in field.Node.Edges)

{

foreach (Edge childField in childRecord.Node.Edges)

{

string childLabel = childField.Label.Text;

if (childLabel == "Name")

{

// Get the name of each direct report.

manages.Add((string)childField.Node.AtomicValue);

}

if (manager != null && manages.Count > 0)

{

// Cache the list of managers and the people they manage.

directReports.Add(manager, manages);

}

if (directReportsField.HasValue)

{

// Remove the direct reports field from each manager; it will be moved to a separate extent.

recordEdge.Node.Edges.Remove(directReportsField.Value);

}

if (manager != null)

{

// Re-write each manager record by adding a brand on the record with the manager name.

root.Store.SetBrand(recordEdge.Node, Identifier.Get(manager));

}

// Create a list of all records for the new Reports extent.

List<Node> reports = new List<Node>();

foreach (KeyValuePair<string, List<string>> item in directReports)

{

string manager = item.Key;

foreach (string report in item.Value)

{

// Create a list of fields for the new record.

List<Edge> fields = new List<Edge>();

// The Name field of the new record to the name of the direct report.

fields.Add(new Edge(Label.Get("Name"), root.Store.CreateConstant(Identifier.Empty, report)));

// The Manager field of the new record is a reference to the Manager's record (the manager name is

// the local record label, the extent is Managers).

fields.Add(new Edge(Label.Get("Manager"), new Node(root.Store, new GraphReference(new string[] {

"Managers", manager }))));

reports.Add(new Node(root.Store, fields));

}

// Turn the list of report records into a new extent node.

Node extraExtent = root.Store.CreateCollection(Identifier.Get("Reports"), NodeKind.Collection, reports);

// Save the graph to an m file. Use the name of the original text file as the name for the new m file.

string destinationFile = Path.ChangeExtension(fileToParse, "m");

StringBuilder fullMFile = new StringBuilder();

fullMFile.AppendFormat("module {0} ", moduleName);

fullMFile.AppendLine("{");

fullMFile.Append(root.WriteToString());

fullMFile.Append(extraExtent.WriteToString());

fullMFile.AppendLine("}");

File.WriteAllText(destinationFile, fullMFile.ToString());

return 0;

}

Sample Compilation

The source code is also available in the download that accompanies this post.

Sample Overview

The basic cookbook for writing an mgx.exe replacement is as follows:

1. Load a parser for the grammar (using DynamicParser.LoadFromFile).

2. Run the parser against an input file (using parser.Parse).

3. Perform some custom actions on the output graph from the parser.

4. Serialize the modified output graph to an .m file.

In this example, the custom steps are adding labels to each manager and moving direct reports to a separate extent. For more information on working with the M graph object model, see the paper by Clemens Szyperski (http://msdn.microsoft.com/en-us/library/dd878360.aspx).

Common Error #5: Changing the Shape between the DSL and the Repository

In many scenarios, the shape of the data in the input DSL does not match the shape you desire for the data stored in the Repository.

For example, suppose you have a flat list of transactions (date, amount) in your DSL:

1/1/2000 -100

2/1/2000 1000

2/1/2000 100

Suppose you want to populate the following schema:

MSchema

module Samples

{

type Summary

{

Date : DateTime;

Amount : Decimal19;

} where identity Date;

Summaries : Summary*;

}

In your schema, you want to have a total of transactions for each day (grouping and summing).

Limitation

Currently, MSchema does not provide any facility to do such reshaping between the input text and the output graph. (Also note that parsing directly to DateTime or Decimal19 rather than Text is not currently supported. See "Common Error #1: Getting Non-Text Output from MGrammar").

Workarounds

The workarounds for these scenarios are similar to those for "Common Error #4: Populating Parent/Child Data" above. Essentially, you can either forgo the transformation, or do the transformation yourself using custom tools at various layers of the process (during parsing, after parsing but before populating the Repository, or after populating the Repository).

Common Error #6: Referencing Data Dynamically

For this scenario, assume you are parsing transaction lines (account, date, amount). Assume you have a static list of accounts in your schema.

Example DSL

Checking 1/1/2000 -100

Checking 2/1/2000 1000

Savings 2/1/2000 100

MGrammar

module Samples

{

language Transactions

{

token Name = ("a".."z" | "A".."Z")+;

token Month = "1".."9" | "10" | "11" | "12";

token Day = "1".."9" | "10" | "11" | "12" | "13" | "14" | "15" | "16" | "17"

| "18" | "19" | "20" | "21" | "22" | "23" | "24" | "25" | "26" | "27" | "28"

| "29" | "30" | "31";

token Year = ("0".."9")+;

token Date = Month "/" Day "/" Year;

token Amount = "-"? ("0".."9")+;

syntax TransactionLine = n:Name " " d:Date " " a:Amount "\r\n"

=> {Account => n, Date => d, Amount => a};

syntax Main = l:TransactionLine* => Transactions{valuesof(l)};

}

MSchema

module Samples

{

type Account : {

Id : Integer32 => AutoNumber();

Name : Text where value.Count <= 100;

} where identity Id;

Accounts : Account* {

Checking {

Name => "Checking"

Savings {

Name => "Savings"

}

type Transaction : {

Id : Integer32 => AutoNumber();

Account : Account;

Date : DateTime;

Amount : Decimal19;

} where identity Id && value.Account in Accounts;

Transactions : Transaction*;

}

Limitation

MGrammar does not currently provide any way to lookup up data from an existing extent and dynamically create a label reference to that record. (Also note that parsing directly to Decimal19 or DateTime rather than Text is not currently supported. See "Common Error #1: Getting Non-Text Output from MGrammar").

Workaround 1

Forgo the lookup and change your schema to something like the following:

module Samples

{

type Transaction : {

Id : Integer32 => AutoNumber();

Account : Text where value.Count <= 100;

Date : DateTime;

Amount : Decimal19;

} where identity Id;

Transactions : Transaction*;

}

Using this approach, you would just store the text value used for lookup directly in the referencing table (instead of storing a foreign key).

Workaround 2

Use two schemas, the schema from Workaround1 for importing data from your DSL (without the foreign key relationship) and another designed for querying your data in the form you expect (with a foreign key relationship). These tables could live in the same database or even two different databases (for example SamplesImport and SamplesQuery). Use your favorite database API (T-SQL scripts, SSIS, ADO.NET via C# or VB, etc.) to write logic to move/transform data from the import format to the query-friendly format.

Workaround 3

Create a custom tool that reads your parsed DSL output and changes the shape as necessary to conform to the a schema you prefer.

In this example, your custom tool would need change the Account name into a label reference (from "Checking" to Accounts.Checking). For the input above, the correct revised M would be:

module Samples {

Transactions {

{

Account => Accounts.Checking,

Date => 2000-01-01,

Amount => -100

{

Account => Accounts.Checking,

Date => 2000-02-01,

Amount => 1000

{

Account => Accounts.Savings,

Date => 2000-02-01,

Amount => 100

}

Note that, when using this approach, your sequence of commands would change as follows.

1. m.exe Transactions.mg

2. mgx.exe Sample.transactions -r:Transactions.mx -m:Samples

3. AddLabelReferences.exe Sample.m (your custom tool that updates Sample.m in place)

4. m.exe TransactionsSchema.m Sample.m -p:Script

Workaround 4

Create a custom tool that replaces mgx.exe and adds the label references before writing Sample.m. The revised set of commands would be:

1. m.exe Transactions.mg

2. MyMgxWithLabelReferences.exe Sample.transactions Transactions.mx Samples

3. m.exe Transactions TransactionsSchema.m Sample.m -p:Script

Below is a skeletal example of how this program could be implemented (for brevity and clarity, no checks are made for errors such as null reference exceptions).

using System;

using System.Collections.Generic;

using System.Dataflow;

using System.Globalization;

using System.IO;

using System.Reflection;

using System.Text;

namespace MyMgxWithLabelReferences

{

class Program

{

static int Main(string[] args)

{

if (args.Length != 3)

{

Console.Error.WriteLine(

"Usage: MyMgxWithLabelReferences.exe FileToParse CompiledGrammarMx ModuleName");

return 1;

}

string fileToParse = args[0];

string compiledGrammarMx = args[1];

string moduleName = args[2];

DynamicParser parser = DynamicParser.LoadFromFile(compiledGrammarMx, null);

parser.GraphBuilder = new NodeGraphBuilder();

Node root = (Node)parser.Parse(fileToParse, null);

GraphStore store = root.Store;

foreach (Edge recordEdge in root.Edges)

{

Node record = recordEdge.Node;

Edge? date = null;

Edge? account = null;

foreach (Edge field in record.Edges)

{

string label = field.Label.Text;

if (label == "Date")

{

date = new Edge(field.Label, new Node(store, Identifier.Empty, Date.Parse((

string)field.Node.AtomicValue)));

}

else if (label == "Amount")

{

store.SetConstantValue(field.Node, decimal.Parse((string)field.Node.AtomicValue));

}

else if (label == "Account")

{

string name = (string)field.Node.AtomicValue;

account = new Edge(field.Label, new Node(store, new GraphReference(new string[] { "Accounts",

name })));

}

if (date.HasValue)

{

store.SetEdge(record, date.Value.Label, date.Value.Node);

}

if (account.HasValue)

{

store.SetEdge(record, account.Value.Label, account.Value.Node);

}

string destinationFile = Path.ChangeExtension(fileToParse, "m");

StringBuilder fullMFile = new StringBuilder();

fullMFile.AppendFormat("module {0} ", moduleName);

fullMFile.AppendLine("{");

fullMFile.Append(root.WriteToString());

fullMFile.AppendLine("}");

File.WriteAllText(destinationFile, fullMFile.ToString());

return 0;

}

class FixedGraphTextWriter : GraphTextWriter

{

TextWriter writer;

public FixedGraphTextWriter(TextWriter writer)

: base(writer)

{

this.writer = writer;

}

public override void Write(GraphStreamTokenKind tokenKind, Label label, Identifier brand, object value)

{

if (tokenKind == GraphStreamTokenKind.Constant && value is Date)

{

// Work around an issue with the serialization of dates.

string correctlyFormatedValue = ((Date)value).ToString("yyyy-MM-dd", CultureInfo.InvariantCulture);

typeof(GraphTextWriter).GetMethod("Write", BindingFlags.NonPublic | BindingFlags.Instance, null, new

Type[] { typeof(GraphStreamTokenKind), typeof(Label), typeof(Identifier), typeof(string), typeof(

char)}, null).Invoke(this, new object[] { tokenKind, label, brand, (object)correctlyFormatedValue,

'\0' });

}

else

{

base.Write(tokenKind, label, brand, value);

}

Sample Compilation

The source code is also available in the download that accompanies this post.

Sample Overview

The basic cookbook for writing an mgx.exe replacement is as follows:

1. Load a parser for the grammar (using DynamicParser.LoadFromFile).

2. Run the parser against an input file (using parser.Parse).

3. Perform some custom actions on the output graph from the parser.

4. Serialize the modified output graph to an .m file.

In this example, the custom steps are 1) finding any fields named Date or Amount and replacing the contents of the field with a parsed version of the original string, and 2) changing the transaction name from text to a record reference . For more information on working with the M graph object model, see the paper by Clemens Szyperski (http://msdn.microsoft.com/en-us/library/dd878360.aspx).

Samples for the "Oslo" May CTP

2009-05-27T21:23:00Z

A few people have asked "where are the samples"? As you can see from the Release Notes, they are no longer distributed with the "Oslo" download itself. So, "where are the samples" you ask? Well, you can now see the list on the Download page, but for those of you that want immediate gratification I’ve included the list below.

Sample Model Data for Microsoft code name "Quadrant"
This sample is composed of pre-compiled model schema and instance data for use with the "Quadrant" Tutorials in the MSDN Library.
Northwind Models with Instance Data
This sample includes the schema and instance data of the Northwind database, giving a complete representation of the entire Northwind database in the Microsoft code name "M" modeling language.
Hiring System C# Source Sample
This sample creates a .NET Framework class library that can be used as an input for Common Language Runtime (CLR) and Unified Modeling Language (UML) domains in Microsoft code name "Oslo" modeling technologies.
Calculator Library C# Source Sample
This sample creates two simple CLR assemblies that are intended to demonstrate the features of the System_Runtime model, part of the CLR domain in the code name "Oslo" repository.
Sample Models
These samples demonstrates many different queries, enumerations and relationships written in the Microsoft code name "M" modeling language.

Enjoy!

About the "Oslo" May 2009 CTP

2009-05-27T04:12:00Z

Hello once again from Kraig, Kent, and Chris, your editors on the "Oslo" Developer Center. We're delighted to announce the release of the "Oslo" May 2009 CTP. The product team has been busy stomping on bugs and adding features for this release--there are plenty of changes to read about in the Release Notes. Meanwhile, the documentation team has been working hard with many new topics and tutorials in the MSDN Library.

And it's easy for us to pick the most prominent and exciting feature in this release, which is...we'll make you wait a little for that unless you're really clever and read the end of this Letter first. But hey, why don't you go ahead and start downloading and installing the CTP now? (Fair warning: it does take a few minutes to churn!)

Overall Changes

Now as soon as you start the installation of this May CTP, you'll see that a few new things are added in, and many things have been taken out. Most significantly, the installation package itself has been simplified to only contain the essential Oslo binaries: the "M" toolchain, a setup for the "Oslo" Repository, and...well, we still can't spoil the surprise now, can we? (But we guess you already saw it in the setup program....)

What you won't see installed with this CTP package are samples and documents and basically anything that's not a binary with the singular exception of the obligatory readme. All of the samples are available now as separate downloads, and did we mention the updated and very much expanded documentation is available in the Library? Do take some time to check out the Library TOC while you're waiting for the install to complete!

The key "M" Language Specification document continues to be available in the Library as well but the CTP package no longer provides Word format copies of this spec or the Library contents. We recognize this will be an inconvenience for some--but check out this MSDN Downloader tool we found on CodeProject that will download a whole tree from MSDN into a single HTML file. (And if you think this sort of coolness should be part of the MSDN Library, tell us! Your feedback will help make it a reality.)

You'll also find that some older samples have disappeared from the Downloads page as they're either obsolete or have been incorporated directly into the tools now. We're also in the process of updating many of the resources on this site (such as videos and a few samples) that will--for the sake of sanity among all the developers and writers!--take a little more time to refresh.

"M" Changes

Speaking of the "M" Language Specification, one of the most significant features in this CTP is that the "M" and MGrammar Languages have been merged into a single specification (the "M" Language spec). This new specification has also absorbed the SQL Mapping Specification, which, like the MGrammar Language Specification, also disappeared from the Library's Technical Articles section.

The command-line tools have merged as well: m.exe now handles both .m and .mg sources, and we bid a fond farewell to the mg.exe tool (mgx.exe still exists but uses .mx files). Now you no longer need to bounce between the two tools to build your solutions! How many keystrokes will you save? Of course, the command-line parameters are all supported by m.exe now, and you should still be able to use your old mg.exe command-lines, batch files, etc., with nothing more than dropping a single "g".

Some new language feature to be jazzed about:

The scope-resolution operator :: that explicitly scopes an identifier to a module.
The about() keyword that queries the catalog for data about "M" declarations
You can reference external functions using extern, for instance: extern PreDefinedFunction() : Integer32;
There's a new MGraph Object Model (also known as the AST API) for working with grammars dynamically. We have a big writeup for that on the way, so stay tuned.

Again, the Release Notes is your place for these details and many more.

There are also, as you probably expect, a number of additional breaking changes in this CTP as the Release Notes discuss. On biggie is the => binding operator. In "M", the equal sign '=' is used to bind entity fields to values. It is also used to bind default constant values for extents. M is a functional language, and assignment (=) gave the wrong intuition about the language. We've thus changed the symbol to the binding operator '=>'. The '=' operator is still accepted, but you will get a deprecation warning for now.

For example:

People { 

John { Name = "John", Age = 25, Friends { Mary, Jane} },

Mary { Name = "Mary", Age = 22, Friends { John} }

Jane { Name = "Jane", Age = 39, Friends { John} }

}

becomes

People { 

John { Name => "John", Age => 25, Friends { Mary, Jane} },

Mary { Name => "Mary", Age => 22, Friends { John} }

Jane { Name => "Jane", Age => 39, Friends { John} }

}

It's also worth noting that there are some changes to mx.exe as well. One is that the /t:Repository switch for mx.exe is now deprecated, though still supported. The new switch is /t:TSql10 moving forward.

"Intellipad" Changes

The Release Notes, say plenty about "Intellipad" too. One thing you'll immediately notice, we imagine, is that "Intellipad" has gotten faster in this CTP. Woohoo! The product team has been putting a lot of effort into improving the startup time and general performance while working in the tool. OK, so this means that you won't have a few extra minutes to goof off every day and call it "work"...sorry about that, but we didn't think you'd mind too much.

Something else you'll enjoy: the ability to pop up the member completion IntelliSense using Ctrl-Space, just as in Visual Studio. You knew it was coming! There has also been some overhaul of the different modes (for which there is now also a reference in the docs). What's been commonly known as "three-pane mode" is now called "DSL Grammar Mode" where you can, as before, type some representation of your language in the input pane, your grammar in the grammar pane, and see whether parsing happened correctly in the output pane. It's really one of the quickest and easiest ways to develop a grammar in short order without needing a Ph.D. in compilers!

A change in the UI also affects what's been known as three-pane mode for working with grammars. How you used to do this we've already forgotten. What you do now is open a grammar file (.mg), then select "Split New Input and Output Views" from the DSL menu. You can read more about this in the wholly new area of the Library for "Intellipad".

You'll also probably want to go out tonight, grab a drink, and celebrate the disappearance of the separate "samples enabled" mode for "Intellipad"...most of those features are now built into "Intellipad" directly. (The one that isn't, Mr. Epl, will soon be provided as a separate download on the "Oslo" Developer Center).

Repository Changes & Domains

You can now create (and delete) repositories and folders using the mx.exe tool. Playing with folders is an important means for applying security, as shown in these two videos: Part 1 | Part 2 (that are slated for refresh, mind you; some of the UI details in the tools are outdated, but the concepts are still valid). More videos are in the works too--watch for them on the DevCenter.

This CTP also comes with repository domains for the Common Language Runtime (CLR), Identity, and for the Unified Modeling Language (UML). What do these mean, you ask? Well, we'll leave that up to the MSDN Library that--did we tell you?--has been very much updated and expanded. (Nevertheless, if you run into any topic in the Library where your mind is saying "Cool stuff, but what the blazes do I do with this?" we want to know! Just post a message on the Oslo Forum or file a bug on the Connect site.)

Oh, and note again the change in mx.exe from /t:Repository to /t:TSql10 as mentioned earlier.

"Other" Changes, a.k.a. Quad...(oh, just wait another moment)

Well, with all these epic changes, maybe it's a bit of an anticlimax to announce that our CTP now includes--as we're sure you've guessed by now--Quadrant! That's right, the star of the old Introduction to Quadrant video (which has quietly disappeared for the moment) has come out and is ready to play. Well, that's not quite true: it's more like a reincarnation. That is, for those of you who got a first look at Quadrant from PDC 2008 or watched the old video, you'll notice that Quadrant has gone through significant changes to simplify its UI, hone its purpose, and be ready in a state for you to start playing with it.

You can read all about Quadrant in the MSDN Library, as you would expect from that very much expanded and updated documentation.

The product team is especially eager to hear your feedback through the Oslo Forum (for questions and comments) and the Connect site (for bugs). Do note that the Release Notes list a number of known issues with Quadrant, so do check there first if you encounter something odd.

And there is yet more, of course, but by now your install should be complete so you can start playing with the bits!

Enjoy--
Kraig (Brockschmidt), Kent (Sharkey), and Chris (Sells) for the big happy the "Oslo" family

How to clean up the Repository

2009-04-06T22:30:00Z

Have you ever wondered, after a session of experimenting and playing with the M tools and the Repository, if there was a simple and reliable way to get your Repository back to its original state? A clean slate, of sorts, without the need to reinstall everything? Well, here’s one way – and while it might not be absolutely perfect, it’s the best way we know, and recommend (just ask your db admin friend): detaching the database and creating a copy.

Here’s some simple SQL and the steps you need to perform right after a new Repository installation. You can use SQL Server Management Studio to do this.

1. Find the name of the database file. You will need to know where the file is to attach the database after you detach it.

select physical_name from sys.database_files where type_desc = N'ROWS';

2. Find the name of the log file, for the same reason as #1.

select physical_name from sys.database_files where type_desc = N'LOG';

3. Detach the database. In order to be able to safely copy the files, you have to perform this operation.

alter database [<db_name>] set single_user with rollback immediate;

execute sp_detach_db @dbname=N'<db_name>', @keepfulltextindexfile=N'true';

4. Copy the files to a safe location. You can compress them, if you want to later move them between many locations, or use them many times and keep your space.

5. Attach the database. Here you use the file names you found in points #1 and #2.

create database [<db_name>] on

( filename = N'<data file path from point #1>' ),

( filename = N'<log file path from point #2>' )

for attach;

6. Run the server instance repair code.

This is a very important step: it will fix Repository-related items on the server – such as triggers – so they work properly after detach/attach. It will also fix the database owner to be ‘sa’, because the attach/detach operation changes the owner to whoever performed it. These operations are what your db admin friend would normally do after detaching and attaching a database.

execute [Repository].[InitializeServer]

7. Work, play, experiment with Repository!

8. Detach the database (#3).

9. Copy files back from safe location (uncompress them, if they were compressed).

10. Attach the database (#5).

11. Run the server instance repair code (#6), and voilà, your Repository is back in its clean state.

12. Rinse and repeat, from steps #7 here on out.

There are alternatives, when it comes to cleaning up Repository, and you may have used some of them to date: backup/restore is one of them. We do not recommend this for restoring clean database state – it is intended for disaster recovery, and is much less reliable than attach/detach shown here.

Going right to the source

2008-11-21T20:09:00Z

With any new language - whether it be a programming language or communication language – it’s always best to refer to a standard to decide if you’re using it correctly. We know that our spelling is correct by checking an authority (even if it’s only Word’s spell checker). Similarly, if there is some question about a programming keyword, checking the language specification gives us the authoritative answer.

“M” has a language specification, and it’s now available online. If you have any questions about “M” the language, you’ll find the answers in those pages. Don’t like the fact that it’s divided into seven sections on the Web site? We'll get the PDF and XPS versions back up shortly

For those of you that don’t like to read off the screen or that don’t want to use your own ink printing it can buy a bound copy from your favorite supplier. It makes a dandy gift as well!