Simon Ince's Blog : MSArchitectPortal

A JavaScript Event Bus

simonince — Mon, 21 Dec 2009 17:39:00 GMT

I was working on a Proof of Concept recently that needed to support loosely coupled Ajax components. The idea is a common one – components need to be able to hook into an existing page without detailed knowledge of how the page works, and immediately start responding appropriately. This is basically using loosely coupled events as an extensibility mechanism. It is by no means a new concept – just check out things like Prism for an example!

After some consideration we came up with an approach that I like – it is actually very simple, but works great. In JavaScript we have very few issues to do with avoiding sharing the wrong types, marshalling to UI threads, etc... so the basic implementation is incredibly simple. We end up with a central "Event Bus" component that receives events and publishes them to a list of listeners... something that Sys.Component and its EventHandlerList is perfect to assist us with. The code I ended up with is below;

Type.registerNamespace('Sample')

Sample.EventBus = function() {

Sample.EventBus.initializeBase(this);

}

Sample.EventBus.prototype = {

subscribe: function subscribe(eventType, callback) {

this.get_events().addHandler(eventType, callback);

publish: function publish(eventType, arg) {

var handler = this.get_events().getHandler(eventType);

if (handler)

handler(arg);

}

Sample.EventBus.registerClass('Sample.EventBus', Sys.Component);

As you can see, this really is quite simple. We have the notion of an "eventType", that identifies the message type passing over the bus, and we have a single argument that can be used to carry a payload.

We could easily use strings to identify these event types, but I decided I'd like to use a static property – and a great way to register such a property is to pinch an idea from Microsoft Ajax's registerComponent method... so we end up with something like this;

Sample.EventBus.registerEventType =

function registerEventType(target, eventType) {

if (!target.Events)

target['Events'] = {};

target.Events[eventType] = eventType;

}

This is designed to allow us to "extend" some type definition with static properties that identify any events we're going to use. It seems logical that the extended type is also the type of the payload argument to the events... so all this now allows me to register my events like this;

// simple class to carry data values with event

Sample.Payload = function(dataValue) {

this.data = dataValue;

}

// definition of event types relevant to Payload class

Sample.EventBus.registerEventType(Sample.Payload, 'Update');

Sample.EventBus.registerEventType(Sample.Payload, 'Click');

... and then to publish and subscribe to Event Bus messages like this;

bus.publish(Sample.Payload.Events.Update, new Sample.Payload('Some Data'));

... and this...

bus.subscribe(Sample.Payload.Events.Update, function(arg) {

// perform some action

});

To make locating the bus simple, I have made it a Sys.Component, and retrieve it using a simple static helper function;

Sample.EventBus.get_Instance = function get_Instance() {

return Sys.Application.findComponent('SampleEventBus');

}

Of course, the subscribers could be in any component or script – as long as they have a reference to the Event Bus and the Payload class (and you could remove this dependency and use strings if you didn't like it) they can listen and respond. The Event Bus could get more intelligent if we liked – perhaps logging, tracing, translation, or scheduled execution (to split up subscriber execution, hence avoiding blocking the UI thread)... and much more.

This brings the power of a loosely coupled event extensibility mechanism to an HTML page with JavaScript, and it could of course be used to compliment equivalent server side behaviour. Easy, huh? What do you think?

A really simple sample site is attached... try tabbing out of the text box, and then click the button. There are two components listening for different events. Try adding another to test my script!

The Microsoft Ajax Content Delivery Network (CDN)

simonince — Thu, 22 Oct 2009 16:36:00 GMT

I keep looking for the URL’s to get Microsoft Ajax, jQuery, the MVC Ajax libraries and more from our new shiny CDN (check here if you’ve not heard of it)... and I can never remember where they are.

So this post is for my benefit, not yours J

The links to all scripts are at http://www.asp.net/ajax/cdn/.

Loading Scripts Late

simonince — Tue, 20 Oct 2009 19:52:00 GMT

On the ASP.NET ScriptManager control there is a useful little property that very few people have heard of called “LoadScriptsBeforeUI”. What does it do?

Well, simply put...

· When “true” (the default) it will render <script> tags for each script you reference using the ScriptManager at the top of your HTML.

· When “false” it will render these <script> tags at the bottom of your HTML.

Oh right. Why would I bother?

Well to put it simply it is because all browsers stop parsing and rendering HTML while they are downloading and executing scripts... which means if you put your <script> tags at the top of a web page none of your page will start to render until all scripts are downloaded and dealt with. The bottom line is that while your page is downloading scripts, the page is usually blank, and your user is wondering why the page is taking so long to load.

In a development environment this may not seem particularly important – but deploy your application to somewhere bandwidth and network speed are unknown, and it can often become an issue. Users don’t like sitting waiting for your site to respond, so you could be losing users!

Let’s see it

Let’s see what this looks like in action – download the attached sample solution and view the two pages. One downloads my scripts first; and I’ve made sure one of the scripts takes 5 seconds to load, so it is obvious how slow the page is. The other page loads the script last... and you’ll notice the page renders immediately, without waiting for the script.

Have a look at the HTML source and see where the references to TestScript.js and GetScript.ashx are.

The Gotcha

I’ve just showed you how to improve the perceived performance of your web site by changing one setting from “true” to “false”! Right?

But I’m afraid there is a catch. You’ll also notice there is a button on my test web pages. This is to demonstrate something important; you need to be very aware of when the scripts are being downloaded. This leads to two opposing issues;

1. If you have scripts that execute as soon as they’re downloaded, they probably assume that the DOM has been built. In this case, they usually must be rendered last. I’d recommend using page load events instead, so that you don’t have this requirement.

My sample demonstrates this; only when TestScript.js is loaded after the UI has been rendered does the button work, because TestScript.js assumes the button exists and attaches an event handler. If I put this in pageLoad instead it wouldn’t be an issue.

2. If you have event handlers embedded in your HTML that depend upon some framework scripts you’re downloading later, the user could click the button before your scripts have downloaded and the handler would error! This approach to wiring up event handlers looks something like this;

<button onclick="callMethod()">Click Me</button>

The solution here is to attach event handlers programmatically, as I have done in TestScript, but in the pageLoad event instead. You could even consider defaulting your buttons/controls to a "disabled" state, enabling them once you've attached the event handlers. This can sit well with the principles of Unobtrusive JavaScript and progressive enhancement if your controls have default non-script behaviours, so is a good practice anyway.

The End...

I think this shows a useful technique that can really improve users’ perception of your web site... as long as you design your use of script to work correctly. Enjoy!

Executing JavaScript after a Partial Render

simonince — Thu, 30 Jul 2009 00:01:00 GMT

I had a fun day debugging some ASP.NET plus jQuery this week, and came across something I’ve known is possible for some time, but that I’ve never actually needed to do... and that was to ensure a bit of JavaScript ran once an UpdatePanel had refreshed as part of a partial render.

It turns out that this is easy; all you need to do is register your JavaScript code block with some methods on ScriptManager, rather than using the standard ClientScript methods.

For example, consider the HTML below (I’ve omitted the rest of the page including the script manager, but it is in the download);

<asp:Button

ID="ClickMeButton"

runat="server"

Text="Click Me to Refresh Below"

onclick="ClickMeButton_Click" />

<asp:UpdatePanel

ID="MyUpdatePanel"

runat="server"

UpdateMode="Conditional">

<asp:AsyncPostBackTrigger

ControlID="ClickMeButton"

EventName="Click" />

</Triggers>

Last Updated:

<asp:Label

ID="UpdateTimeLabel"

runat="server" />

</ContentTemplate>

</asp:UpdatePanel>

</div>

</div>

Here we have a simple UpdatePanel, with a button that triggers the async postback. There is also an empty DIV with ID “log”.

In the server-side code of the button click event I have some code that looks a bit like this;

string time = DateTime.Now.ToLongTimeString();

ScriptManager.RegisterStartupScript(

UpdateTimeLabel,

typeof(Label),

"UpdateTimeLabelStartupScript",

CreateScript(time),

true);

The CreateScript function simply returns some JavaScript in a string that reads as follows (the {0} is replace with a time);

$('#log').append('Last Server Run: {0}<br />');

This uses jQuery to append a message to our “log” DIV.

There is also an alternative though; using the Sys.Application.add_load method to attach a page load script function, or creating a pageLoad JavaScript function that is a synonym for the same thing. I’ve done the latter;

function pageLoad() {

$('#log').append('pageLoad executed<br />');

}

This function is executed when the page initially renders and when every partial render completes. This means the content of my “log” DIV rapidly starts to look like this;

pageLoad executed

Last Run: 21:56:12

pageLoad executed

Last Run: 21:56:14

pageLoad executed

Last Run: 21:56:19

pageLoad executed

Easy huh?

I hope that adds some clarity to your options for executing script as the result of a partial postback. Check out the attached if you want to try it yourself.

Do you need web client development help?

simonince — Tue, 21 Jul 2009 15:01:00 GMT

Then tell us what you need! Blaine has blogged a link to a survey that patterns & practices will use to help them understand what challenges that are affecting you the most... so that they can choose the right areas to focus effort in providing guidance.

If you’ve got a couple of minutes spare head over to the survey...

What tasks or areas are difficult? What consumes the most time? What is it that causes you post-deployment headaches? What parts of the process are tricky? What don't you do because it is hard?

Monitor your ASP.NET Cache API Behaviour

simonince — Mon, 20 Jul 2009 14:36:00 GMT

Recently I was working on diagnosing a performance issue with a customer’s web site with a colleague (this is one of our favourite engagement types so if you need some help let me know J), and we found that items were being trimmed very regularly from the ASP.NET Cache, causing excessive back-end work, and in turn reduced scalability.

* a “cache trim” is when ASP.NET looks for unused (based on a LRU algorithm) cache entries and deletes them.

So if you’re using the ASP.NET Cache API (or indeed any cache provider) to store custom application data, the moral of the story is to make sure you size your cache appropriately, and monitor it during test and live to see how it behaves.

It is worth knowing that an un-configured ASP.NET Cache takes a memory limit of the minimum of either 60% of physical RAM, or a fixed size RAM. This fixed size is different depending upon your processor architecture - see Thomas Marquardt's post here for some more detail. The idea is that when the limits imposed by configuration are approached, the cache will start trimming.

I’ve attached a very simple ASP.NET application that you can play with to study the cache behaviour while you’re reading the next section. Host it in IIS for the best results.

Setting Cache Limits

It is possible to set limits on how much memory an ASP.NET Application Pool can consume in IIS (see here), and you can also configure behaviour using settings on the cache web.config element. It is important to understand exactly what these mean;

· privateBytesLimit (in web.config, on the cache element): This is the maximum number of bytes that the host process is permitted to consume before the cache is trimmed. In other words, it is not the memory to dedicate to the cache, nor a limit to the memory that the worker process can use... but rather the total memory usage by the w3wp process (in IIS7) at which cache trimming will occur.

· percentagePhysicalMemoryUsedLimit (in web.config, on the cache element): This is the maximum percentage of RAM that the machine can use before the cache is trimmed. So on a 4gb machine with this set to 80%, the cache would be trimmed if Task Manager reported over 3.2gb (i.e. 80% of 4gb) was in use. Note that this is not the percentage memory used by ASP.NET or caching – it is the memory used machine-wide.

· Private Memory Limit (in IIS7 manager, as an advanced property of the App Pool): This is the maximum number of private bytes a worker process can consume before it is recycled (which will of course also empty the cache). If you set this limit lower than privateBytesLimit in web.config, you’ll always get a recycle before a cache trim... which sounds unlikely to be what you would want.

Things to Watch

The remaining content of this post itemises things you should consider monitoring to assess your cache’s performance. Some of it is obvious – but it is surprising how easy it is to overlook the obvious when your boss is breathing down your neck about your slow application J

Overall Application Performance

The best indicator that something might be going wrong is if your application is performing badly. Or maybe it was performing fine, but you’ve changed something or gotten more users, and suddenly it’s not so fast. Make sure you collect timing data in your IIS logs, and use LogParser to parse and analyse them. Also use standard performance counters from ASP.NET, WCF, and whatever other technologies you’re using. Get a feel for where and when things are slow or failing. This isn’t specific to caching, but it is an essential first step!

Hits on the resource you’re caching data from

Do some simple maths on the number of hits your back-end resource is getting. For example, if you have a “GetCities” web service, the results of which are cached for 2 hours, and you have 4 front-end load balanced web servers each with their own cache, you should expect a maximum of 4 hits to that web service every 2 hours. If you’re seeing more than that, alarm bells should be ringing.

Cache Entries

There are some great performance counters for the ASP.NET Cache API, so use those to monitor the state of the cache. Specifically the “ASP.NET Apps\Cache API Entries” counter is the number of items that are in the ASP.NET Cache right now. It should be broadly stable over longer periods with approximately the same load. If you cache an item per user, per region, or anything similar, be aware that this can dramatically affect your cache behaviour and memory consumption... in which case Cache Entries can be quite revealing.

Cache Trims

The “ASP.NET Apps\Cache API Trims” counter is the number of cache items that have been removed due to a memory limit being hit (i.e. they were trimmed). Obviously this should ideally be very low or zero under normal operation. Too many trims could indicate you need to revisit your caching strategy, or manually configure cache memory limits.

Cache Turnover Rate

The “ASP.NET Apps\Cache API Turnover Rate” counter shows the number of newly added and removed/expired/trimmed cache items per second. A target for this number depends on the behaviour of your application, but generally it should be fairly low – ideally items should be cached for quite some time before they expire or are removed. A high turnover rate could imply frequent trims, frequent explicit removals in application code, dependencies firing frequently (e.g. a SqlCacheDependency), or a sudden increase in user load (if, for example, you’re caching something per user).

Private Bytes

Monitoring the “Process\Private Bytes” counter for the w3wp process (assuming you’re using IIS7) tells you how much memory IIS is using for your ASP.NET application. A cache trim is likely to show up as a sudden drop in consumed bytes, and equally you should be able to see how close it is to memory limits for your configuration.

Worker Process Restarts

It is initially a little easy to confuse the cause of a drop in consumed Private Bytes between heavy cache trimming and an Application Pool recycle, so you should also watch the “ASP.NET\Worker Process Restarts” performance counter to ensure you know which happened.

Cache Removed Reasons

When you add items to the ASP.NET cache you can optionally specify a CacheItemRemovedCallback. This means that when the item is removed from the cache the call-back you’ve specified is called, passing in a CacheItemRemovedReason. This is great for adding some debugging instrumentation – if your item was trimmed due to memory pressure then the reason will be “underused”. Other reasons are Expired (you specified a time limit), Removed (you called Cache.Remove) and DependencyChanged (a file, SQL table, or other dependency setup with a CacheDependency was fired).

If you decide to add some logging using this approach I’d recommend adding a switch that enables or disables setting a call-back, as there is a small overhead in dealing with it.

Cache Memory Percentages

There are two slightly mysterious sounding cache counters too. I must admit, at first it took me a few minutes to get my head around exactly what these counters meant.

The first is “ASP.NET Apps\Cache % Machine Memory Limit Used”. This is literally the current physical memory usage by the whole machine as a percentage of the configured (or default) maximum physical memory usage. What?! Well, if you have edited the “percentagePhysicalMemoryUsedLimit” setting to 60%, this means your machine can use up to 60% of its physical memory before cache trimming occurs (not a very realistic example I know!). This counter reports the current usage as a percentage of the maximum... so if your machine is using 40% of available physical RAM, and the limit is 60%, this counter would report 66% (40 divided by 60, multiplied by 100 to get a percentage). It is important to note that this is memory consumed across the whole machine – not just by ASP.NET.

The second is “ASP.NET Apps\Cache % Process Memory Limit Used”. This is the total Private Bytes in use by the current ASP.NET process as a percentage of the limit specified in “privateBytesLimit”. So if you set the limit at 400mb, and your process is currently using 350mb, that would be reported as 87.5% (350 divided by 400, multiplied by 100 to get a percentage).

If either of these counters hits 100% ASP.NET will almost immediately trim 50% of your cache by picking the least recently used items... so obviously you don’t want to be hitting these limits often.

Conclusion

Phew. Well I hope that’s useful stuff. The best advice I can give is to do some real sums. Many customers cache data items per user, plus huge lists of reference data that can grow over time. The end result can be caches that are crippled over a fixed concurrent user level, or take a long time to reload large reference data sets when they have been trimmed. It is more than possible to do some rough maths to work out how much memory your application is using for the cache, and how this changes according to user numbers, regions, languages, locations, roles, user types, base offices, or other parameters – and the results can be very illuminating.

Finally, remember that you should always test using hardware, configuration, and simulated user loads that are as close to live as you can possibly afford, as this gives you the best possible chance of catching problems early.

Enforcing Unobtrusive JavaScript

simonince — Fri, 10 Jul 2009 21:07:00 GMT

Unobtrusive JavaScript is a concept that’s been around for quite a while, but it’s now finally starting to make its way into the mainstream. The basic idea is that you should separate your JavaScript from your HTML – so instead of;

<a href="Default.aspx" onclick="alert('Sorry, that is unavailable'); return false;" >Click Here</a>

You should have something like this;

<a id="ClickHereLink" href="Default.aspx">Click Here</a>

... with some matching JavaScript in another file somewhere;

function pageLoad() {

var link = $get('ClickHereLink');

$addHandler(link, 'click', preventClick);

}

function preventClick() {

alert('Sorry, that is unavailable');

return false;

}

I’m not here to tell you why this is so great, but the best discussion and walkthrough of issues I’ve found is this article on onlinetools.org, so check it out if you’re interested.

How do I enforce it?

Recently I found myself cursing some code I’d written a long time ago, when I found it littered with onclick and onload handlers. It was frankly a pain to work out how it works. It was at this point that I thought “I should write something to shout at me when I write don’t write unobtrusive JavaScript”.

Coincidentally I have also been doing some work on HTML parsing, and I have to say I’ve found the XPath-like features in the HTML Agility Pack to be incredibly useful. If you’ve never used it I’d recommend taking a look.

It occurred to me that this was the perfect tool to do what I wanted – so I set about creating a simple MSBuild task that uses the HTML Agility Pack to scan ASPX and ASCX files for Obtrusive JavaScript – basically a form of static code analysis. If you’re interested in customising MSBuild, check this out.

The implementation is very simple;

· It is derived from Microsoft.Build.Utilities.Task.

· There is a property that is an array of ITaskItems, flagged with a [Required] attribute, as a way to pass in a list of files to check.

· There’s a hard-coded (for now) list of validation XPath expressions, with matching explanations.

· In the Execute method override, it loops through all the files.

· For each file, it opens the HTML and applies every XPath to it.

· If there are any matches for the XPath, it creates a build failure.

The key to success here is the simplicity with which the HTML Agility Pack lets me use XPath to find elements. The following is the basics of my code;

HtmlDocument doc = new HtmlDocument();

doc.Load(path);

HtmlNodeCollection results =

doc.DocumentNode.SelectNodes(

validator.SearchXPath);

Easy huh? The XPath expressions themselves mostly look like this;

//*[@onkeypress]

This will match any HTML element that has an onkeypress attribute... which of course means that the developer has specified a JavaScript event handler in the HTML, and broken the rules of unobtrusivitiy (hey, if that’s not a word I’m having first dibs on inventing it J).

Remembering that we’re actually scanning ASP.NET mark-up, not pure HTML, we can also check for ASP.NET attributes that actually create JavaScript event handlers in HTML too;

//*[@onclientclick]

Again, this is not difficult. The tricky one appears when you realise that it is valid to have OnClick in ASP.NET code! Of course, an <asp:button> can have an OnClick attribute referring to the server-side event handler. I wondered about saying “match any with an onclick that don’t have a runat attribute”, but this isn’t quite accurate (HtmlControls can have runat=”server” too remember).

In the end I went for explicitly excluding the ASP.NET controls that I knew were OK to use OnClick;

//*[@onclick and not(

name()='asp:button' or name()='asp:bulletedlist' or

name()='asp:imagebutton' or name()='asp:imagemap' or

name()='asp:linkbutton')]

Finally, I wanted to check that there was no JavaScript inside the page HTML itself... even between <script> tags. In other words, the only valid <script> tags should reference an external file, as follows;

[Of course, you could also use the <asp:ScriptManager> control if you wanted to; in fact I’d prefer it]

To achieve this I check that there is nothing other than whitespace between the start and end script tags;

//script[string-length(normalize-space(text()))>1]

... and that’s it! As usual, the source code for this MSBuild Task is attached to this post, along with a demonstration Web App.

Wiring up the Task

Now we have a simple MSBuild task, all that is left to do is add it to a build. I created a new ASP.NET Web Application, right clicked the project file and chose “unload”, and then edited the XML manually.

The only bit I needed to add was as follows;

<UsingTask

AssemblyFile="..\JsBuildTasks.dll"

TaskName="ValidateUnobtrusive" />

<ValidateUnobtrusive

SourceFiles="@(Content)" />

</Target>

First I import my task (I’ve copied a Release build of the assembly to the parent directory above my Web App project). Next, I override the BeforeBuild target, and add an instance of my ValidateUnobtrusive task. I’ve passed in an expansion of all the Content files in the project. The task filters for *.aspx and *.ascx at the moment – if you wanted to add *.html that would be trivial.

Save the file, and right click it to choose Reload Project. At this point Visual Studio 2008 warns you about a customised MSBuild project file – just choose “Load Project Normally” and you’re away.

From this point on, whenever you build a project containing Obtrusive JavaScript you’ll see errors like these;

Error Use of the 'onclick' JavaScript event on 'a' tag in markup causes Obtrusive Javascript. You should attach this handler in script instead.

Error Failed on 'script' tag . A JavaScript region containing something other than white space was found. You should link to related JavaScript resources files rather than embed script in a page.

Conclusion

I think this works really well so far (although I’m at the early stages of trialling it); if you try it do let me know how you get on. One of the particularly cool things (that I haven’t tried yet as I’m working isolated on my laptop most of the time right now, and I’m short of time) is that this could be wired up into a Continuous Integration build, under Team Build – I guess that would be a good plan if you didn’t like running the validation as part of your local build, or if it starts to perform badly due to the complexity or size of your pages.

I also think it raises other questions and possibilities. For example; how about some static analysis of “.js” files themselves? Could this help check for Directly Accessible Scripts, for example?

String.Unformat: I've created a monster

simonince — Thu, 09 Jul 2009 20:42:00 GMT

I don’t know if you’re the same, but when coding away I often find myself wishing for a String.Unformat function – call it the evil twin of String.Format. With String.Format I can build up strings like this;

var result = String.Format(

"http://{0}:{1}/{2}",

"localhost",

"12345",

"TestPage.aspx");

... which will return "http://localhost:12345/TestPage.aspx".

But what if I want to do the opposite; split out the contents of this URL, in one easy statement, much like String.Format? What if I could do this...?

string input = @"http://localhost:12345/TestPage.aspx";

object[] results = input.Unformat(@"http://{0}:{1}/{2}");

CollectionAssert.AreEquivalent(

new object[] { "localhost", "12345", "TestPage.aspx" },

results);

Of course something similar is already possible with Regular Expressions, but I find them much more unwieldy, they result in longer code, they’re overkill for what I’m trying to achieve, and they’re difficult to remember the syntax (oh, and even harder to get it right when coding late at night!). Take this equivalent example and judge for yourself;

string input = @"http://localhost:12345/TestPage.aspx";

string matchingformat = @"^http://(?<C1>.+):(?<C2>.+)/(?<C3>.+)$";

Regex expression = new Regex(matchingformat);

Match match = expression.Match(input);

Assert.AreEqual<object>("localhost", match.Groups["C1"].Value);

Assert.AreEqual<object>("12345", match.Groups["C2"].Value);

Assert.AreEqual<object>("TestPage.aspx", match.Groups["C3"].Value);

To get around this I have created my own String.Unformat function... all it does is take a simple format string and convert it into a regular expression by applying a number of transforms to it. And how do we do these transforms? With regular expressions, of course J

Now before we dive in here, I must make a point. This is not a replacement for regular expressions. It will be difficult to get the precision you need for anything complex – I’m only after a quick and simple solution to the basic case problem. Knowing this, read on...

How it does it

My sample code (and a bunch of demonstrations in the form of unit tests) is attached. Basically there are five phases to the code;

1. It escapes any special Regular Expression characters (e.g. ^, [, ], $ and so on) with a backslash so that they don’t accidentally affect the matching... they are assumed to be literal in the input string.

2. It replaces the {0} match syntax with regular expression (?<C1>.+) syntax.

3. It adds begin (^) and end ($) markers to the match string.

4. It performs the match.

5. It loops through the results, extracting matches and adding them to an object array to return.

Have a look in the attached and see what you think.

But...

The regular expressions I’m using are compiled where possible... but of course I’m applying multiple transforms (i.e. regular expressions) to a string just to do a simple match.

What does this mean? It means this will always be slower than if you just wrote a regular expression yourself (check out here and here if you want to)... so if performance really is that key to your code, or you’re doing this in a long loop, you might want to avoid it. If on the other hand you can see a great use for this when performance isn’t key – give it a go and let me know what you think!

I have also not had time to write a full suite of real unit tests, so if you find a bug shout up.

Conclusion

What do you think? Have I created a monster, or cooked up a treat?

After writing this I came across an interesting conversation here... it seems I have not been alone in my desires for String.Unformat.

Running two Visual Studio 2008 configurations side-by-side

simonince — Mon, 29 Jun 2009 17:02:00 GMT

Recently I’ve come across a requirement to easily switch between two Visual Studio 2008 configurations. I might need these two configurations open at the same time (so can’t just manually edit settings in Tools->Options), and don’t want any lengthy scripts, log on/log off activities, or virtualisation involved... mainly because each of those adds effort and is therefore a possible blocker to adoption.

One solution seemed to be using the “RunAs” command, and to configure each user’s settings differently. But I have seen issues with using RunAs – for example, a TFS installation with a customer failed a number of times and we couldn’t work out why (and never did), but logging on instead of using RunAs solved the issue. It can also take a while to do the logon before starting the executable. So I’m a little cautious of recommending it.

After a couple of internal conversations around solutions the Visual Studio SDK was discussed, which suddenly reminded me of the “Experimental Hive”. This is just another place in the Registry that Visual Studio can be instructed to load settings from instead of the usual place, and is generally used when testing customisations to Visual Studio (e.g. Domain Specific Languages) that might cause it to fail – so you don’t want to corrupt your main VS install.

It occurred to me that this is pretty much exactly what I’m after! Two sets of Visual Studio settings that can be used independently without machine virtualisation, user virtualisation, or anything else that might seem like overkill.

Solution

To get this working is quite simple. First, you need the VsRegEx.exe command from the Visual Studio 2008 SDK 1.1. Next, change into the directory it is in (I found it under “C:\Program Files (x86)\Microsoft Visual Studio 2008 SDK\VisualStudioIntegration\Tools\Bin”, but it may differ on your machine), and run the following command;

vsregex GetOrig 9.0 Test ranu

I’ve chosen to call my hive “Test”, but you could call it something else. This creates a new Registry hive under “HKCU\Software\Microsoft\VisualStudio\9.0Test” (alongside the usual “HKCU\Software\Microsoft\VisualStudio\9.0”) and file system storage under “%USERPROFILE%\Application Data\Microsoft\VisualStudio\9.0Test” (alongside the usual “%USERPROFILE%\Application Data\Microsoft\VisualStudio\9.0”).

Next, change to the directory that Visual Studio 2008’s “devenv.exe” command is stored. On my machine this is under “C:\Program Files (x86)\Microsoft Visual Studio 9.0\Common7\IDE” – again, yours may differ. You can then start Visual Studio with the following command line;

devenv /RootSuffix Test /ranu

This starts VS using your new “Test” hive. You could even setup a shortcut to start this version of Visual Studio running off your different configuration.

The first time it starts Visual Studio will take longer than usual as it is effectively a new install. But don’t worry; next time it will be just as quick as what you’re used to.

You are now free to go into Tools->Options and change your settings in this Visual Studio instance, and they won’t affect your standard Visual Studio install (the one started with “devenv.exe”, or from the standard shortcuts).

Caveats

As always there are some caveats. I have this working, and it seems fine, but I haven’t been using it long term for development. When developing for Visual Studio developers often use this approach for testing the software they have written, but generally not for doing their day-to-day development work in VS. Therefore I’d recommend giving it a try and seeing if you notice any odd behaviour specific to your configuration (Note: personally I think this is pretty unlikely as this approach was designed for testing Visual Studio! But I’m just being cautious J)

The final thing to be aware of is the “run as a normal user” switch (“/ranu”) that is used above. You may need to do some configuration as an elevated administrative user to get Visual Studio working if you’re planning on running VS as an admin (there are some notes on the ranu docs here). The most common reason for running Visual Studio as an admin is if you’re testing web apps in a locally hosted IIS rather than the built in Visual Studio Development Web Server. I haven’t tested this stuff, so try it first if this matters to you.

If you do use this and find anything interesting, feel free to report back here using a comment.

Anyway, I hope that helps – and I hope it saves you some time!

Using ASP.NET Dynamic Data in an existing Web Site

simonince — Fri, 01 May 2009 15:39:00 GMT

One of the tasks I used to hate when building a new system was adding admin pages to allow simple create/update functionality for lookup data. For example, maintaining a simple list of Products. It takes ages to write, and is very boring – even if you write templates to generate code it consumes a lot of time for such a commonly repeated and simple task.

ASP.NET Dynamic Data has now arrived, which gives a great way to “scaffold” CRUD pages over an Entity Framework Object Context (or other sources). But what if I have a web site that is mostly a line-of-business system, with complex business logic and custom UI, and I need to administer a small set of tables too?

Well, you can add Dynamic Data to a portion of your existing web site! This means you no longer need to write your own administration pages.

* Before you start off on this merry path, read this to make sure the relevant sections of your Entity Framework model are supported by Dynamic Data... for example, Complex Types are not.

How?

There’s a walkthrough that covers adding Dynamic Data to an existing web site here, so give that a read and you’ll be 80% of the way there. [Note: I also had to add a reference to System.Web.Entity at the end of the walkthrough to get it all working]

Next, make sure you have set ScaffoldAllTables to false – the last thing you want to do is expose your non-lookup data for direct editing by admin staff. All that should be exposed is a few key tables of static data. So, in Global.asax, make sure your data model registration looks like this;

model.RegisterContext(

typeof(OrderDbEntities),

new ContextConfiguration()

{ ScaffoldAllTables = false });

But how does the user get to this administration functionality? I’d recommend putting it under a virtual subfolder in your web site – perhaps named “Admin”. This is really just a route (read up on the new ASP.NET Routing features if this is unfamiliar) and needn’t really exist as a physical folder. So in Global.asax, we set it up as follows;

routes.Add(new DynamicDataRoute("Admin/{table}/{action}.aspx")

{

Constraints = new RouteValueDictionary(

new { action = "List|Details|Edit|Insert" }),

Model = model

});

Note that my route begins with “Admin/...”, which means the user will browse to URLs as follows to get to the admin pages;

http://{add address here}/myfakewebsite/Admin/Product/List.aspx

You can of course link to these pages from elsewhere within your site to save the user typing them in.

At the moment, we’re not scaffolding any of our tables, so none of these URLs will work yet. The next step, then, is to pick a table to expose for administration. I’m going to use a Product table that has ID, Description, Price and CreatedBy/CreatedDate fields. In my EF model the entity also has an OrderItems navigation property which allows me to get all Order Item records that are an instance of an order for the current product...

Due to the way EF supports partial classes, I can tell the Dynamic Data runtime to scaffold the Product table using an attribute (from the System.ComponentModel.DataAnnotations namespace & assembly) on a partial class. I added a C# class to the Web Site with the same namespace as the Entity Framework model, and pasted in the following code;

[ScaffoldTable(true)]

public partial class Product

{

}

Immediately this attribute is added the Product table is available for editing. There are a few problems when I browse to the new page though;

1. It is displaying the link to Order Items even though it isn’t editable (because Order Item isn’t scaffolded).

2. I can edit the columns I’m using for Audit – CreatedBy and CreatedDate. Not good.

It turns out changing this is quite easy. Using some additional attributes and a metadata class as follows customises the model sufficiently (see here for more info on this metadata class approach);

[MetadataType(typeof(ProductMetadata))]

[DisplayColumn("Description")]

[ScaffoldTable(true)]

public partial class Product

{

}

public class ProductMetadata

{

[UIHint("TextReadOnly")]

public string CreatedBy;

[UIHint("TextReadOnly")]

public string CreatedDate;

[ScaffoldColumn(false)]

public EntityCollection<OrderItem> OrderItem;

}

I’ve now instructed the runtime not to display the Order Item navigation property, and to use the Description field should I display Products in a list somewhere (as part of a foreign key drop-down, for example).

I’ve also asked it to use the “TextReadOnly” user control to render my CreatedDate and CreatedBy fields instead of the standard Text control, by placing a [UIHint] attribute on two metadata properties. To create the TextReadOnly control I copied-and-pasted the “Text.ascx” control twice, and renamed the copies to “TextReadOnly.ascx” and “TextReadOnly_Edit.ascx”. This control will now be used whenever the CreatedDate and CreatedBy fields are displayed (note I’ve cheated here – CreatedDate is really a date but as I’m happy with the default ToString display mechanism for it I didn’t bother creating a DateTime ReadOnly control).

As you can see, these user controls are available for you to edit under the DynamicData/FieldTemplates folder that was copied in as part of the original walkthrough. If you want to change this folder to be somewhere else, check out this article.

Finally, we probably want to limit access to the admin pages to a limited set of privileged users. The easiest way to do this is using standard ASP.NET authorisation, by adding a location and authorisation element to web.config;

<system.web>

</authorization>

</system.web>

</location>

Next steps...

There’s plenty more you might like to do, so I would recommend looking further into the features that Dynamic Data provides.

One that is particularly likely to be of interest is customising validation of data that is maintained... a good starting point is the ValidationAttribute.

You probably also want to change Site.master to better reflect your site (or move it, rename it, and update the templates under DynamicData/PageTemplates to use the new one). As long as you have a ScriptManager and the required Content Placeholder on there, it should work great. You can obviously make use of your own CSS too, or even go as far as changing all the field templates.

If you’d like a general overview of how Dynamic Data fits together, have a read of the Infrastructure Overview.

Finally, remember that I’m basing mine on an Entity Framework model, so I could customise the model too – for example, hooking up to the SavingChanges event to perform processing and/or validation on the Product table before it is saved to the database.

I hope all that is useful, and saves you time and boredom for your next system... enjoy!

Flattening XML Data in SQL Server

simonince — Fri, 24 Apr 2009 19:11:00 GMT

If you’ve got some XML data in a SQL Server column, how can you flatten it out and query it or present it as though it was relational data? It turns out this is quite easy...

Setup

Let’s create a simple table to hold our data;

CREATE TABLE XmlSourceTable

(

RecordId INT IDENTITY(1,1) NOT NULL PRIMARY KEY,

XmlData XML NOT NULL

)

And we’ll define some straightforward XML to import;

<?xml version="1.0" ?>

<Root>

<Name>Simon</Name>

<Skill>Cooking</Skill>

<Skill>Cleaning</Skill>

</Skills>

</Person>

<Name>Peter</Name>

<Skill>Ironing</Skill>

</Skills>

</Person>

</Root>

Ages may have been changed to protect the innocent J

Next, we’ll import it into my table using one of the mechanisms SQL Server provides – each XML file will be imported into a single row in the target table.

INSERT INTO XmlSourceTable(XmlData)

SELECT *

FROM OPENROWSET(

BULK 'C:\XmlSource.xml', SINGLE_BLOB)

AS ImportSource

After this, if we do a quick query...

SELECT * FROM XmlSourceTable

... we can see that we get a single row back containing an ID and some XML;

RecordId XmlData

----------- -------

1 <Root><Person><Name>Simon</Name><Age>20</Age... (snip)

(1 row(s) affected)

Queries

The simplest way to extract this data is to use the CROSS APPLY keyword, as this executes a function against each row and then adds the returned data to the result set. Combining this with a method that can be called on the XML data type called nodes, we get some great results. A quick query like this;

SELECT

pref.value('(Name/text())[1]', 'varchar(50)') as PersonName,

pref.value('(Age/text())[1]', 'int') as PersonAge,

pref.query('Skills') as PersonSkills

FROM

XmlSourceTable CROSS APPLY

XmlData.nodes('/Root/Person') AS People(pref)

... yields a completely different result set to our last query;

PersonName PersonAge PersonSkills

---------- --------- ----------------------

Simon 20 <Skills><Skill>Cooking</Ski... (snip)

Peter 21 <Skills><Skill>Ironing</Ski... (snip)

(2 row(s) affected)

We can see this query has flattened my single row of relational data with embedded hierarchical XML into two rows and columns of relational data. I’ve also included a subset of the XML as a column, just to show I can! Of course, if I wanted to I could modify this to get a list of people and their skills;

SELECT

pref.value('(Name/text())[1]', 'varchar(50)') as PersonName,

sref.value('(text())[1]', 'varchar(50)') as PersonSkill

FROM

XmlSourceTable CROSS APPLY

XmlData.nodes('//Person') AS People(pref) CROSS APPLY

pref.nodes('Skills/Skill') AS Skill(sref)

What is it Doing?

This SQL can be difficult to understand when you first look at it, but it isn’t really that complex. Breaking it down there are three key concepts that we’ll cover below – using my first query above that fetches PersonName, PersonAge, and PersonSkills as an example.

CROSS APPLY

The first concept is the use of CROSS APPLY, which many people haven’t used before. What this is doing is roughly equivalent to the following steps (note: this is my idiots guide to how I think about it, not a description of how the query optimiser does it);

1. Fetch the rows from the XmlSourceTable table.

2. For each row, call the “nodes” function on the XmlData column. This could be some other function too – it needn’t be on XML data. See the docs on APPLY for more info.

3. Duplicate the XmlSourceTable row once for every row returned by the table valued function “nodes”.

4. Add the columns returned by the “nodes” function to the columns in the result set.

5. Continue doing filtering, joining, and column selection as for any other SQL query.

I hope that makes it a little clearer.

nodes() function

The XML data type in SQL Server defines a number of methods that can be called on it. One of these is “nodes” – and what this does is basically select a list of XML nodes that match an XQuery expression. Knowing this, look at the statement in my SQL;

XmlData.nodes('/Root/Person') AS People(pref)

This is using the path “/Root/Person” to ensure that all Person nodes that exist under the Root are selected. The result is aliased as a table named People, and each resulting XML node will be output as a separate row, in the “pref” column.

Plenty of alternative syntaxes are available for selecting this list of nodes, and this really is the core of how to flatten out the XML. I’ve also used “//Skill” syntax in my second query, for example, which selects every Skill node found in the XML it is used on.

Other XML functions

The last concept is the use of the selector XML functions – in my example I’ve used “value()” and “query()”. Both of these use XQuery expressions to select XML data.

Firstly, I’ve used the value() function to extract a specific value from the XML, and convert it to a SQL Server data type;

pref.value('(Name/text())[1]', 'varchar(50)') as PersonName

The “text()” function here retrieves the inner text from within the XML “Name” node. The “[1]” suffix acts as an indexer, and fetches the first result matched. I know there’s only one name node per person, but I need to make sure SQL Server knows this too. Finally, the second argument is the data type it should be converted to. For an Age, I’ve used ‘int’, but here we’re converting to a varchar type.

The “query()” function allows me to return an XML fragment;

pref.query('Skills') as PersonSkills

This returns the XML that matches the “Skills” node, and is found underneath the current Person element held in “pref” (i.e. the search is relative to the contents of pref). This means it returns fragments such as;

<Skills><Skill>Ironing</Skill></Skills>

Conclusion

None of this is all that difficult once you know how – so I hope this has given you a quick start to using XML data in SQL! Don’t forget to read up more generally on the XML data type and XML indexes.

Auditing Data Changes in the Entity Framework: Part 2

simonince — Mon, 20 Apr 2009 17:02:00 GMT

In my previous post I described the basic concept behind my auditing approach using the Entity Framework, and covered some of the problems I encountered. This post focuses on the solution I went with; do feel free to comment if you have any thoughts.

Audit Lifecycle

To get to the root of my requirements I wrote down the lifecycle of a piece of data, and what needed to be tracked against it.

Action	Audit Requirements
Record Added	The date it was created, by whom, and what the original values are.
Record Modified * repeated many times	The date it was changed, by whom, and which values were changed.
Record Deleted	The date it was deleted, and by whom.

The problem is that creating an audit record is tricky based on point (2) in my previous post. Therefore, I chose to record the original values when each change is made. This means I actually track the following;

Action	Audit Requirements
Record Added	The date it was created, and by whom.
Record Modified * repeated many times	The date it was changed, by who, and what the pre-change values were.
Record Deleted	The date it was deleted, by who, and what the pre-delete values were.

This has effectively flipped the approach on its head, but I can still get a point-in-time view of the data whenever I need it. Therefore my data model becomes something like this;

When a new Product is added, the CreatedBy and CreatedDate fields are recorded on the Product entity. When changes are made, the previous unchanged version of the entity is copied to the ProductHistory table, and then the new Product entity is saved. When a Product is deleted, it is removed from the Product table and a copy of its last values is saved to ProductHistory.

This means my ChangedDate field on ProductHistory could arguably be described better as “ExpiredDate”, as it is the date and time that the data on that record became out of date.

Enabling Auditing on the ObjectContext

I described in my previous post how I would use the SavingChanges event to create audit records. To wire this up, I’ve created an extension method that can be called on an ObjectContext;

public static void Audit<fromType, toType>(

this ObjectContext context,

Func<IDataRecord, EntityState, toType> mapping,

Action<toType> addToContext)

{

context.SavingChanges +=

new EventHandler((o, e) =>

CreateAuditRecord<fromType, toType>(

context, mapping, addToContext));

}

Calling this method sets up auditing for one specific type of entity by adding an event handler to the SavingChanges event. The handler invokes a method called CreateAuditRecord.

You might think it would be better to set all audit types up at once, or you might want to configure auditing in the ObjectContext’s OnContextCreated partial method... and that should be easy enough. The point of this code is not to be final, but that it should be easy for you to understand and adapt.

So why have I used two type parameters, a Func<T,U,V> and an Action<T> on the Audit method then? It comes down to points (4) and (5) in my previous post; I wanted auditing to be very explicit about how it behaved, and I wanted to avoid passing string values around to identify field names or entity sets.

To demonstrate this, consider the following example of how I would enable auditing on the Product entity, saving changes to ProductHistory.

using (MyEntities db = new MyEntities())

{

db.Audit<Product, ProductHistory>(

(record, action) => new ProductHistory() {

Id = record.Field<Product, int>(f => f.Id),

Description = record.Field<Product, string>(

f => f.Description),

Price = record.Field<Product, double>(f => f.Price),

CreatedDate = record.Field<Product, DateTime>(

f => f.CreatedDate),

CreatedBy = record.Field<Product, string>(

f => f.CreatedBy),

ChangedBy = "Simon",

ChangedDate = DateTime.Now,

ChangeType = action.ToString() },

(ph) => db.AddToProductHistory(ph));

// TODO: make some changes to Product here...

db.SaveChanges();

}

In my Audit method I specify the source entity (Product) and target audit store entity (ProductHistory) as type parameters. This then enables me to easily create a strongly typed lambda for the first parameter that I know accepts an IDataRecord (“record”) and an EntityState (“action”), returning a ProductHistory record;

(record, action) => new ProductHistory() {

Id = record.Field<Product, int>(f => f.Id), ... <snip>

The purpose of this method is to map a Product data record to a ProductHistory entity. It has an IDataRecord input as I pass the OriginalValues to it, not the full Product entity (remember I’m saving the previous values in my audit table, not the new ones). Therefore this code creates a ProductHistory record, and initialises its properties using object initialiser syntax. If this code became too unwieldy I could easily factor it out into a helper method.

Next, you’ll notice that I’m using another extension method named “Field” that applies to the IDataRecord type. I use this to fetch a property value in a strongly typed way, and to make it easy to get at fields that represent an entity property.

In this case, I’m specifying that I expect the IDataRecord to contain fields that represent Product’s members. I’m also saying the particular property I’m after is an integer, and then I use a lambda to identify the property itself (Id in this case).

The second parameter to my Audit call looks like this;

(ph) => db.AddToProductHistory(ph)

This is because I need to know how to add audit records that are created to the ObjectContext, ensuring they are saved as part of the same SaveChanges transaction as the actual changes we’re tracking. The lambda receives a ProductHistory entity (which is known because of the “toType” type parameter in my call to Audit) which I choose to add to the ObjectContext by calling AddToProductHistory.

For this task I could have used the more general AddObject method available on an ObjectContext – but this needs a string value to identify the entity set, and I want as much compile time checking as possible.

The Mechanics of Auditing

Now you see how we setup auditing, let’s dive into the CreateAuditRecord method to see how it actually creates the audit trail.

First, the method retrieves a list of ObjectStateEntry objects that describe entities that have either been modified or deleted. Remember that my approach doesn’t record anything in the audit table when they are added;

IEnumerable<ObjectStateEntry> entities =

from e in context.ObjectStateManager.GetObjectStateEntries(

EntityState.Modified | EntityState.Deleted)

where

e.IsRelationship == false &&

typeof(fromType).IsAssignableFrom(e.Entity.GetType())

select e;

This also ensures that we only get results for entity types that are of type “fromType”, or inherit from it. Arguably this would be more efficient if it looped through all changes looking for the audit configuration for each type as it went... or there may be other desired behaviours for inheritance heirarchies – but that is out of scope for this post, and it should be easy for you to see how you might change it.

Next, we loop through all of our results;

foreach (ObjectStateEntry item in entities)

{

toType auditRecord =

mapping(item.OriginalValues, item.State);

addToContext(auditRecord);

}

... and it is just a case of using the supplied helper functions to map the OriginalValues IDataRecord to a ProductHistory (in this example), and then add it to the ObjectContext. Of course these helper functions are in the form of the lambdas we passed to our call to the Audit extension method.

Retrieving Point-in-Time Products

This design is based on the assumption that I will rarely need to retrieve audited data, as it requires some calculations and slightly complex SQL to complete. This is fine, but if you were retrieving audit data very frequently I would consider doing some further testing, with a view to optimising the approach.

So how do I get a Product record as it looked at 3pm on the 20^th April 2009, for example?

Simply put, to get the correct data fields for a given time I need to find the first ProductHistory record after the specified date & time. If there are none, I need the values on the Product record (as this means no changes had been made to the record at that point in time). Let’s break that down.

First I need to get the first record from ProductHistory for products that were created before the specified date & time, but were changed after it...

SELECT

Id, [Description], Price, CreatedBy, CreatedDate

FROM

(

SELECT

Id, [Description], Price, CreatedBy, CreatedDate,

ROW_NUMBER() OVER(ORDER BY ChangedDate) AS DateOrder

FROM

ProductHistory

WHERE

CreatedDate <= @PointInTime

AND ChangedDate > @PointInTime

AND Id = @Id

) A

WHERE

A.DateOrder = 1

Next, I need all Products that have no ProductHistory records that fall into the above category, but were created before the specified date & time;

SELECT

Id, [Description], Price, CreatedBy, CreatedDate

FROM

Product

WHERE

NOT EXISTS (

SELECT * FROM ProductHistory ph

WHERE ph.Id = Product.Id AND

CreatedDate <= @PointInTime

AND ChangedDate > @PointInTime)

AND CreatedDate <= @PointInTime

AND Id = @Id

Put these two query sections together using a UNION statement and we have the content of my [GetPointInTimeProduct] Stored Procedure (see the code download), which allows us to find the exact state of a Product given its identifier and a specific date and time. I’ve brought it into my model using a Function Import.

The Good and the Bad

I hope all that has made sense – as usual I’ve included the code as a download (standard disclaimers apply). There are of course some pros and cons to my approach – some that spring to mind are below. If you have comments on better ways to do this, or thoughts about my approach, feel free to chip in!

· We can easily record the username of whoever made the changes as we can deduce it in our C# code; even in a trusted subsystem model.

· The approach is easy to configure in a compile-time checked fashion. It doesn’t rely on possibly flawed convention (e.g. always store audit records in tables ending “Audit”). It should also therefore be easy to refactor to suit your needs.

· Mappings between entity and audit records are explicit, so it does not dictate too much about how you would do this. You could store completely different audit data to me if you’d like.

· The audit records participate in the ObjectContext’s Unit of Work; that is, they are saved in a batch of SQL statements with the data changes. It also means that due to the ordering and batching of updates that the likelihood of locks and deadlocks is arguably reduced when compared to some other approaches.

· The configuration of auditing for each type independently keeps the code clear, but introduces some inefficiency (such as many subscribers to the SavingChanges event, and many loops through the ObjectStateEntry collection).

· The data model, although simple, needs explanation. The history records may not work quite how a newcomer would assume.

· The complexity of the point-in-time SQL is slightly higher than I would like.

· [Edit] It is important to use some kind of concurrency checking to ensure that multiple audit records are not written by different users... I tend to use a timestamp with “Fixed” concurrency.

· [Edit] The biggest drawback of the code so far is that it doesn’t record changes to relationships... so it is suited well to resource data tables, but not so well to those involved in a complex model.

On balance, overall I like the approach. What do you think?

Note: Regarding the edits, I just reread this and realised I’d oversimplified for this post and missed two important points from my notes... that’ll teach me to type up blog posts late at night! Sorry.

Auditing Data Changes in the Entity Framework: Part 1

simonince — Fri, 17 Apr 2009 16:43:00 GMT

If you’ve read my last post on Types of Auditing, you should be primed for this one; I’m looking at a way to do Data Operation Auditing in the Entity Framework. That is, I want to track who changes which tables & columns, when, and in what way.

Why does this specifically apply to the Entity Framework? Well there are three main reasons;

1. A lot of DBAs don’t like triggers in their database. In fact, I don’t like triggers in my database! I find them to be more difficult to manage, and often overlooked during defect investigation. I also don’t like the cascading updates that can occur as a result of triggers.

2. In a Trusted Subsystem model, which is often my preferred security configuration, the database is accessed as a service account. This means that triggers and SQL statements have no automatic concept of who is performing the data changes, as the user’s security context has not been delegated through to the data tier.

3. People are starting to use the Entity Framework, so I’m keen to maximise the benefits of using it. It also places some constraints on an auditing approach – for example, out of the box, the Entity Framework cannot pass values that are not contained in an entity to the database during DML operations... so username, for example, can’t be passed down to a Stored Procedure when saving entity changes.

In this post I’ll cover the basic solution concept I’ve used, and some of the problems I encountered. In the next post I’ll detail my final solution. This isn’t some poor attempt at leaving a post on a cliff-hanger to keep you coming back – I just want to keep each post short enough to digest easily!

Basic Design

The Entity Framework extensibility point that seems to fit my requirements best is the SavingChanges event on an ObjectContext. This fires when changes are just about to be saved (i.e. before the SQL is actually generated). Therefore this is a great opportunity for us to check what has been changed, and generate our audit records. I can even add more entities (containing audit information) at this point to the ObjectContext and they’ll get saved too.

To find what has changed we can use ObjectStateManager.GetObjectStateEntries – this is accessible via the ObjectContext’s ObjectStateManager property, and gives us access to information about the original and current values of each property on an entity, whether it has been Added, Modified, or Deleted, and some other useful metadata.

To store data audit records I created a copy of the table I’m auditing, and added some metadata fields. For example, if I have a Product table, I would add a ProductHistory table – something like the following;

Note that the Primary Key in Product is the Id field – but in ProductHistory a basic counter field is the Primary Key (AuditId), and Id is a reference to the Id field in the Product table. This is because we’re expecting multiple history records per product.

I’ve also added metadata about who changed the record, when, and in what way (i.e. Modified, Deleted, etc).

None of this is particularly surprising – it is a pretty standard way to record audit information, in the form of data “deltas”.

Gotchas

I did try a number of different approaches before I settled on one, and this was due to some interesting gotchas...

1: Don’t Create a Foreign Key

You may have noticed there’s no relationship in my data model. That’s intentional; if I delete a Product, I still want to keep my ProductHistory records for that Product, and that means I can’t have ProductHistory.Id related to Product.Id – as referential integrity would break.

2: Can’t Save Added Records

The SavingChanges event fires before changes are saved to the database. In my data model I’m using database-generated keys (i.e. IDENTITY(1,1) fields). This means only after the changes have been saved to the database is the value of the key available in our in-memory representation of the database – our ObjectContext.

This means that if we detect an entity has been added to the ObjectContext, I might want to add a record to the history table... but I cannot get the value to put in the Id field, because it hasn’t been created yet. To get around this, I could allow SaveChanges to complete, and then create my audit records, and call SaveChanges again... but that feels like a hack and would lead to other compromises, so I avoided it.

3: Don’t Create a Conceptual Association

To get around point (2), I thought I had a solution. The Entity Framework automatically manages this key-relationship problem for us using navigation properties. If I just add an association in my CSDL linking Product and ProductHistory, this would mean ProductHistory records would be inserted after Product records, and the framework will fix up the Id values for me.

But of course, as soon as we come to deleting records we come across the same issue as point (1), but within the ObjectContext rather than the database – it will not permit us to add a ProductHistory record with an Id that refers to a Product record that we’re about to delete in the same transaction. In fact, a bit of hacking and I managed to create some pretty serious bugs in my code trying to work around this!

There are possibly some solutions here that involve Conditional Mappings, QueryViews, DefiningQuery, database-level updateable Views, or more. But each one felt wrong to me – they had maintenance implications, could create quite brittle code, would be overwritten when using the “update from database” menu option, or worse. If you have a good solution, let me know – but I avoided them.

4: Avoid Passing Strings Around

One of the things I love about the Entity Framework is that a lot of code becomes strongly typed, and is therefore checked at compile time. In a few places this isn’t necessarily the case – such as accessing DbDataRecord field values, or calling ObjectContext.AddObject.

To address this I took a tip from Stuart Leeks – check out this great post on how lambda expressions can be used to bring compile-time checking to your application. I added similar code to my solution to address slightly different areas.

5: Avoid Auditing by Convention

Convention over configuration can be a valid approach in some circumstances. For a general blog post and flexible solution, though, it felt wrong – I wanted to explicitly select which entities are audited to which tables, through compile-time checked, strongly typed, clear code.

You might not like my approach to this – but I chose it to ensure it was clear what I was doing. Therefore it’s easy for you to put your own flavour on it if needed; I hope you agree!

Solution

So how did I implement it in the end? Watch this space and I’ll post my code shortly.

Types of Auditing

simonince — Thu, 16 Apr 2009 15:00:00 GMT

Auditing almost means something different to everyone I speak to... just a few things people mean when they say “our system has auditing” to me are;

Business Operation Auditing. This is recording when a business operation is performed, such as “Hire Employee”. They usually signify that a whole lot of work has been done across the system’s databases and services.

Data Operation Auditing. This refers to tracking individual changes to database tables. For example, this might involve recording that a user performed an operation that caused the Employee table’s row with identifier “5” to have the “Hired” flag set to “true”.

System Auditing. This is usually what I would call tracing or instrumentation rather than auditing – i.e. outputting detailed information about the running of the system. This often uses the TraceSource class, performance counters, or might be custom statistics recorded in a database/file about long running tasks – it can take many forms.

I’m sure this list is by no means complete – it isn’t intended to be. It’s just designed to make you think “what do they mean by auditing?” when you’re reading the requirements for the shiny new system you’re about to write. What is it you need to achieve?

I’m also hopeful that when I visit you one day in the future and say “what do you mean by auditing?”... you’ll realise I’m just trying to make sure we’re talking about the same thing J

My next couple of posts will cover one approach to Data Operation Auditing in the Entity Framework, so stay tuned...

* You might think there’s a rather large gap in the above... what about Security Auditing? Well, I’ve left it out deliberately as I see that as more of a vertical slice compared to the horizontal list that I’ve come up with. In other words, it’s just a slightly different way to different cut it – I’d categorise Security Auditing as either System or Business Operation auditing, depending upon your requirements, the technology you’re using for authentication and authorisation, and how you choose to implement it.

Performance, Scalability, and Stability

simonince — Tue, 14 Apr 2009 17:56:00 GMT

I spend quite a lot of time nowadays advising on architecting applications to be scalable, recommending performance testing approaches and processes, and being called in to help an organisation remediate problems with an existing application. It’s one of my favourite sides to the job.

But in spite of spending so much time on this, I still catch myself apparently confusing three terms – Performance, Scalability, and Stability. Why? Well, I’m not really confusing them – it’s just that they’re so closely related I don’t think you can consider any one of them in isolation. Let’s consider my (unofficial and buzzword-free) definition of each;

Performance is about the time taken to execute tasks. The quicker a task completes, the better the application is performing.

Scalability is about the number of tasks a system can execute at the same time. This usually maps almost directly to the number of concurrent users that an application can support.

Stability is about how many failures an application exhibits; whether that is manifested as unexpected or unintended behaviour, users receiving errors, or a catastrophic failure that brings a system down. The fewer failures that are observed the more stable an application is.

I’ve over-simplified a little and left out some other concepts, but I think it serves to illustrate the point. A nice way to visualise this is in the form of a triangle;

This visualisation helps to emphasise that each quality can affect the other. The word “quality” is key here – each of these is basically a “Quality Attribute” (or “Quality of Service” requirement) – although Stability arguably covers a number of attributes. If this term is unfamiliar to you, check out the patterns & practices Application Architecture Guide 2.0, in section II (Design), under Quality Attributes.

It is easy to say they affect each-other, but how exactly does this happen? Let’s consider them one at a time;

Poor Performance affects Scalability

Slow performance means tasks take longer. If they take longer, they are more likely to overlap when multiple users do them at the same time. Therefore improving performance reduces the likelihood of contention for shared resources (such as a database), and therefore increases scalability.

Poor Performance affects Stability

Slow performance means tasks take longer. If they take longer, they are more likely to overlap when multiple users do them at the same time. This is likely to lead to more frequent locks, deadlocks, and resource contention, which is more likely to lead to errors and stability problems. It also means that poor programming practices (such as race conditions and incorrectly shared state) are more likely to be spotted.

Poor Scalability affects Performance

Poor scalability means an application supports fewer concurrent tasks. Therefore tasks may have to queue up if there is no capacity to process them. This usually reduces both perceived and actual performance.

Poor Scalability affects Stability

Poor scalability means an application has two options when demand exceeds capacity; queue requests, or reject requests. Rejecting requests results in an error or unexpected behaviour, and is my definition of instability. Queuing requests leads to reduced performance, but also puts demands on sharing resources, connections, memory, CPU, and more. All this increased demand inevitably leads to further stability issues eventually.

Poor Stability affects Performance

Poor stability generally means three things. Firstly, errors must be propagated throughout the system. .NET exceptions do carry overheads and therefore increase CPU and memory requirements of the application.

Secondly, partial success and partial failure must be handled, usually with database rollbacks but also with manual compensation logic. This inevitably leads to additional resource requirements on the system – whether that is the database server dealing with rollbacks, or the additional processing of compensation logic.

Thirdly, when unexpected results are received the user is likely to retry them. This means the load on parts of an application can literally double (or worse) during times of instability.

The first two of these impacts the actual physical performance of the system, but the third also affects the perceived performance from the user’s point of view.

Poor Stability affects Scalability

Poor stability means that additional resources are used, and users are likely to retry tasks. This means more tasks are being performed by fewer users, and therefore ultimately less users can be supported concurrently as a result.

... wow, what should I do?!

After all that, you might be thinking “wow, what should I do?”

The answer is that you should always consider performance, scalability, and stability when architecting, building, and testing your applications. To be precise, you should;

1. Set objectives for each Quality Attribute (including those I’ve not mentioned in this post).

2. Be aware of how each Quality Attribute affects the others.

3. Invest time, hardware, and money in testing that your application meets the objectives you set for each Quality Attribute.

4. Fix issues that you find through re-architecting, code refactoring or tuning, or by tuning your deployment configuration and/or environment.

5. Do all this as early as possible, as it is considerably cheaper to fix performance, scalability, and stability issues early in the development lifecycle.