Simon Ince's Blog : Architecture

Do you need web client development help?

simonince — Tue, 21 Jul 2009 15:01:00 GMT

Then tell us what you need! Blaine has blogged a link to a survey that patterns & practices will use to help them understand what challenges that are affecting you the most... so that they can choose the right areas to focus effort in providing guidance.

If you’ve got a couple of minutes spare head over to the survey...

What tasks or areas are difficult? What consumes the most time? What is it that causes you post-deployment headaches? What parts of the process are tricky? What don't you do because it is hard?

Monitor your ASP.NET Cache API Behaviour

simonince — Mon, 20 Jul 2009 14:36:00 GMT

Recently I was working on diagnosing a performance issue with a customer’s web site with a colleague (this is one of our favourite engagement types so if you need some help let me know J), and we found that items were being trimmed very regularly from the ASP.NET Cache, causing excessive back-end work, and in turn reduced scalability.

* a “cache trim” is when ASP.NET looks for unused (based on a LRU algorithm) cache entries and deletes them.

So if you’re using the ASP.NET Cache API (or indeed any cache provider) to store custom application data, the moral of the story is to make sure you size your cache appropriately, and monitor it during test and live to see how it behaves.

It is worth knowing that an un-configured ASP.NET Cache takes a memory limit of the minimum of either 60% of physical RAM, or a fixed size RAM. This fixed size is different depending upon your processor architecture - see Thomas Marquardt's post here for some more detail. The idea is that when the limits imposed by configuration are approached, the cache will start trimming.

I’ve attached a very simple ASP.NET application that you can play with to study the cache behaviour while you’re reading the next section. Host it in IIS for the best results.

Setting Cache Limits

It is possible to set limits on how much memory an ASP.NET Application Pool can consume in IIS (see here), and you can also configure behaviour using settings on the cache web.config element. It is important to understand exactly what these mean;

· privateBytesLimit (in web.config, on the cache element): This is the maximum number of bytes that the host process is permitted to consume before the cache is trimmed. In other words, it is not the memory to dedicate to the cache, nor a limit to the memory that the worker process can use... but rather the total memory usage by the w3wp process (in IIS7) at which cache trimming will occur.

· percentagePhysicalMemoryUsedLimit (in web.config, on the cache element): This is the maximum percentage of RAM that the machine can use before the cache is trimmed. So on a 4gb machine with this set to 80%, the cache would be trimmed if Task Manager reported over 3.2gb (i.e. 80% of 4gb) was in use. Note that this is not the percentage memory used by ASP.NET or caching – it is the memory used machine-wide.

· Private Memory Limit (in IIS7 manager, as an advanced property of the App Pool): This is the maximum number of private bytes a worker process can consume before it is recycled (which will of course also empty the cache). If you set this limit lower than privateBytesLimit in web.config, you’ll always get a recycle before a cache trim... which sounds unlikely to be what you would want.

Things to Watch

The remaining content of this post itemises things you should consider monitoring to assess your cache’s performance. Some of it is obvious – but it is surprising how easy it is to overlook the obvious when your boss is breathing down your neck about your slow application J

Overall Application Performance

The best indicator that something might be going wrong is if your application is performing badly. Or maybe it was performing fine, but you’ve changed something or gotten more users, and suddenly it’s not so fast. Make sure you collect timing data in your IIS logs, and use LogParser to parse and analyse them. Also use standard performance counters from ASP.NET, WCF, and whatever other technologies you’re using. Get a feel for where and when things are slow or failing. This isn’t specific to caching, but it is an essential first step!

Hits on the resource you’re caching data from

Do some simple maths on the number of hits your back-end resource is getting. For example, if you have a “GetCities” web service, the results of which are cached for 2 hours, and you have 4 front-end load balanced web servers each with their own cache, you should expect a maximum of 4 hits to that web service every 2 hours. If you’re seeing more than that, alarm bells should be ringing.

Cache Entries

There are some great performance counters for the ASP.NET Cache API, so use those to monitor the state of the cache. Specifically the “ASP.NET Apps\Cache API Entries” counter is the number of items that are in the ASP.NET Cache right now. It should be broadly stable over longer periods with approximately the same load. If you cache an item per user, per region, or anything similar, be aware that this can dramatically affect your cache behaviour and memory consumption... in which case Cache Entries can be quite revealing.

Cache Trims

The “ASP.NET Apps\Cache API Trims” counter is the number of cache items that have been removed due to a memory limit being hit (i.e. they were trimmed). Obviously this should ideally be very low or zero under normal operation. Too many trims could indicate you need to revisit your caching strategy, or manually configure cache memory limits.

Cache Turnover Rate

The “ASP.NET Apps\Cache API Turnover Rate” counter shows the number of newly added and removed/expired/trimmed cache items per second. A target for this number depends on the behaviour of your application, but generally it should be fairly low – ideally items should be cached for quite some time before they expire or are removed. A high turnover rate could imply frequent trims, frequent explicit removals in application code, dependencies firing frequently (e.g. a SqlCacheDependency), or a sudden increase in user load (if, for example, you’re caching something per user).

Private Bytes

Monitoring the “Process\Private Bytes” counter for the w3wp process (assuming you’re using IIS7) tells you how much memory IIS is using for your ASP.NET application. A cache trim is likely to show up as a sudden drop in consumed bytes, and equally you should be able to see how close it is to memory limits for your configuration.

Worker Process Restarts

It is initially a little easy to confuse the cause of a drop in consumed Private Bytes between heavy cache trimming and an Application Pool recycle, so you should also watch the “ASP.NET\Worker Process Restarts” performance counter to ensure you know which happened.

Cache Removed Reasons

When you add items to the ASP.NET cache you can optionally specify a CacheItemRemovedCallback. This means that when the item is removed from the cache the call-back you’ve specified is called, passing in a CacheItemRemovedReason. This is great for adding some debugging instrumentation – if your item was trimmed due to memory pressure then the reason will be “underused”. Other reasons are Expired (you specified a time limit), Removed (you called Cache.Remove) and DependencyChanged (a file, SQL table, or other dependency setup with a CacheDependency was fired).

If you decide to add some logging using this approach I’d recommend adding a switch that enables or disables setting a call-back, as there is a small overhead in dealing with it.

Cache Memory Percentages

There are two slightly mysterious sounding cache counters too. I must admit, at first it took me a few minutes to get my head around exactly what these counters meant.

The first is “ASP.NET Apps\Cache % Machine Memory Limit Used”. This is literally the current physical memory usage by the whole machine as a percentage of the configured (or default) maximum physical memory usage. What?! Well, if you have edited the “percentagePhysicalMemoryUsedLimit” setting to 60%, this means your machine can use up to 60% of its physical memory before cache trimming occurs (not a very realistic example I know!). This counter reports the current usage as a percentage of the maximum... so if your machine is using 40% of available physical RAM, and the limit is 60%, this counter would report 66% (40 divided by 60, multiplied by 100 to get a percentage). It is important to note that this is memory consumed across the whole machine – not just by ASP.NET.

The second is “ASP.NET Apps\Cache % Process Memory Limit Used”. This is the total Private Bytes in use by the current ASP.NET process as a percentage of the limit specified in “privateBytesLimit”. So if you set the limit at 400mb, and your process is currently using 350mb, that would be reported as 87.5% (350 divided by 400, multiplied by 100 to get a percentage).

If either of these counters hits 100% ASP.NET will almost immediately trim 50% of your cache by picking the least recently used items... so obviously you don’t want to be hitting these limits often.

Conclusion

Phew. Well I hope that’s useful stuff. The best advice I can give is to do some real sums. Many customers cache data items per user, plus huge lists of reference data that can grow over time. The end result can be caches that are crippled over a fixed concurrent user level, or take a long time to reload large reference data sets when they have been trimmed. It is more than possible to do some rough maths to work out how much memory your application is using for the cache, and how this changes according to user numbers, regions, languages, locations, roles, user types, base offices, or other parameters – and the results can be very illuminating.

Finally, remember that you should always test using hardware, configuration, and simulated user loads that are as close to live as you can possibly afford, as this gives you the best possible chance of catching problems early.

Enforcing Unobtrusive JavaScript

simonince — Fri, 10 Jul 2009 21:07:00 GMT

Unobtrusive JavaScript is a concept that’s been around for quite a while, but it’s now finally starting to make its way into the mainstream. The basic idea is that you should separate your JavaScript from your HTML – so instead of;

<a href="Default.aspx" onclick="alert('Sorry, that is unavailable'); return false;" >Click Here</a>

You should have something like this;

<a id="ClickHereLink" href="Default.aspx">Click Here</a>

... with some matching JavaScript in another file somewhere;

function pageLoad() {

var link = $get('ClickHereLink');

$addHandler(link, 'click', preventClick);

}

function preventClick() {

alert('Sorry, that is unavailable');

return false;

}

I’m not here to tell you why this is so great, but the best discussion and walkthrough of issues I’ve found is this article on onlinetools.org, so check it out if you’re interested.

How do I enforce it?

Recently I found myself cursing some code I’d written a long time ago, when I found it littered with onclick and onload handlers. It was frankly a pain to work out how it works. It was at this point that I thought “I should write something to shout at me when I write don’t write unobtrusive JavaScript”.

Coincidentally I have also been doing some work on HTML parsing, and I have to say I’ve found the XPath-like features in the HTML Agility Pack to be incredibly useful. If you’ve never used it I’d recommend taking a look.

It occurred to me that this was the perfect tool to do what I wanted – so I set about creating a simple MSBuild task that uses the HTML Agility Pack to scan ASPX and ASCX files for Obtrusive JavaScript – basically a form of static code analysis. If you’re interested in customising MSBuild, check this out.

The implementation is very simple;

· It is derived from Microsoft.Build.Utilities.Task.

· There is a property that is an array of ITaskItems, flagged with a [Required] attribute, as a way to pass in a list of files to check.

· There’s a hard-coded (for now) list of validation XPath expressions, with matching explanations.

· In the Execute method override, it loops through all the files.

· For each file, it opens the HTML and applies every XPath to it.

· If there are any matches for the XPath, it creates a build failure.

The key to success here is the simplicity with which the HTML Agility Pack lets me use XPath to find elements. The following is the basics of my code;

HtmlDocument doc = new HtmlDocument();

doc.Load(path);

HtmlNodeCollection results =

doc.DocumentNode.SelectNodes(

validator.SearchXPath);

Easy huh? The XPath expressions themselves mostly look like this;

//*[@onkeypress]

This will match any HTML element that has an onkeypress attribute... which of course means that the developer has specified a JavaScript event handler in the HTML, and broken the rules of unobtrusivitiy (hey, if that’s not a word I’m having first dibs on inventing it J).

Remembering that we’re actually scanning ASP.NET mark-up, not pure HTML, we can also check for ASP.NET attributes that actually create JavaScript event handlers in HTML too;

//*[@onclientclick]

Again, this is not difficult. The tricky one appears when you realise that it is valid to have OnClick in ASP.NET code! Of course, an <asp:button> can have an OnClick attribute referring to the server-side event handler. I wondered about saying “match any with an onclick that don’t have a runat attribute”, but this isn’t quite accurate (HtmlControls can have runat=”server” too remember).

In the end I went for explicitly excluding the ASP.NET controls that I knew were OK to use OnClick;

//*[@onclick and not(

name()='asp:button' or name()='asp:bulletedlist' or

name()='asp:imagebutton' or name()='asp:imagemap' or

name()='asp:linkbutton')]

Finally, I wanted to check that there was no JavaScript inside the page HTML itself... even between <script> tags. In other words, the only valid <script> tags should reference an external file, as follows;

[Of course, you could also use the <asp:ScriptManager> control if you wanted to; in fact I’d prefer it]

To achieve this I check that there is nothing other than whitespace between the start and end script tags;

//script[string-length(normalize-space(text()))>1]

... and that’s it! As usual, the source code for this MSBuild Task is attached to this post, along with a demonstration Web App.

Wiring up the Task

Now we have a simple MSBuild task, all that is left to do is add it to a build. I created a new ASP.NET Web Application, right clicked the project file and chose “unload”, and then edited the XML manually.

The only bit I needed to add was as follows;

<UsingTask

AssemblyFile="..\JsBuildTasks.dll"

TaskName="ValidateUnobtrusive" />

<ValidateUnobtrusive

SourceFiles="@(Content)" />

</Target>

First I import my task (I’ve copied a Release build of the assembly to the parent directory above my Web App project). Next, I override the BeforeBuild target, and add an instance of my ValidateUnobtrusive task. I’ve passed in an expansion of all the Content files in the project. The task filters for *.aspx and *.ascx at the moment – if you wanted to add *.html that would be trivial.

Save the file, and right click it to choose Reload Project. At this point Visual Studio 2008 warns you about a customised MSBuild project file – just choose “Load Project Normally” and you’re away.

From this point on, whenever you build a project containing Obtrusive JavaScript you’ll see errors like these;

Error Use of the 'onclick' JavaScript event on 'a' tag in markup causes Obtrusive Javascript. You should attach this handler in script instead.

Error Failed on 'script' tag . A JavaScript region containing something other than white space was found. You should link to related JavaScript resources files rather than embed script in a page.

Conclusion

I think this works really well so far (although I’m at the early stages of trialling it); if you try it do let me know how you get on. One of the particularly cool things (that I haven’t tried yet as I’m working isolated on my laptop most of the time right now, and I’m short of time) is that this could be wired up into a Continuous Integration build, under Team Build – I guess that would be a good plan if you didn’t like running the validation as part of your local build, or if it starts to perform badly due to the complexity or size of your pages.

I also think it raises other questions and possibilities. For example; how about some static analysis of “.js” files themselves? Could this help check for Directly Accessible Scripts, for example?

Using ASP.NET Dynamic Data in an existing Web Site

simonince — Fri, 01 May 2009 15:39:00 GMT

One of the tasks I used to hate when building a new system was adding admin pages to allow simple create/update functionality for lookup data. For example, maintaining a simple list of Products. It takes ages to write, and is very boring – even if you write templates to generate code it consumes a lot of time for such a commonly repeated and simple task.

ASP.NET Dynamic Data has now arrived, which gives a great way to “scaffold” CRUD pages over an Entity Framework Object Context (or other sources). But what if I have a web site that is mostly a line-of-business system, with complex business logic and custom UI, and I need to administer a small set of tables too?

Well, you can add Dynamic Data to a portion of your existing web site! This means you no longer need to write your own administration pages.

* Before you start off on this merry path, read this to make sure the relevant sections of your Entity Framework model are supported by Dynamic Data... for example, Complex Types are not.

How?

There’s a walkthrough that covers adding Dynamic Data to an existing web site here, so give that a read and you’ll be 80% of the way there. [Note: I also had to add a reference to System.Web.Entity at the end of the walkthrough to get it all working]

Next, make sure you have set ScaffoldAllTables to false – the last thing you want to do is expose your non-lookup data for direct editing by admin staff. All that should be exposed is a few key tables of static data. So, in Global.asax, make sure your data model registration looks like this;

model.RegisterContext(

typeof(OrderDbEntities),

new ContextConfiguration()

{ ScaffoldAllTables = false });

But how does the user get to this administration functionality? I’d recommend putting it under a virtual subfolder in your web site – perhaps named “Admin”. This is really just a route (read up on the new ASP.NET Routing features if this is unfamiliar) and needn’t really exist as a physical folder. So in Global.asax, we set it up as follows;

routes.Add(new DynamicDataRoute("Admin/{table}/{action}.aspx")

{

Constraints = new RouteValueDictionary(

new { action = "List|Details|Edit|Insert" }),

Model = model

});

Note that my route begins with “Admin/...”, which means the user will browse to URLs as follows to get to the admin pages;

http://{add address here}/myfakewebsite/Admin/Product/List.aspx

You can of course link to these pages from elsewhere within your site to save the user typing them in.

At the moment, we’re not scaffolding any of our tables, so none of these URLs will work yet. The next step, then, is to pick a table to expose for administration. I’m going to use a Product table that has ID, Description, Price and CreatedBy/CreatedDate fields. In my EF model the entity also has an OrderItems navigation property which allows me to get all Order Item records that are an instance of an order for the current product...

Due to the way EF supports partial classes, I can tell the Dynamic Data runtime to scaffold the Product table using an attribute (from the System.ComponentModel.DataAnnotations namespace & assembly) on a partial class. I added a C# class to the Web Site with the same namespace as the Entity Framework model, and pasted in the following code;

[ScaffoldTable(true)]

public partial class Product

{

}

Immediately this attribute is added the Product table is available for editing. There are a few problems when I browse to the new page though;

1. It is displaying the link to Order Items even though it isn’t editable (because Order Item isn’t scaffolded).

2. I can edit the columns I’m using for Audit – CreatedBy and CreatedDate. Not good.

It turns out changing this is quite easy. Using some additional attributes and a metadata class as follows customises the model sufficiently (see here for more info on this metadata class approach);

[MetadataType(typeof(ProductMetadata))]

[DisplayColumn("Description")]

[ScaffoldTable(true)]

public partial class Product

{

}

public class ProductMetadata

{

[UIHint("TextReadOnly")]

public string CreatedBy;

[UIHint("TextReadOnly")]

public string CreatedDate;

[ScaffoldColumn(false)]

public EntityCollection<OrderItem> OrderItem;

}

I’ve now instructed the runtime not to display the Order Item navigation property, and to use the Description field should I display Products in a list somewhere (as part of a foreign key drop-down, for example).

I’ve also asked it to use the “TextReadOnly” user control to render my CreatedDate and CreatedBy fields instead of the standard Text control, by placing a [UIHint] attribute on two metadata properties. To create the TextReadOnly control I copied-and-pasted the “Text.ascx” control twice, and renamed the copies to “TextReadOnly.ascx” and “TextReadOnly_Edit.ascx”. This control will now be used whenever the CreatedDate and CreatedBy fields are displayed (note I’ve cheated here – CreatedDate is really a date but as I’m happy with the default ToString display mechanism for it I didn’t bother creating a DateTime ReadOnly control).

As you can see, these user controls are available for you to edit under the DynamicData/FieldTemplates folder that was copied in as part of the original walkthrough. If you want to change this folder to be somewhere else, check out this article.

Finally, we probably want to limit access to the admin pages to a limited set of privileged users. The easiest way to do this is using standard ASP.NET authorisation, by adding a location and authorisation element to web.config;

<system.web>

</authorization>

</system.web>

</location>

Next steps...

There’s plenty more you might like to do, so I would recommend looking further into the features that Dynamic Data provides.

One that is particularly likely to be of interest is customising validation of data that is maintained... a good starting point is the ValidationAttribute.

You probably also want to change Site.master to better reflect your site (or move it, rename it, and update the templates under DynamicData/PageTemplates to use the new one). As long as you have a ScriptManager and the required Content Placeholder on there, it should work great. You can obviously make use of your own CSS too, or even go as far as changing all the field templates.

If you’d like a general overview of how Dynamic Data fits together, have a read of the Infrastructure Overview.

Finally, remember that I’m basing mine on an Entity Framework model, so I could customise the model too – for example, hooking up to the SavingChanges event to perform processing and/or validation on the Product table before it is saved to the database.

I hope all that is useful, and saves you time and boredom for your next system... enjoy!

Auditing Data Changes in the Entity Framework: Part 2

simonince — Mon, 20 Apr 2009 17:02:00 GMT

In my previous post I described the basic concept behind my auditing approach using the Entity Framework, and covered some of the problems I encountered. This post focuses on the solution I went with; do feel free to comment if you have any thoughts.

Audit Lifecycle

To get to the root of my requirements I wrote down the lifecycle of a piece of data, and what needed to be tracked against it.

Action	Audit Requirements
Record Added	The date it was created, by whom, and what the original values are.
Record Modified * repeated many times	The date it was changed, by whom, and which values were changed.
Record Deleted	The date it was deleted, and by whom.

The problem is that creating an audit record is tricky based on point (2) in my previous post. Therefore, I chose to record the original values when each change is made. This means I actually track the following;

Action	Audit Requirements
Record Added	The date it was created, and by whom.
Record Modified * repeated many times	The date it was changed, by who, and what the pre-change values were.
Record Deleted	The date it was deleted, by who, and what the pre-delete values were.

This has effectively flipped the approach on its head, but I can still get a point-in-time view of the data whenever I need it. Therefore my data model becomes something like this;

When a new Product is added, the CreatedBy and CreatedDate fields are recorded on the Product entity. When changes are made, the previous unchanged version of the entity is copied to the ProductHistory table, and then the new Product entity is saved. When a Product is deleted, it is removed from the Product table and a copy of its last values is saved to ProductHistory.

This means my ChangedDate field on ProductHistory could arguably be described better as “ExpiredDate”, as it is the date and time that the data on that record became out of date.

Enabling Auditing on the ObjectContext

I described in my previous post how I would use the SavingChanges event to create audit records. To wire this up, I’ve created an extension method that can be called on an ObjectContext;

public static void Audit<fromType, toType>(

this ObjectContext context,

Func<IDataRecord, EntityState, toType> mapping,

Action<toType> addToContext)

{

context.SavingChanges +=

new EventHandler((o, e) =>

CreateAuditRecord<fromType, toType>(

context, mapping, addToContext));

}

Calling this method sets up auditing for one specific type of entity by adding an event handler to the SavingChanges event. The handler invokes a method called CreateAuditRecord.

You might think it would be better to set all audit types up at once, or you might want to configure auditing in the ObjectContext’s OnContextCreated partial method... and that should be easy enough. The point of this code is not to be final, but that it should be easy for you to understand and adapt.

So why have I used two type parameters, a Func<T,U,V> and an Action<T> on the Audit method then? It comes down to points (4) and (5) in my previous post; I wanted auditing to be very explicit about how it behaved, and I wanted to avoid passing string values around to identify field names or entity sets.

To demonstrate this, consider the following example of how I would enable auditing on the Product entity, saving changes to ProductHistory.

using (MyEntities db = new MyEntities())

{

db.Audit<Product, ProductHistory>(

(record, action) => new ProductHistory() {

Id = record.Field<Product, int>(f => f.Id),

Description = record.Field<Product, string>(

f => f.Description),

Price = record.Field<Product, double>(f => f.Price),

CreatedDate = record.Field<Product, DateTime>(

f => f.CreatedDate),

CreatedBy = record.Field<Product, string>(

f => f.CreatedBy),

ChangedBy = "Simon",

ChangedDate = DateTime.Now,

ChangeType = action.ToString() },

(ph) => db.AddToProductHistory(ph));

// TODO: make some changes to Product here...

db.SaveChanges();

}

In my Audit method I specify the source entity (Product) and target audit store entity (ProductHistory) as type parameters. This then enables me to easily create a strongly typed lambda for the first parameter that I know accepts an IDataRecord (“record”) and an EntityState (“action”), returning a ProductHistory record;

(record, action) => new ProductHistory() {

Id = record.Field<Product, int>(f => f.Id), ... <snip>

The purpose of this method is to map a Product data record to a ProductHistory entity. It has an IDataRecord input as I pass the OriginalValues to it, not the full Product entity (remember I’m saving the previous values in my audit table, not the new ones). Therefore this code creates a ProductHistory record, and initialises its properties using object initialiser syntax. If this code became too unwieldy I could easily factor it out into a helper method.

Next, you’ll notice that I’m using another extension method named “Field” that applies to the IDataRecord type. I use this to fetch a property value in a strongly typed way, and to make it easy to get at fields that represent an entity property.

In this case, I’m specifying that I expect the IDataRecord to contain fields that represent Product’s members. I’m also saying the particular property I’m after is an integer, and then I use a lambda to identify the property itself (Id in this case).

The second parameter to my Audit call looks like this;

(ph) => db.AddToProductHistory(ph)

This is because I need to know how to add audit records that are created to the ObjectContext, ensuring they are saved as part of the same SaveChanges transaction as the actual changes we’re tracking. The lambda receives a ProductHistory entity (which is known because of the “toType” type parameter in my call to Audit) which I choose to add to the ObjectContext by calling AddToProductHistory.

For this task I could have used the more general AddObject method available on an ObjectContext – but this needs a string value to identify the entity set, and I want as much compile time checking as possible.

The Mechanics of Auditing

Now you see how we setup auditing, let’s dive into the CreateAuditRecord method to see how it actually creates the audit trail.

First, the method retrieves a list of ObjectStateEntry objects that describe entities that have either been modified or deleted. Remember that my approach doesn’t record anything in the audit table when they are added;

IEnumerable<ObjectStateEntry> entities =

from e in context.ObjectStateManager.GetObjectStateEntries(

EntityState.Modified | EntityState.Deleted)

where

e.IsRelationship == false &&

typeof(fromType).IsAssignableFrom(e.Entity.GetType())

select e;

This also ensures that we only get results for entity types that are of type “fromType”, or inherit from it. Arguably this would be more efficient if it looped through all changes looking for the audit configuration for each type as it went... or there may be other desired behaviours for inheritance heirarchies – but that is out of scope for this post, and it should be easy for you to see how you might change it.

Next, we loop through all of our results;

foreach (ObjectStateEntry item in entities)

{

toType auditRecord =

mapping(item.OriginalValues, item.State);

addToContext(auditRecord);

}

... and it is just a case of using the supplied helper functions to map the OriginalValues IDataRecord to a ProductHistory (in this example), and then add it to the ObjectContext. Of course these helper functions are in the form of the lambdas we passed to our call to the Audit extension method.

Retrieving Point-in-Time Products

This design is based on the assumption that I will rarely need to retrieve audited data, as it requires some calculations and slightly complex SQL to complete. This is fine, but if you were retrieving audit data very frequently I would consider doing some further testing, with a view to optimising the approach.

So how do I get a Product record as it looked at 3pm on the 20^th April 2009, for example?

Simply put, to get the correct data fields for a given time I need to find the first ProductHistory record after the specified date & time. If there are none, I need the values on the Product record (as this means no changes had been made to the record at that point in time). Let’s break that down.

First I need to get the first record from ProductHistory for products that were created before the specified date & time, but were changed after it...

SELECT

Id, [Description], Price, CreatedBy, CreatedDate

FROM

(

SELECT

Id, [Description], Price, CreatedBy, CreatedDate,

ROW_NUMBER() OVER(ORDER BY ChangedDate) AS DateOrder

FROM

ProductHistory

WHERE

CreatedDate <= @PointInTime

AND ChangedDate > @PointInTime

AND Id = @Id

) A

WHERE

A.DateOrder = 1

Next, I need all Products that have no ProductHistory records that fall into the above category, but were created before the specified date & time;

SELECT

Id, [Description], Price, CreatedBy, CreatedDate

FROM

Product

WHERE

NOT EXISTS (

SELECT * FROM ProductHistory ph

WHERE ph.Id = Product.Id AND

CreatedDate <= @PointInTime

AND ChangedDate > @PointInTime)

AND CreatedDate <= @PointInTime

AND Id = @Id

Put these two query sections together using a UNION statement and we have the content of my [GetPointInTimeProduct] Stored Procedure (see the code download), which allows us to find the exact state of a Product given its identifier and a specific date and time. I’ve brought it into my model using a Function Import.

The Good and the Bad

I hope all that has made sense – as usual I’ve included the code as a download (standard disclaimers apply). There are of course some pros and cons to my approach – some that spring to mind are below. If you have comments on better ways to do this, or thoughts about my approach, feel free to chip in!

· We can easily record the username of whoever made the changes as we can deduce it in our C# code; even in a trusted subsystem model.

· The approach is easy to configure in a compile-time checked fashion. It doesn’t rely on possibly flawed convention (e.g. always store audit records in tables ending “Audit”). It should also therefore be easy to refactor to suit your needs.

· Mappings between entity and audit records are explicit, so it does not dictate too much about how you would do this. You could store completely different audit data to me if you’d like.

· The audit records participate in the ObjectContext’s Unit of Work; that is, they are saved in a batch of SQL statements with the data changes. It also means that due to the ordering and batching of updates that the likelihood of locks and deadlocks is arguably reduced when compared to some other approaches.

· The configuration of auditing for each type independently keeps the code clear, but introduces some inefficiency (such as many subscribers to the SavingChanges event, and many loops through the ObjectStateEntry collection).

· The data model, although simple, needs explanation. The history records may not work quite how a newcomer would assume.

· The complexity of the point-in-time SQL is slightly higher than I would like.

· [Edit] It is important to use some kind of concurrency checking to ensure that multiple audit records are not written by different users... I tend to use a timestamp with “Fixed” concurrency.

· [Edit] The biggest drawback of the code so far is that it doesn’t record changes to relationships... so it is suited well to resource data tables, but not so well to those involved in a complex model.

On balance, overall I like the approach. What do you think?

Note: Regarding the edits, I just reread this and realised I’d oversimplified for this post and missed two important points from my notes... that’ll teach me to type up blog posts late at night! Sorry.

Auditing Data Changes in the Entity Framework: Part 1

simonince — Fri, 17 Apr 2009 16:43:00 GMT

If you’ve read my last post on Types of Auditing, you should be primed for this one; I’m looking at a way to do Data Operation Auditing in the Entity Framework. That is, I want to track who changes which tables & columns, when, and in what way.

Why does this specifically apply to the Entity Framework? Well there are three main reasons;

1. A lot of DBAs don’t like triggers in their database. In fact, I don’t like triggers in my database! I find them to be more difficult to manage, and often overlooked during defect investigation. I also don’t like the cascading updates that can occur as a result of triggers.

2. In a Trusted Subsystem model, which is often my preferred security configuration, the database is accessed as a service account. This means that triggers and SQL statements have no automatic concept of who is performing the data changes, as the user’s security context has not been delegated through to the data tier.

3. People are starting to use the Entity Framework, so I’m keen to maximise the benefits of using it. It also places some constraints on an auditing approach – for example, out of the box, the Entity Framework cannot pass values that are not contained in an entity to the database during DML operations... so username, for example, can’t be passed down to a Stored Procedure when saving entity changes.

In this post I’ll cover the basic solution concept I’ve used, and some of the problems I encountered. In the next post I’ll detail my final solution. This isn’t some poor attempt at leaving a post on a cliff-hanger to keep you coming back – I just want to keep each post short enough to digest easily!

Basic Design

The Entity Framework extensibility point that seems to fit my requirements best is the SavingChanges event on an ObjectContext. This fires when changes are just about to be saved (i.e. before the SQL is actually generated). Therefore this is a great opportunity for us to check what has been changed, and generate our audit records. I can even add more entities (containing audit information) at this point to the ObjectContext and they’ll get saved too.

To find what has changed we can use ObjectStateManager.GetObjectStateEntries – this is accessible via the ObjectContext’s ObjectStateManager property, and gives us access to information about the original and current values of each property on an entity, whether it has been Added, Modified, or Deleted, and some other useful metadata.

To store data audit records I created a copy of the table I’m auditing, and added some metadata fields. For example, if I have a Product table, I would add a ProductHistory table – something like the following;

Note that the Primary Key in Product is the Id field – but in ProductHistory a basic counter field is the Primary Key (AuditId), and Id is a reference to the Id field in the Product table. This is because we’re expecting multiple history records per product.

I’ve also added metadata about who changed the record, when, and in what way (i.e. Modified, Deleted, etc).

None of this is particularly surprising – it is a pretty standard way to record audit information, in the form of data “deltas”.

Gotchas

I did try a number of different approaches before I settled on one, and this was due to some interesting gotchas...

1: Don’t Create a Foreign Key

You may have noticed there’s no relationship in my data model. That’s intentional; if I delete a Product, I still want to keep my ProductHistory records for that Product, and that means I can’t have ProductHistory.Id related to Product.Id – as referential integrity would break.

2: Can’t Save Added Records

The SavingChanges event fires before changes are saved to the database. In my data model I’m using database-generated keys (i.e. IDENTITY(1,1) fields). This means only after the changes have been saved to the database is the value of the key available in our in-memory representation of the database – our ObjectContext.

This means that if we detect an entity has been added to the ObjectContext, I might want to add a record to the history table... but I cannot get the value to put in the Id field, because it hasn’t been created yet. To get around this, I could allow SaveChanges to complete, and then create my audit records, and call SaveChanges again... but that feels like a hack and would lead to other compromises, so I avoided it.

3: Don’t Create a Conceptual Association

To get around point (2), I thought I had a solution. The Entity Framework automatically manages this key-relationship problem for us using navigation properties. If I just add an association in my CSDL linking Product and ProductHistory, this would mean ProductHistory records would be inserted after Product records, and the framework will fix up the Id values for me.

But of course, as soon as we come to deleting records we come across the same issue as point (1), but within the ObjectContext rather than the database – it will not permit us to add a ProductHistory record with an Id that refers to a Product record that we’re about to delete in the same transaction. In fact, a bit of hacking and I managed to create some pretty serious bugs in my code trying to work around this!

There are possibly some solutions here that involve Conditional Mappings, QueryViews, DefiningQuery, database-level updateable Views, or more. But each one felt wrong to me – they had maintenance implications, could create quite brittle code, would be overwritten when using the “update from database” menu option, or worse. If you have a good solution, let me know – but I avoided them.

4: Avoid Passing Strings Around

One of the things I love about the Entity Framework is that a lot of code becomes strongly typed, and is therefore checked at compile time. In a few places this isn’t necessarily the case – such as accessing DbDataRecord field values, or calling ObjectContext.AddObject.

To address this I took a tip from Stuart Leeks – check out this great post on how lambda expressions can be used to bring compile-time checking to your application. I added similar code to my solution to address slightly different areas.

5: Avoid Auditing by Convention

Convention over configuration can be a valid approach in some circumstances. For a general blog post and flexible solution, though, it felt wrong – I wanted to explicitly select which entities are audited to which tables, through compile-time checked, strongly typed, clear code.

You might not like my approach to this – but I chose it to ensure it was clear what I was doing. Therefore it’s easy for you to put your own flavour on it if needed; I hope you agree!

Solution

So how did I implement it in the end? Watch this space and I’ll post my code shortly.

Types of Auditing

simonince — Thu, 16 Apr 2009 15:00:00 GMT

Auditing almost means something different to everyone I speak to... just a few things people mean when they say “our system has auditing” to me are;

Business Operation Auditing. This is recording when a business operation is performed, such as “Hire Employee”. They usually signify that a whole lot of work has been done across the system’s databases and services.

Data Operation Auditing. This refers to tracking individual changes to database tables. For example, this might involve recording that a user performed an operation that caused the Employee table’s row with identifier “5” to have the “Hired” flag set to “true”.

System Auditing. This is usually what I would call tracing or instrumentation rather than auditing – i.e. outputting detailed information about the running of the system. This often uses the TraceSource class, performance counters, or might be custom statistics recorded in a database/file about long running tasks – it can take many forms.

I’m sure this list is by no means complete – it isn’t intended to be. It’s just designed to make you think “what do they mean by auditing?” when you’re reading the requirements for the shiny new system you’re about to write. What is it you need to achieve?

I’m also hopeful that when I visit you one day in the future and say “what do you mean by auditing?”... you’ll realise I’m just trying to make sure we’re talking about the same thing J

My next couple of posts will cover one approach to Data Operation Auditing in the Entity Framework, so stay tuned...

* You might think there’s a rather large gap in the above... what about Security Auditing? Well, I’ve left it out deliberately as I see that as more of a vertical slice compared to the horizontal list that I’ve come up with. In other words, it’s just a slightly different way to different cut it – I’d categorise Security Auditing as either System or Business Operation auditing, depending upon your requirements, the technology you’re using for authentication and authorisation, and how you choose to implement it.

Performance, Scalability, and Stability

simonince — Tue, 14 Apr 2009 17:56:00 GMT

I spend quite a lot of time nowadays advising on architecting applications to be scalable, recommending performance testing approaches and processes, and being called in to help an organisation remediate problems with an existing application. It’s one of my favourite sides to the job.

But in spite of spending so much time on this, I still catch myself apparently confusing three terms – Performance, Scalability, and Stability. Why? Well, I’m not really confusing them – it’s just that they’re so closely related I don’t think you can consider any one of them in isolation. Let’s consider my (unofficial and buzzword-free) definition of each;

Performance is about the time taken to execute tasks. The quicker a task completes, the better the application is performing.

Scalability is about the number of tasks a system can execute at the same time. This usually maps almost directly to the number of concurrent users that an application can support.

Stability is about how many failures an application exhibits; whether that is manifested as unexpected or unintended behaviour, users receiving errors, or a catastrophic failure that brings a system down. The fewer failures that are observed the more stable an application is.

I’ve over-simplified a little and left out some other concepts, but I think it serves to illustrate the point. A nice way to visualise this is in the form of a triangle;

This visualisation helps to emphasise that each quality can affect the other. The word “quality” is key here – each of these is basically a “Quality Attribute” (or “Quality of Service” requirement) – although Stability arguably covers a number of attributes. If this term is unfamiliar to you, check out the patterns & practices Application Architecture Guide 2.0, in section II (Design), under Quality Attributes.

It is easy to say they affect each-other, but how exactly does this happen? Let’s consider them one at a time;

Poor Performance affects Scalability

Slow performance means tasks take longer. If they take longer, they are more likely to overlap when multiple users do them at the same time. Therefore improving performance reduces the likelihood of contention for shared resources (such as a database), and therefore increases scalability.

Poor Performance affects Stability

Slow performance means tasks take longer. If they take longer, they are more likely to overlap when multiple users do them at the same time. This is likely to lead to more frequent locks, deadlocks, and resource contention, which is more likely to lead to errors and stability problems. It also means that poor programming practices (such as race conditions and incorrectly shared state) are more likely to be spotted.

Poor Scalability affects Performance

Poor scalability means an application supports fewer concurrent tasks. Therefore tasks may have to queue up if there is no capacity to process them. This usually reduces both perceived and actual performance.

Poor Scalability affects Stability

Poor scalability means an application has two options when demand exceeds capacity; queue requests, or reject requests. Rejecting requests results in an error or unexpected behaviour, and is my definition of instability. Queuing requests leads to reduced performance, but also puts demands on sharing resources, connections, memory, CPU, and more. All this increased demand inevitably leads to further stability issues eventually.

Poor Stability affects Performance

Poor stability generally means three things. Firstly, errors must be propagated throughout the system. .NET exceptions do carry overheads and therefore increase CPU and memory requirements of the application.

Secondly, partial success and partial failure must be handled, usually with database rollbacks but also with manual compensation logic. This inevitably leads to additional resource requirements on the system – whether that is the database server dealing with rollbacks, or the additional processing of compensation logic.

Thirdly, when unexpected results are received the user is likely to retry them. This means the load on parts of an application can literally double (or worse) during times of instability.

The first two of these impacts the actual physical performance of the system, but the third also affects the perceived performance from the user’s point of view.

Poor Stability affects Scalability

Poor stability means that additional resources are used, and users are likely to retry tasks. This means more tasks are being performed by fewer users, and therefore ultimately less users can be supported concurrently as a result.

... wow, what should I do?!

After all that, you might be thinking “wow, what should I do?”

The answer is that you should always consider performance, scalability, and stability when architecting, building, and testing your applications. To be precise, you should;

1. Set objectives for each Quality Attribute (including those I’ve not mentioned in this post).

2. Be aware of how each Quality Attribute affects the others.

3. Invest time, hardware, and money in testing that your application meets the objectives you set for each Quality Attribute.

4. Fix issues that you find through re-architecting, code refactoring or tuning, or by tuning your deployment configuration and/or environment.

5. Do all this as early as possible, as it is considerably cheaper to fix performance, scalability, and stability issues early in the development lifecycle.

Are you using the WCSF?

simonince — Fri, 05 Dec 2008 17:10:00 GMT

Have you been using the Web Client Software Factory? Or have you tried it and not adopted it? Or have you used some of the bundle content but not the whole factory?

Michael Puleio has just started a thread on codeplex requesting feedback on who is using the factory, and talks about it briefly in this post.

Combine that with Blaine’s WCSF Roadmap post and you can see p&p are already thinking about the next version of guidance for ASP.NET applications. Now is the chance to comment on what really matters to you, and what would help you in your day-to-day job.

AJAX Script Patterns: Service Agent

simonince — Fri, 05 Dec 2008 00:45:00 GMT

I’ve been thinking for a while about how people tend to build AJAX applications, as there seems to be something a lot of people have forgotten. This post examines how applying a pattern you probably know very well to AJAX could help – and leaves it to you to decide whether this is useful. Let me know what you think.

Setting the Scene

In your average application architecture, you’ll generally find a “Resource Access” layer that contains Service Agents (also known as Service Gateways) for accessing remote web services. What are these for? Well, I think the MSDN documentation on the pattern sums it up nicely in the Problem statement;

“How do you decouple the details of fulfilling the contract responsibilities defined by the [remote web] service from the rest of your application?”

The answer is of course that you create a class to manage the interaction with your service and hide the inner workings and contract implementations from the rest of your code. This class and any supporting entities make up your Service Agent.

AJAX: Why so Different?

With the growing popularity of AJAX, and the great features available in Microsoft AJAX, more and more web sites use script to call services hosted back on the ASP.NET server to load data, and dynamically update the page with DOM operations.

So how come so many sites have forgotten what we learnt years ago? This script calls a web service – and this web service forms a part of a larger system. It may change – the contracts, the implementation, the name, the location... Consider that these web services may also be shared between AJAX, Silverlight, ASP.NET, WPF, and/or other technologies as clients. So each of these could require tweaks to your shared web service (even if they’re using a different endpoint that uses a different binding, chances are the contracts and implementation are shared).

Microsoft AJAX helps us by auto-generating the web service proxy and related entities for calling a script-enabled web service. But if you change the web service, this changes the generated proxy, and you could find many pages in your web site suddenly need JavaScript changes to work correctly again. We don’t get the rich compile-time support in JavaScript that we get with .NET languages, so you might not find these problems until testing, or worse, live.

Therefore I see two options for handling this;

1. You could create a reusable ASP.NET AJAX server control that wraps up the web service calls and UI manipulation. This means you’ve just got one place to make changes, so is a great solution.

2. If the service is used differently in each place, perhaps it just doesn’t suit being a control. In which case, I propose creating a script-based Service Agent.

A Simple Example

To demonstrate this, I’ve created the bare minimum of a simple WCF service to be called from AJAX;

[ServiceContract(

Namespace =

"http://samples.microsoft.com/jslayering")]

[AspNetCompatibilityRequirements(

RequirementsMode =

AspNetCompatibilityRequirementsMode.Allowed)]

public class PersonService

{

[OperationContract]

public GetPersonResponse GetPerson(

GetPersonRequest request)

{

Implementation

}

[MessageContract]

public class GetPersonRequest

{

[MessageBodyMember]

public int Identifier { get; set; }

}

[MessageContract]

public class GetPersonResponse

{

[MessageBodyMember]

public bool Found { get; set; }

[MessageBodyMember]

public Person Person { get; set; }

}

[DataContract]

public class Person

{

[DataMember]

public int Identifier { get; set; }

[DataMember]

public string Name { get; set; }

[DataMember]

public int Age { get; set; }

}

You can see I’ve also chosen to use Message Contracts... we’ll see why in a moment J

All that happens in this service is the client requests a Person’s details, and they either get Found=true and a Person entity back, or Found=false. In my implementation this is hard-coded and randomised, so excuse the grim code.

Creating a Service Agent

The first thing we need is a JavaScript entity to hold person details in. This is because the Person class above is part of the contract of the web service – so if we change it, the JavaScript entity generated by Microsoft AJAX will change too... so we need to protect the rest of our JavaScript code from this. Let’s see my simple entity;

/// <reference name="MicrosoftAjax.js"/>

/// <reference path="~/PersonService.svc" />

Type.registerNamespace("ServiceAgents");

ServiceAgents.Person = function(id, name, age) {

this._id = id;

this._name = name;

this._age = age;

}

ServiceAgents.Person.prototype = {

get_id: function() {

return this._id;

get_name: function() {

return this._name;

get_age: function() {

return this._age;

}

ServiceAgents.Person.registerClass(

'ServiceAgents.Person');

if (typeof (Sys) !== 'undefined')

Sys.Application.notifyScriptLoaded();

* Note the use of the <reference> tag so I get intellisense in Visual Studio 2008!

This is the simplest of objects, and just stored the three fields found on a Person entity. Next, we need a Service Agent. I’ve chosen to create this as a component (inherited from Sys.Component) mainly because I might want to add event handlers and more to it another time, so having it as a managed disposable object is potentially useful.

/// <reference name="MicrosoftAjax.js"/>

/// <reference path="~/PersonService.svc" />

/// <reference path="Person.js" />

Type.registerNamespace("ServiceAgents");

ServiceAgents.PersonServiceAgent = function() {

ServiceAgents.PersonServiceAgent.

initializeBase(this);

}

ServiceAgents.PersonServiceAgent.prototype = {

initialize: function() {

ServiceAgents.PersonServiceAgent.

callBaseMethod(this, 'initialize');

dispose: function() {

ServiceAgents.PersonServiceAgent.

callBaseMethod(this, 'dispose');

GetPerson: function(identifier, callback) {

var success = Function.createDelegate(

this, this._onSuccess);

var failure = Function.createDelegate(

this, this._onFailure);

// call the web service

samples.microsoft.com.jslayering.PersonService.

GetPerson(identifier, success, failure, callback);

_onSuccess: function(result, context) {

// check we got a result back

if (!result.Found)

this._onFailure({ get_message: function()

{ return 'Person not found'; } },

context);

else {

// translate to our Person entity

var returnentity = new ServiceAgents.Person(

result.Person.Identifier,

result.Person.Name,

result.Person.Age);

// call the callback that was passed in

context(returnentity);

}

_onFailure: function(result, context) {

alert('A call to the Person Service failed: '

+ result.get_message());

}

ServiceAgents.PersonServiceAgent.registerClass(

'ServiceAgents.PersonServiceAgent', Sys.Component);

// ensure an instance of this component is created

// when the page is initialising

if (typeof (Sys) !== 'undefined') {

Sys.Application.add_init(

function() {

$create(

ServiceAgents.PersonServiceAgent,

{ id: 'PersonSA' },

null, null, null);

});

Sys.Application.notifyScriptLoaded();

}

There are four things to note in this component;

1. The only public method we’ve added is GetPerson. This literally sets up some call-backs and passes the call on to the auto-generated script proxy.

2. When the call succeeds, we check a result was found (using the Found property on the Message Contract) and if it was, we create a new instance of our own Person entity to pass back, by calling the supplied call-back method.

3. If there’s a failure we handle it; I could have let the caller do this, but this component is for illustrative purposes only... it is by no means perfect and I wanted to keep it quite brief!

4. We ensure an instance of our component is created when the page initialises, so that page script can easily consume it.

Using the Service Agent

Making use of this component is easy – we need to import the relevant services and scripts, and then get a reference to it using the $find method, and call it! First I register the scripts with a ScriptManager. We still need to point out on every page that we want to import the service, and now have to specify our Service Agent scripts too;

<asp:ScriptManager

ID="PageScriptManager"

runat="server">

<asp:ServiceReference

Path="~/PersonService.svc" />

</Services>

<asp:ScriptReference

Path="~/Scripts/Person.js" />

<asp:ScriptReference

Path="~/Scripts/PersonServiceAgent.js" />

</Scripts>

</asp:ScriptManager>

Then I’ve got the following script on my page;

function getPerson() {

var id = $get('RequestIdentifierTextBox').value;

var serviceAgent = $find('PersonSA');

serviceAgent.GetPerson(id, displayPerson);

// prevent post-back as we've handled it client side

return false;

}

function displayPerson(result) {

$get('IdentifierLabel').innerHTML = result.get_id();

$get('NameTextBox').value = result.get_name();

$get('AgeTextBox').value = result.get_age();

}

... which drives a page that looks like this;

Breaking the Application

Well this is not worthwhile unless we can demonstrate a tangible benefit. Imagine that code similar to that above is scattered throughout our web site, and we have many pages to maintain. Now imagine that our web service needs to change. Maybe the XML namespace is updated... or maybe we decide to rip out all of our Message Contracts (a bit contrived, but it demonstrates what we’re talking about!). My service becomes;

[ServiceContract(

Namespace =

"http://samples.microsoft.com/jslayering")]

[AspNetCompatibilityRequirements(

RequirementsMode =

AspNetCompatibilityRequirementsMode.Allowed)]

public class PersonService

{

[OperationContract]

public Person GetPerson(int identifier)

{

Implementation

}

Immediately our auto-generated proxy changes and the JavaScript code breaks. There’s no Found property anymore, we just get a null Person back if there is no match. And we can’t access a Person property on the Message Contract because it isn’t there... we’ve got a Person instead.

Obviously applying these changes site-wide would be a real headache. But for us, it’s easy. We just change a few lines! We’ll just tweak our _onSuccess call-back in the Service Agent;

_onSuccess: function(result, context) {

// note we only check for a result, not result.Found

if (!result)

this._onFailure({ get_message: function()

{ return 'Person not found'; } },

context);

else {

// note the changes when building our return entity –

// no ".Person." message contract syntax

var returnentity =

new ServiceAgents.Person(

result.Identifier,

result.Name,

result.Age);

context(returnentity);

}

Hey presto! Our site works again, and you can leave the office at a sensible time!

Conclusion

Obviously this is extra coding effort to put together a Service Agent, but do you see the benefits? Is it worthwhile? What about other patterns? How about View-Presenter, for example? Do they apply to JavaScript? Do you use this kind of approach already?

I’d love to know what you think, and what you do right now... feel free to comment or blog back at me.

The attached code should demonstrate all this – there are “-before” and “-after” web sites, both containing implementations of this page with a Service Agent and using direct calls against the auto-generated proxy.

In the “after” site I’ve removed the Message Contracts and tweaked the Service Agent... but I haven’t fixed the direct proxy calls so you can see how it breaks.

Enjoy...

Sufficient Architecture

simonince — Thu, 16 Oct 2008 14:16:00 GMT

“Sufficient Architecture? What does that mean?!”, I hear you say!! Bear with me.

Over a beer this week I had a good conversation with a colleague about the principles of architecture and design, and as a result I feel compelled to recount something that I have tried to adhere to for a long time. I know some people will disagree with what I’m saying, so if that’s you, shout up and let me know your thoughts.

When designing a new system I always have two principles in mind (and I’ll probably use the term “design” to refer to both Architecture and Design throughout this post, as it applies to both);

1. Design and Build only what you need today.

2. Design knowing that you’ll need something else tomorrow.

Let me elaborate a little.

Design and Build only what you need today

What I mean by this is that you should not try and anticipate the future. You shouldn’t add methods to classes “because in the future I might need to X”, and you shouldn’t complicate your architecture if there’s no need.

A good example that I discussed with a colleague is adding a Service Layer to an application; perhaps putting all business logic behind web services, and calling those from ASP.NET. I firmly believe you should add a Service Layer if you have good reasons for having one. If you need to scale out components separately, or perform asynchronous processing, or there are hosting prerequisites, or you have other applications that need to communicate with your application, etc – go ahead and add one.

However, if at this point you don’t have any good reason – don’t add one!! If you think an application might need to communicate with yours in the future, remember it is only that – “might”. You cannot anticipate accurately what it might need – you’d get contracts and message exchange patterns wrong, you’d be increasing the attackable surface area of your application, and you’d be complicating your development, maintenance, and deployment.

You should always build to meet the requirements.

What I don’t mean is that you should make your application rubbish – for example, having an application with a good User Experience should always be a requirement. But having one that enables the user to change the layout of your pages – that might or might not be a requirement. If it isn’t a requirement, don’t spend extra time designing and building it.

If you’re convinced that something should be done but it isn’t a requirement, you need to revisit your requirements (a great example of this is instrumentation – so often missed from non-functional requirements!). If you don’t revisit the requirements, you (the techie) are making up your own requirements, and that becomes very dangerous territory.

Design knowing that you’ll need something else tomorrow

The problem with only building what you need today is that we all know systems change and evolve, need maintaining, upgrading, or fixing. Building only what you need today compliments this though – as I’m sure regular readers know I firmly believe simplicity is something you should strive for in all your systems. It keeps learning curves shallow, and therefore helps keep maintenance and enhancement costs down. Building only what you need helps keep your application simple.

But we must also acknowledge that we know your system is probably going to change. This means you need to consider good architectural and design practices, use well known successful patterns (for example a UI pattern like MVP or MVC), write automated tests to validate your code (both now and when you’re enhancing it later, as they then become great regression tests), use clear naming conventions, ensure you can monitor your system, appropriately componentise the code, and so on.

When you design the API for your application, ask yourself something – “if in 5 years time someone comes to change this, will they understand what my code is doing?”.

Designing your application well means that if we need to add a Service Layer in the future (or perhaps just expose a small part of your logic for another application) the task of estimating, designing, and building these changes is not as complex as it could have been, and it is entirely appropriate that the effort is expended on doing this once you know you need to.

The argument against

What I am also not saying is “build applications as quickly as possible”, or “just databind a data grid directly to your database”. Please don’t just build a bad application, say “it meets requirements”, and blame me! J The term Rapid Application Development scares the life out of me – and I’m certainly not aiming at that! This is why the second principle exists. Your non-functional requirements should describe the need for the system to be maintainable, scalable, supportable, etc – and we as developers and architects know this means following good practices.

So what do you think?

The small print: Do remember that the examples in this post are primarily targeted at a single system, living in a fairly isolated world. In particular, my comments about Service Layers could be misinterpreted if you’re building an application that is part of an Enterprise’s Service Oriented Architecture – but of course you should have requirements that dictate how it should participate in the SOA, and what systems must integrate with your new application, so the principles hold true.

The “Service Interface” Pattern

simonince — Mon, 25 Aug 2008 20:06:00 GMT

I am constantly surprised when speaking with people how few have heard of or use the “Service Interface” pattern. It is actually a very straightforward pattern, is very little work to use, but brings such practical, quantifiable, visible, benefits I think it has to be one of my all time favourites. The Web Service Software Factory embodies a particularly good implementation of the pattern, too, so be sure to check that out. In fact, I’d go so far as to say that the easiest way to get familiar with this pattern is to read this post (obviously J) and then try the factory.

There’s already a bunch of documentation out there, including the pattern description on MSDN (although the corresponding implementation advice is a little dated due to the advent of things like WCF). The Web Service Software Factory documentation also describes the pattern, adding some more information and a sample to the MSDN docs.

The basics

What on earth is this pattern then? Well basically it is there to keep the “surface area” of a web service independent from the Business Logic. In exactly the same way as you would partition your User Interface from your Business Logic, it is wise to partition your Service Interface.

Separating these concerns means that your logic can evolve in implementation without breaking consumers of your service (remember that contracts between services are quite brittle in many cases), and also to evolve the communication mechanisms and formats your service uses without affecting your logic. Makes a lot of sense, huh?!

The Service Interface is also the ideal place to do things like message logging, exception shielding, versioning, and so on. All of these concepts are highly recommended for consideration before you deploy the first version of your service.

Pattern Components

There are a number of components that work together to form a well structured Service Interface. These components are depicted below;

I’m sure you’ve noticed that I don’t like to draw and talk too much without dropping down to see what this really means in code. So taking each component, one at a time, let’s see what they are, what they do, and what the code looks like. I’ll use WCF in this example, but the sample concepts can apply to other technologies.

Data Contracts

Data Contracts are the reusable entities used in service communications. They are similar to business entities, but are specifically used for over-the-wire transmission. Think of similarities to how Fowler’s description of Data Transfer Objects is implemented.

[DataContract(

Name="Employee",

Namespace="http://samples.microsoft.com/contracts/2008/08")]

public class Employee

{

[DataMember]

public string Name { get; set; }

[DataMember]

public int PayrollNumber { get; set; }

}

This shows a simple Data Contract. Note that this is an entity – all the data it contains is intrinsically related, in that it describes properties of an Employee. The DataContract and DataMember attributes are used to decorate the class for the DataContractSerializer’s benefit.

Do also pay attention to the fact that I’ve used the Namespace parameter of the DataContract attribute – this is an absolutely key step in ensuring your versioning strategy is up to scratch. I should really have also implemented IExtensibleDataContract, as per Craig’s post on Versioning WCF Services. I won’t cover this in detail but do make sure you read up on this before diving in.

Message Contracts

So Data Contracts define entities – but our service doesn’t deal in entities! It deals in messages. Therefore Message Contracts group together Data Contracts to form meaningful and useful messages.

Usually each operation (i.e. method on a service) has two Message Contracts – a Request and a Response. One Way operations obviously have no Response message, but you get the picture!

[MessageContract(

IsWrapped=true,

WrapperName="AddEmployeeRequest",

WrapperNamespace="http://samples.microsoft.com/contracts/2008/08")]

public class AddEmployeeRequest

{

[MessageBodyMember]

public int DepartmentIdentifier { get; set; }

[MessageBodyMember]

public Employee Employee { get; set; }

[MessageBodyMember]

public Employee Manager { get; set; }

}

[MessageContract(

IsWrapped=true,

WrapperName="AddEmployeeResponse",

WrapperNamespace="http://samples.microsoft.com/contracts/2008/08")]

public class AddEmployeeResponse

{

[MessageBodyMember]

public bool Success { get; set; }

[MessageBodyMember]

public string FailureMessage { get; set; }

}

Here we see I’ve introduced an “AddEmployee” operation that our service will provide. I always use the “Request” and “Response” naming suffix convention to ensure that the intended use of these messages is clear. Message Contracts are also not reused like Data Contracts are; they are inextricably bound to an operation, and exist purely to define that operation and to allow versioning strategies to exist.

Note that we use the MessageContract and MessageBodyMember attributes to decorate our contract, and that again I have used the Namespace parameter to help with versioning. I always use “IsWrapped=true” to ensure maximum compatibility (see Using Message Contracts).

Members of the Message Contract can be either Data Contracts or primitive types – the example shows the use of both our Employee contract and an Int32.

Service Contract

The Service Contract brings together our Message Contracts (and therefore implicitly our Data Contracts) to use them to define operations, and to group operations together to form a service.

[ServiceContract(

Namespace="http://samples.microsoft.com/contracts/2008/08")]

public interface IEmpoyeeManagement

{

[OperationContract(

Action="http://samples.microsoft.com/contracts/2008/08/ É

IEmployeeManagement/AddEmployee")]

AddEmployeeResponse AddEmployee(AddEmployeeRequest request);

}

Note that in this case we’re using a .NET interface, not a class. We’re also using the ServiceContract and OperationContract attributes to decorate the interface. The Namespace and Action parameters to these attributes are used for versioning as usual.

Here you can see that my Request and Response messages are tied together by the Operation Contract definition for AddEmployee.

Service Configuration

The Service Configuration consists of many different things. I would like to define it as things that can be added to a service to customise the behaviour without requiring code or contracts to be changed... but unfortunately that isn’t quite true. For example, changing the InstanceContextMode can have a big effect on how you write your service. I guess this means that some specific areas of configuration and implementation are actually quite closely coupled, for good reason.

Anyway, configuration of a service includes things like the web.config settings (under <system.serviceModel>), and the use of the ServiceBehavior attribute. My service is so basic all I have in configuration is the default WCF web.config settings, so I’ll not bore you with that here!

Service Implementation

Well this one is pretty obvious, right? This is the meat of my service – the point of all these contracts, right? It’s the logic?

Nope.

I don’t mean to be so ruthless with this statement, but I just think it is important to understand. A service implementation is still part of the Service Interface – and it is purely responsible for translating between wire-format entities and business entities, maybe applying custom authorisation, message logging, exception shielding, and so on. It is almost an “adapter” layer that maps incoming messages onto the correct business logic – I firmly believe a service implementation should be very thin as a result of this, delegating most work to the business layer. The service implementation is not the business logic itself.

Let’s see a simple example implementation then;

public class EmployeeManagementService : IEmpoyeeManagement

{

public AddEmployeeResponse AddEmployee(AddEmployeeRequest request)

{

try

{

ITranslator<Employee, Person> translator =

Container.Get<ITranslator<Employee, Person>>();

Logic.SavePerson(

translator.Translate(request.Employee),

translator.Translate(request.Manager),

request.DepartmentIdentifier);

AddEmployeeResponse response = new AddEmployeeResponse();

response.Success = true;

return response;

}

catch (BusinessLogicException exception)

{

AddEmployeeResponse response = new AddEmployeeResponse();

response.Success = false;

response.FailureMessage =

ErrorMapper.SanitiseErrorMessage(exception);

return response;

}

catch (Exception exception)

{

SystemFailure failure =

ErrorMapper.SanitiseFailure(exception);

throw new FaultException<SystemFailure>(failure);

}

We can see our service implementation is a class that implements our service contract, and there is very little else of note other than the method code itself. So let’s inspect the method code and see what we find.

Firstly we have some entity translation. It is really common for a service implementation to need to translate from a Data Contract to a Business Entity, and therefore a simple yet consistent approach to this has been born. Consider the following implementations;

public interface ITranslator<From, To>

{

To Translate(From from);

}

public class EmployeeToPersonTranslator :

ITranslator<Employee, Person>,

ITranslator<Person, Employee>

{

public Person Translate(Employee from) ..

public Employee Translate(Person from) ..

}

These translators can easily be registered with a Service Locator (or we could use Dependency Injection or Dependency Resolution) as in my example, and retrieved with a call to a container.

Next we see our service implementation calls into some business logic, passing in translated entities and any additional data from the request message that is needed. Finally it constructs a success message and passes it back.

The other interesting part of this code is what happens in event of an error. My implementation is primarily designed to demonstrate a few different approaches, but the important point is that no error is returned to the caller without being “sanitised” first – i.e. I am applying the concepts of Exception Shielding. This ensures private data about my service (for example connection strings) is never exposed to the caller.

I’ve shown a few different approaches – using a return value in the response message, or using a Fault Contract (just a Data Contract used to communicate a fault) by throwing a new FaultException<T>. Fault Contracts should really be used in WCF if there is an unexpected error, but I do think there are some valid circumstances to use a “roll your own” response message; perhaps this is a topic for another post.

One point to note is that using this Fault Contract requires me to decorate my service contract with a FaultContract attribute;

[ServiceContract(

Namespace="http://samples.microsoft.com/contracts/2008/08")]

public interface IEmpoyeeManagement

{

[FaultContract(typeof(SystemFailure)]

[OperationContract(

Action="http://samples.microsoft.com/contracts/2008/08/ É

IEmployeeManagement/AddEmployee")]

AddEmployeeResponse AddEmployee(AddEmployeeRequest request);

}

Summary

Well that’s it – I hope this has helped you see how Service Interface fits together and what some of the benefits are. I’ve found that this approach helps with versioning, but also it helps to encourage a message-exchange approach to design instead of RPC (Remote Procedure Call).

What I mean by this is that services become designed to have course-grained interfaces, avoiding chatty calls. RPC is often seen like Java’s RMI – just calling methods on an object that happens to be physically located somewhere else – and this leads to forgetting about the costs of crossing service boundaries. Defining Data Contracts helps maximise reuse, and defining Message Contracts helps decrease granularity.

So now you love Service Interface just as much as I do, have a read of Don’s post regarding Message Contracts, and see what you think. I’m a firm believer in always using Message Contracts as I’m sure is obvious from this post, but make up your own mind.

WCSF Application Architecture 7: Remote Logic with WCF Services

simonince — Fri, 18 Jul 2008 18:55:00 GMT

This article is part of a series;

· WCSF Application Architecture 1: Introduction

· WCSF Application Architecture 2: Application Controller

· WCSF Application Architecture 3: Model View Presenter

· WCSF Application Architecture 4: Environment Abstraction

· WCSF Application Architecture 5: Defining Modules

· WCSF Application Architecture 6: Structuring Modules

Unfortunately it is also the worst named post in the series J

Introduction

It is a common requirement for customers to want to split web applications across physically separate tiers, usually using some kind of services technology. This post discusses some approaches to this using the Web Client Software Factory.

Although I focus on using WCF services, many of the concepts apply to ASMX and other technologies too – I have just chosen WCF as I believe it to be the best way to write services in the current .NET Framework environment.

* Note: Chances are I’m going to use the term Web Services a lot in this post. What I really mean when I say that is any kind of remote service – pretty much anything that can be written with WCF, for example.

Food for Thought

So first things first – let’s clear up some common myths.

1: SOA != Services

Just because your application uses Web Services (or any other kind of remote service), that doesn’t make it Service Oriented. It could still be just an app that uses Web Services. The current Wikipedia definition of SOA starts with a nice succinct statement about this;

“(SOA) is ... where functionality is grouped around business processes and packaged as interoperable services ... SOA separates functions into ... services ... which are made accessible over a network in order that they can be combined and reused in the production of business applications”.

In other words, if you’re writing a Web Service and Web Application as an inextricably linked pair that make up a single system, chances are you’re not doing SOA. SOA is an Enterprise Architecture, not an Application Architecture (although some applications will be built on the Enterprise Architecture by aggregating and reusing business services, and hence some apps are validly referred to as Service Oriented).

What this means to me is two things

1. Be pragmatic about what rules you follow when you are writing a completely isolated internal application.

2. Ask yourself this: “Why are we using a Web Service?” If your answer is “Because our company policy is that all new apps must be Service Oriented” (or something similar) I would recommend stepping back, and thinking deeply for a short while. How is adding a Web Service making your application Service Oriented? Is it really adding any value? If so, great – there are many valid reasons... but make sure you know what they are!

2: Services cost performance

Using a remote service is expensive (in terms of processing time, for example). If you previously had a method call that looked like this;

View.Orders = myBusinessLogic.GetOrders(user, startDate, endDate);

... and you move the implementation of GetOrders to be a remote Web Service, you will notice a significant performance degradation. Raw performance and scalability are often at odds like this! Suddenly we must serialize user information and dates, locate the remote service and open a channel, pass the data across a potentially unreliable network, spin up a service instance, deserialize the information, perform the business logic call that used to exist on the client, serialize the results, pass them back across the network, deserialize the results and turn them into a .NET list, and finally close the channel to the remote service. Phew. Now all that might be pretty fast in .NET, but it will be slower than an in-process method call.

Having said that, if the logic you must execute is expensive itself, you may find that removing load from the client Web Server and scaling it out to another physical tier improves the scalability of your system. I’m not saying don’t use services here – I just want you to be realistic about it!

3: RPC != Messaging

Because services add overheads, it is important to adopt a message exchange approach rather than a remote procedure call approach. What does this mean? Basically it comes down to two things;

1. Granularity. RPC consists of chatty, frequent calls between client and service, very quickly making it difficult to scale your system. When you think RPC, think of many systems written with Remoting or Java’s RMI. Message exchanges are courser grained and minimise the amount of calls and proportion of overheads involved.

2. Programming Model. I tend to find that RPC encourages the sharing of concepts internal to a service with the outside world. Use a message oriented approach makes you think about defining the contracts, abstracting where necessary, and encapsulating behaviour appropriately. Watch for a future post on the Service Interface pattern when I get a chance!

Candidate Architectures

So now that we’ve thought a little about what we want to avoid, let’s see a couple of approaches and discuss their characteristics.

1: Baseline

First up is our baseline architecture; this is what future examples will evolve from, and represents a basic WCSF application (ignoring splits across modules) without any kind of service layer.

Note that I’ve decided to expose my business logic here using a WCSF application service (not a web service). This communicates with my “model” (basically the business logic), which in turns talks to various data access classes (e.g. DbAccess). I hope that’s pretty straightforward.

2: Split Tiers

I nearly named this “RPC approach” because I feel that it encourages an RPC style of programming. Components above the dotted line are in my WCSF Web Application, and those below it are in the WCF Web Service.

What we have done here is just move the logic into a Web Service, and then change the WCSF application service that previously exposed business logic to purely act as a facade to the remote Web Service – in other words, it is a Service Agent or Service Proxy. This is a key concept – a Web Service is a resource, so if you are communicating with a Web Service directly in a class, you’re writing a resource (or data) access component.

The problem with this approach, in my opinion, is that this encourages developers to just call methods as though they were local to the web application, potentially leading to an RPC style system with very chatty communications and a serious scalability problem. Often people don’t realise that they are effectively writing an application that has no business logic layer (the Web Application)!! This is obviously contrary to well understood wisdom nowadays, and looking at the diagram above, I hope you see how unnatural it looks.

This lack of separation can also lead to changes in the service contracts affecting the UI. This is by no means ideal.

3: Remote Services

My preferred approach is shown below;

Here you can see that our missing logic layer has made a triumphant return! This might seem subtle but I think it completely changes class responsibilities and how your developers perceive the system. The logic in the Web Application is responsible for deciding when to go to a web service, or when to use a local cache, how to co-ordinate calls between different resources (you might have multiple web services!), when to enforce security or other policy, and so on. This then leaves your Resource Access layer able to dedicate itself to providing a real shield from the inner workings of the service calls.

As I’m sure regular readers have noticed, I’m a big fan of talking in concrete examples, so let’s see some (simplified) code that demonstrates this approach. Firstly, the Presenter and implied View;

public class MyPresenter<IMyView>

{

private ILogicService _logicService;

public MyPresenter([ServiceDependency] ILogicService logicService)

{

_logicService = logicService;

}

public void OnRetrieveOrders()

{

List<OrderEntity> results =

_logicService.FetchOpenOrdersForCurrentUser(

View.StartDate, View.EndDate);

View.Orders = results;

}

The key here is that the presenter is using a WCSF service (badly named as ILogicService) to fetch orders for display to the user. These orders are represented by an “OrderEntity” entity. Next, let’s see an example of an ILogicService implementation;

public class LogicService : ILogicService

{

private IAuthenticationService _authenticationService;

private IEntityTranslatorService _entityTranslatorService;

public LogicService(

[ServiceDependency] IAuthenticationService

authenticationService,

[ServiceDependency] IEntityTranslatorService

entityTranslatorService)

{

_authenticationService = authenticationService;

_entityTranslatorService = entityTranslatorService;

}

public List<OrderEntity> FetchOpenOrdersForCurrentUser(

DateTime startDate,

DateTime endDate)

{

OrderServiceProxy proxy = null;

try

{

proxy = new OrderServiceProxy();

proxy.Open();

proxy.LogOnService(CERTIFICATE_DETAILS);

string orderStatusCode =

proxy.GetOrderStatusCode(OrderStatus.Open);

List<Order> orders = proxy.GetOrders(

_authenticationService.CurrentUserName,

orderStatusCode,

startDate,

endDate);

List<OrderEntity> results =

_entityTranslatorService.Translate<Order, OrderEntity>(orders);

proxy.Close();

return results;

}

catch (Exception)

{

if (proxy != null)

proxy.Abort();

throw;

}

There are a few really important points to make here;

1. I have chosen to abstract environmental issues, such as authentication, away into a framework service. See my previous post on Environment Abstraction for more information on this.

2. All the management of the interaction with the service proxy is kept out of the Presenter, and instead is handled by our WCSF application service.

3. I am performing Entity Translation between the “Order” and “OrderEntity” objects. The former is actually part of the definition of the Order Service’s Service Contract, in that it defines how the remote service is communicated with. The latter is a Business Entity defined by my WCSF module. Therefore it is important that I expose the Business Entity not the proxy communication entity further up my architecture.

4. The service performs some limited Orchestration – it calls a number of methods on the service to achieve what is seen by the Presenter as a single business operation. Obviously I would prefer a very course grained service design, but I’ve done this to illustrate that this is the right place to perform any required orchestration (potentially across multiple services) and workflow.

5. The disposal of the proxy is done correctly. A “using” block is not the best way to deal with WCF proxies – see Avoiding Problems with the Using Statement.

6. This is also the right place for rich error handling, publishing, and compensation – please don’t follow my poor example in this respect J

7. I haven’t used the rich implementation of the Service Interface pattern I would usually, because constructing the messages would increase the volume of sample code and detract from the point. I’ll probably post about Service Interfaces soon, so watch this space!

The final important point is that this separation of business logic, resource access, entity translation, etc is likely to be repeated by the Order Service.

Potential Gotchas

When using WCF proxies there are a number of things worth knowing about. The first is the best way to use a “using” block, mentioned above.

Secondly, remember that a WCSF service is a singleton – so if you cache the WCF proxy you will be using the same one for every call, no matter which end user is browsing your site. No probs... oh, hang on! But what if you’re using a session-full binding? That would mean all your WCF calls would be using the same session, no matter which end user the call was for! This could certainly be a problem – so be very careful about this.

Thirdly, if you are caching your WCF proxy remember that a faulted proxy cannot be used again. So if your backend service fails and then comes back up, and you’re caching the proxy forever and for every user, your web application will be unusable until you cause it to flush the cache (e.g. by restarting the App Pool). The correct way to handle a faulted proxy is by adding the following code when creating it;

proxy = new OrderServiceProxy();

((IOrderServiceChannel)proxy).Faulted +=

new EventHandler(OrderService_Faulted);

In the event handler you can then react in some way – perhaps by Aborting the proxy and replacing it with a fresh one. This is obviously pretty academic for my example code above, but if you are caching proxies, consider this carefully. Also, if you don’t have a business logic layer, you might find it more difficult to deal with this kind of situation!

Finally, I’ve used the class generated by “Add Service Reference” in Visual Studio. It is actually pretty slow to new up one of these for every business operation when compared to using a cached ChannelFactory. In .NET 3.5 this is less true, but I’d recommend this post comparing the two approaches.

Summary

There are many issues I’ve not covered here – security architecture, for example (Trusted Subsystem versus Delegation), and many, many, many, more. But I hope this has gotten you started on how to structure your code, freeing you up to investigate some of the more subtle complexities. Good luck!

And finally... a Summary of Summaries

This is the last post in my series on Web Client Software Factory Application Architecture. That doesn’t mean I’ll not post about WCSF again of course, just that we have now covered plenty, from the basic patterns involved to some more “fluffy” concepts of designing modules or service layers. I hate television series that have no end (wow, it is seriously difficult not to name and shame here, that’s how much I hate it!), so I’ll take my own advice and finish on a high J

Remember how much I haven’t covered – there is plenty of information out there, from the WCSF community site, landing page, other bloggers, obviously in the factory documentation, and so on. And if you’re blogging about it, do let me know!

WCSF Application Architecture 6: Structuring Modules

simonince — Fri, 04 Jul 2008 13:13:00 GMT

This article is part of a series;

· WCSF Application Architecture 1: Introduction

· WCSF Application Architecture 2: Application Controller

· WCSF Application Architecture 3: Model View Presenter

· WCSF Application Architecture 4: Environment Abstraction

· WCSF Application Architecture 5: Defining Modules

Introduction

This post is intended to address some common scenarios when using the Web Client Software Factory, in particular relating to how modules should be structured internally, and how they should relate to each other... as with all my other posts, this is based on what I’ve done and what I’ve seen others do – I’m sure there are other ways out there, so please feel free to chip in if you have something to say, suggest, or disagree with.

I’m going to present architectural candidates for you to consider, and discuss why you might choose to use them, which I hope should give you ideas about how they might apply to your situation.

So, get comfortable, and read on!

For consistency with my previous post in this series, I will assume that initially we have some business modules (Customers, Accounts, and Orders) and a foundation module (Infrastructure). I’ll pick and choose from these to illustrate my points as we go.

Model 1: Classic

I’ve named this model “Classic” as it is pretty much the basic use of the WCSF as it was intended;

Note: An arrow indicates a reference – so Customers has a project reference to Customers.Interfaces. The coloured bars indicate responsibilities for each project.

This diagram shows us a number of things;

1. The WCSF provides an out-of-the-box option to define an “interfaces” project for each module, in which the public “contract” of the module is defined. An “Interfaces” project contains contracts (i.e. for services a module exposes) and entities.

2. The Web Site contains the Web Pages and User Controls for a module.

3. The implementation projects (what I call the module projects that are not suffixed with “.interfaces”) contain the full remaining layers of the architectural stack, from UI (presenters etc) through business logic, to resource access.

4. The Web Site may reference both interface and implementation projects.

5. The implementation projects may reference their own or other modules’ “interfaces” projects. This allows modules to take dependencies on services exposed by other modules in a loosely coupled way.

6. As shown by the X marks, modules should not reference each other in any other way. Doing so would cause their implementations to be tightly coupled, and can cause circular references – which would be really bad news.

I’m a big believer in showing some code to demonstrate how this works, so let’s take a simple example and consider the following;

namespace Customers.Interfaces

{

public class Customer

{

public int Id { get; set; }

public string Name { get; set; }

}

public interface ICustomerService

{

List<Customer> GetCustomers();

}

namespace Customers

{

public class CustomerService : ICustomerService

{

public List<Customer> GetCustomers()

{

// implementation

}

Here we see the Customer entity and the ICustomerService contract are defined in the Customers.Interfaces project, and a real implementation of CustomerService in the implementation project. Obviously there would be IView definitions and Presenters in the implementation module too, and ASPX pages etc in the web site, but these are not relevant to this example.

Model 2: Shared Entities

Model 1 is in many ways the ideal – but the problem is life is often more complicated than this! The first thing you’re likely to encounter is when two modules need to share the same entity. So consider what happens if I define the following (slightly contrived) contract in my Orders.Interfaces project;

namespace Orders.Interfaces

{

public class OrderDetails

{

public string OrderReference { get; set; }

}

public interface IOrderService

{

void PlaceOrder(Customer customer, OrderDetails order);

}

Looks great huh? Ah, but Customer is defined in Customers.Interfaces... I could add a reference from Orders.Interfaces to Customers.Interfaces, but how long would it be before I needed to do the same in reverse and had a circular reference?

This leads me to a piece of advice;

Avoid having references between “.interfaces” projects

Also, remember that in my previous article we discovered that it is good practice to ensure that one and only one module owns access to any particular data source. So we shouldn’t create two different customers entities. So what do we do?

This diagram has way too many arrows on it to indicate acceptable references, but I wanted to provide a complete view, so I apologise! Basically, then, this boils down to creating a separate shared project in which you put common entities and contracts. Entities that only relate to a single module can stay in that module’s interfaces project.

So in our example, this becomes as follows;

namespace Shared

{

public class Customer

{

public int Id { get; set; }

public string Name { get; set; }

}

namespace Customers.Interfaces

{

public interface ICustomerService

{

List<Customer> GetCustomers();

}

namespace Customers

{

public class CustomerService : ICustomerService

{

public List<Customer> GetCustomers()

{

// implementation

}

namespace Orders.Interfaces

{

public class OrderDetails

{

public string OrderReference { get; set; }

}

public interface IOrderService

{

void PlaceOrder(Customer customer, OrderDetails order);

}

You see that the Customer entity has moved out, and now we have no circular reference concerns.

Model 3: Layered Modules

I have historically always separated my application layers by project – I find this helps keep the layering clear, and keeps the volume of code in any one project to a manageable level. The WCSF does not do this by default – but it doesn’t prevent you from doing it! In fact, some of the reference implementations split modules in a similar way;

This diagram shows that I have added two new standard C# projects to the solution, and named them Customers.Data and Customers.Logic (note I have omitted the Orders functionality here just to keep the diagram simple). My layers are now kept nice and separate. Each layer may reference the layer below, but not vice-versa.

Now there is an interesting point to make from this diagram – my Logic and Data projects have no reference to the Customers.Interfaces project. This means they cannot deal with the “Customer” entity!

I actually think this is a matter of choice – it depends on how much you wish to ensure your layers are loosely coupled. Using the approach I have taken, you would need to use an adapter or entity translation pattern in the Customers implementation project. Here, the UI facing classes (such as presenters or services) would translate between the outward facing entities defined in Customers.Interfaces, and those used internally by the module (perhaps database-focused entities). The services in the Customers module would simply act as a facade to real logic, contained in the Logic project.

The truth is, it can be a bit of a pain taking this approach, so you may wish to compromise – but the point is, it needs a few moments of thought. Choose an approach that you feel comfortable with and stick to it throughout your solution.

Model 4: Split Modules

An alternative approach to bringing extra clarity to your layering is to physically split up your modules;

Note: I have excluded the Web Site from this diagram, as I’m sure you’re getting the point!

Here you can see that we have Customers and Orders business modules, without matching interfaces projects. These only contain UI code, allowing you to keep the projects really focused. We then have two foundation modules – CustomersLogic and OrdersLogic (not very well named, I admit!). These modules expose services to the UI modules to allow them to access the “model” (as Model-View-Presenter) that is our business logic. The reference between the UI code and the business logic is loosely coupled as it goes via the foundation modules’ interfaces projects. If you really wanted, you could take this one step further and split the foundation modules into more projects as we discussed in the Layered Modules approach.

This is indeed very similar to the Layered Modules approach, except that it gives you an opportunity to deploy business logic separately to your user interface. This can have big benefits for team working (e.g. different teams working on the UI and business logic), or for deployment – perhaps your UI works for a number of systems, with different business logic plugged in behind?

Model 5: Split Master Logic Module

All of these approaches so far have one thing in common; they expect you to be able to vertically partition your code to some greater or lesser extent. In reality, this is the hardest bit of getting your WCSF implementation right – what parts of the system are really independent from each-other?

In particular, if you are migrating an existing system you may well find all of your business logic and data access is so tightly related, interlinked, and interdependent, that you cannot split it into modules without all sorts of circular references, shared entities, and so on. You could try, and end up with spaghetti dependencies between modules – even if it is loosely coupled, too many dependencies on each-other’s services is not a good sign. Or maybe you just happen to be building a small system that only really has one single area of functionality?

So all this means you could well end up with one single module, immediately losing some of the benefits to team working, componentisation, etc, that come with modularisation. Right?

Well, this is where the Split Master Logic Module approach comes in. Split your UI into modules – because that is often pretty easy - and then keep all your back end logic in a single foundation module that exposes services to the rest of your application;

I must make it clear that with this model you are missing out on the benefits of the complete vertical slice through your architecture you would get with a purist approach, but it does still have a place I believe.

This allows you to work in a loosely coupled way, modularising your UI (including independent deployment of modules if you wish), yet maximising the reuse of any existing business logic you have – it could be as simple as surfacing existing code as WCSF application services. You of course still have all the flexibility of a loosely coupled architecture too, so should you wish to migrate your existing code to a new, more modular architecture, just plug-and-play different service implementations as time goes by.

Conclusion

Well, that’s it – a lightning fast tour of some approaches to splitting your modules up. Have I missed an approach? Have you come up with something better? Do you hate one of the approaches above? As always, shout up and let me know.

The one approach I haven’t discussed is using a Service Layer to physically split the UI from a logic tier. Watch this space and I’ll try to post some thoughts on this soon.

Finally, I threw a few comments into the text above that might have got you thinking... if so, check out;

· The Adapter pattern

· Entity Translation – best explained by Don Smith, in context of the Web Service Software Factory but relevant to us too!

· My thoughts on LINQ solution structures (aimed at a WPF app, but some relevance to the web)

Enjoy!

Dependency Injection is Dead!

simonince — Mon, 30 Jun 2008 23:10:00 GMT

Long live “Dependency Resolution”!

OK, so I’m not really serious – but I got your attention right? Truth is, I personally love Dependency Injection, but that doesn’t mean it isn’t without its flaws. The Service Locator pattern is often touted as Dependency Injection’s nearest rival, so let’s very briefly examine some of the characteristics of each of these to see what works and what doesn’t.

What is great about these patterns?

It is hard to condense my enthusiasm into just a few bullet points! I’m also sure I’ve missed some, but just to get our minds in sync;

1. Testability; the loose coupling of your implementations means you can easily substitute mock classes, enabling your tests to focus on a single “unit”. You can also supply different implementations to allow testing at all (e.g. replace the calls to HttpContext.Current when you have no HttpContext).

2. Design approach; enforcing loose coupling helps to encourage good design. It helps to shield consumers from the service’s implementation, enabling you to swap implementation if necessary.

3. Configuration; there is often a big push towards highly configurable containers, both from the point of view of behaviour and their contents. This gives systems a feeling of agility, and the ability to adapt to different deployments or reuse in future systems.

4. Object lifetime management flexibility; objects can be cached, be a singleton, or be short lived – all controlled through configuration.

What is wrong with Dependency Injection?

There are some potentially painful issues with DI;

1. I find it counter intuitive that often some object somewhere needs to remember to “Build Up” an object. In other words, the chain of iterating over objects to fulfil their dependencies needs to start somewhere. This can be abstracted into HttpModules, framework code, and so on, but I still don’t like this. An object’s consumer shouldn’t need to know that X must be done before Y will work if at all possible.

2. When using property injection you generally need to mark your dependencies as public. This can seem counter intuitive, and certainly limits some use cases. Constructor injection can help here though.

3. Reflection; most (perhaps all?) Dependency Injection containers rely on some extent on Reflection – dynamically inspecting objects and determining their dependencies. I know there are optimisations out there for this, but the fact still remains that this can be an overhead.

4. Object chains can become quite large. If they’re needed this is no problem, but what if they’re not? Consider this code;

public class Chain

{

[CreateNew]

public Customer PrimaryCustomer { get; set; }

[CreateNew]

public Partner Partner { get; set; }

[CreateNew]

public ProductCatalogue Catalogue { get; set; }

[CreateNew]

public AddressLookupService AddressLookup { get; set; }

public string GetPrimaryCustomerName()

{

return PrimaryCustomer.Name;

}

public class SecurityService

{

[CreateNew]

public LoginProvider Login { get; set; }

[CreateNew]

public RoleProvider Role { get; set; }

[CreateNew]

public TraceProvider Trace { get; set; }

[CreateNew]

public AuditService Audit { get; set; }

}

OK, so it isn’t meant to be nice code, so please don’t flame me J. Still, you see the point? If all I want to do is call GetPrimaryCustomerName, and all that uses is one single dependency, my container will still have created instances of all my other types and populated the dependencies of my Chain object... and of course done the same to all the dependent objects. This could quite easily lead to an increase 10-fold in needless object creation and initialisation! “Use a singleton” I hear you say – sure, if applicable. But why use a pattern that has an enormous design impact on my code purely to reduce object creation just because my DI container is doing something slightly crazy? And even if I’m not creating objects, I’m still resolving their references.

What is wrong with Service Locator?

My main bug-bear with Service Locator is that it permeates throughout all my code. And most importantly, every object I write must know about the Service Locator in order to be able to fetch its dependencies.

Consider the following code adapted from the example above;

public class Chain

{

public string GetPrimaryCustomerName()

{

Customer primaryCustomer = ServiceContainer.Resolve<Customer>();

return primaryCustomer.Name;

}

This is must nicer – but it does mean my Chain class needs to know how to get hold of a Customer. Is this really so different to just creating a new one? Obviously you get the configurability of the Service Locator, but the separation of concerns is wrong in my opinion.

I should point out at this point that some clever people I know actually positively like this approach – they want to be explicit about fetching services and working with them. I see their point, and guess this comes down to personal preference, but the key is, if you like this approach chances are you don’t like DI at all – so this article is hopefully still of interest!

Lazy Dependency Resolution

So can’t we get the benefits of Dependency Injection and Service Locator without some of the drawbacks? I hope so; enter “Lazy Dependency Resolution” (which I already seem to keep abbreviating to just “Dependency Resolution”).

This is actually nothing new; I’ve just given it a name that sounds appropriate, having found a use for a new framework I’ve been playing with. Really, this is an approach cobbled together from various ideas and sources, and some similar ideas flying around “out there” on the web, but then that’s where all the best patterns start I think!

With the advent of static weavers that allow us to intercept calls using Aspect Oriented Programming techniques (see my previous article), we suddenly have an opportunity. The fact of the matter is that the Service Locator pattern does pretty much everything your average DI framework does, except that the service consumer needs some knowledge about how to call the Service Locator. Well, if we pull that concept out of our consumer classes and stuff it into a shiny new Aspect, what do you get? I think you get an implementation of a “Lazy Service Locator” that isn’t tightly coupled into the service consumer.

Enough talk, show me the code!

An Example with PostSharp

I’ve chosen to use PostSharp here as I’ve been playing with it a bit recently, but the idea is based on the concept of static weavers in general. If you want to know why this matters, read my previous article! </shameless-plug>

Quick Disclaimer: Note that this sample is really really rough-and-ready. It is not meant to be neat and tidy – it is just showing an alternative approach that I like. The reality is that I think some of the AOP libraries could help a bit by providing alternative interception models, support for explicitly only targeting properties a little more easily, or better ways of manipulating arguments and return values.

Anyway, for the simple demo we need two classes that must be loosely coupled; a consumer and a provider. The consumer consumes the provider... so to keep this dependency nice and flexible we’ll use an interface for our provider (so we could mock it if we liked, or switch implementations, etc).

public interface IProvider

{

void PrintId();

}

I told you this was going to be a simple example J

Next we need a consumer class that makes a call to an IProvider implementation;

public class Consumer

{

public IProvider Provider { get; set; }

public void DisplayInfo()

{

Provider.PrintId();

}

As you can see, we simply declare that we need an IProvider using a property here; we don’t specify where we get this from, or what concrete implementation we’re using. This matches how many Dependency Injection frameworks string objects together. Next, let’s have a concrete IProvider implementation (although we could have many different implementations if we liked);

public class ConcreteProvider : IProvider

{

private Guid _id = Guid.NewGuid();

public void PrintId()

{

Console.WriteLine(String.Format("My ID is {0}.", _id));

}

So what I want to be able to do now is call my Consumer class in a completely transparent way, avoiding calls to build the object up, or fetch it from a factory, etc;

public class Program

{

static void Main(string[] args)

{

Consumer consumer = new Consumer();

consumer.DisplayInfo();

}

OK, build it, run it... oh wait, we got a NullReferenceException because the Provider property is null.

Somehow, then, we want to tell our Consumer class which IProvider implementation it should be using. Well consider the following Aspect code;

[Serializable]

public class DependencyResolutionAspect : OnMethodInvocationAspect

{

public override void OnInvocation(MethodInvocationEventArgs eventArgs)

{

eventArgs.ReturnValue = ServiceContainer.Resolve(

eventArgs.Delegate.Method.ReturnType);

}

What this is doing is inspecting the declared return type of the method call we’re currently intercepting, and then looking up an implementation in a Service Container to return to the caller. This Service Container could be a facade to any number of registry techniques, so I’ve not included the code. Of course, this means it might return a singleton instance, a new instance, a mock, a concrete live implementation – all the configurability of Dependency Injection or Service Locator remains.

We then simply apply this aspect to our Consumer class’ properties using an assembly-level attribute;

[assembly: DependencyResolutionAspect(

AttributeTargetTypes="PostSharpDR.Consumer",

AttributeTargetMembers="get_*")]

Of course, there are other ways to apply aspects, but this seemed the quickest and simplest sample. Note that I’m filtering for methods that begin with “get_”, which is how .NET compiles the “getter” for a property.

This now means my application code can run as we wanted – assuming the Service Container is suitably initialised. I’ve chosen to do this in code here, but of course this could easily come from configuration;

public class Program

{

static void Main(string[] args)

{

ServiceContainer.Register<IProvider>(new ConcreteProvider());

Consumer consumer = new Consumer();

consumer.DisplayInfo();

}

As if by magic, our application now works.

One unpleasant implementation detail is that we effectively have a property that does nothing on our Consumer class. In fact, code that assigns to this property will compile, but is counter-intuitive at run time as the value you pass in will never be passed out again from the getter! Therefore, consider the following aspect instead;

[Serializable]

public class DependencyResolutionAspect : OnMethodInvocationAspect

{

public override void OnInvocation(MethodInvocationEventArgs eventArgs)

{

object existingImplementation =

eventArgs.Delegate.DynamicInvoke(

eventArgs.GetArgumentArray());

if (existingImplementation != null)

eventArgs.ReturnValue = existingImplementation;

else

eventArgs.ReturnValue = ServiceContainer.Resolve(

eventArgs.Delegate.Method.ReturnType);

}

* Note this doesn’t support value types correctly! Again, I’m trying to keep my code simple

Now our aspect will invoke the property that it is intercepting, and return the property’s existing value if it is non-null. If the current value of the property is null, we return an implementation resolved via our Service Container. Of course we could do this in a number of ways (e.g. intercept the set call too, and use a private Boolean field that flags whether a manual set has occurred). This now means we can override the Dependency Resolution behaviour – something that could be really useful for testing if we don’t think we need (or want) to spin up a configured Service Container.

Therefore the following code in a unit test will always use our MockProvider, without changing how any aspects are compiled, and without worrying about whether a Service Container has been initialised or not.

Consumer consumer = new Consumer();

consumer.Provider = new MockProvider();

consumer.DisplayInfo();

Simple huh? The last thing I should mention is that caching the resolved service would probably be worthwhile – because at the moment we’re calling to the service container on every access to the property. This is easy to fix though, so I haven’t bothered for the sake of this post J

Good, Bad, or Ugly?

I think the best way to consider how this stacks up against the alternatives is to compare against what I see as the down-sides to DI and Service Locator;

1. Responsibility for building up objects. This responsibility has now been pulled out into an Aspect; ideal for AOP, this is a cross-cutting behaviour that I can apply to many methods using configuration statements. My code stays clean, my concerns are separated, and my developers, testers, and architects are all happy (oh, and the flying pigs are too).

2. Public dependencies. I don’t see any reason why I couldn’t modify the IL for protected and private properties too. We’d just need to be a little careful of this – is a developer really expecting a private object to be resolved by an apparently “external” component? Oh, and I don’t support constructor injection in this post, but I’m sure we could.

3. Reflection; Doh!! See comments below.

4. Large object chains; This is perhaps the best bit; as all resolution is done in a “lazy” fashion, only those paths through the object chain get created or resolved. Hey presto, no over-the-top object instantiation, yet no loss of benefits from a Dependency Injection style approach.

What don’t I like?

One issue is that really this needs a static weaver to achieve our aims – so many frameworks (Spring, the Enterprise Library, and so on) are out of the picture... for now.

[Edit: The following paragraph is my original content, but it turns out is not a completely accurate picture; be sure to check out Gael's comments below - approx the 27th comment down! Simon]

Secondly, the fact that my implementation using PostSharp still relies on reflection is a shame, and this is down to how the framework functions; really I’d like to edge this out of the door to maximise the benefits of the pattern, but I don’t fancy writing my own IL parsing cleverness! I haven’t had time to flesh out my ideas around this yet, but if I ever do I’ll be sure to post them!

Conclusion

Well, what do you think? Let the debate begin! I think this is a very simple evolution of Dependency Injection that removes some of the common objections to it – and hence could increase the audience that benefit from what I consider a good pattern to encourage good software design.

Be aware I haven’t used this in a real implementation yet... I’m floating a concept for discussion, so feel free to chip in.