Tom Hollander's blog

See you at TechEd Australia and New Zealand

2009-08-24T14:18:58Z

Just a quick post for anyone attending TechEd Australia and/or New Zealand, in the first couple of weeks of September 2009. I’ll be doing two presentations, so please stop by and say hello (and maybe even watch the sessions!).

The sessions I’m presenting are:

ARC302 – Introducing Enterprise Library 5.0
Be one of the first to get an early look at the next release of Enterprise Library. The EntLib team feels your pain, and have been working hard to dramatically simplify the architecture, improve performance and usability, and inject more of that DI goodness into your applications.
ARC303 – Making enterprise code re-use work in the real world (co-presented by Guy Rigby in Australia)
Most of you would have either built or consumed a component, framework, library or factory before. But why is the holy grail of efficient code reuse so hard to attain? Why do frameworks often cause way more pain than they solve? Come along to hear some good and bad stories that may help your next code reuse effort be more successful.

I’m looking forward to seeing you there!

Beware the perils of local time

2009-06-21T05:06:28Z

Where were you at 2:15am (Sydney time) on 5 April 2009? Other than the fact that your memory is pretty hazy now that a few weeks have passed, you’d expect there to be an unambiguous answer to that question. However it turns out you could well have been in two completely different places at that time, as Sydney was coming out of daylight savings on that night, so the hour from 2am to 3am was repeated.

If you’re like me, you’ve probably known that this happens forever, but never really worried about it since you’re normally fast asleep at the time. However in the last few months I’ve been building an application that needs to keep an accurate record of when events happen (for example, in transaction and audit logs). In the early stages of development we recorded all of these events using local time, which was easy to build and easy for everyone to understand. However we realised this wasn’t a good idea when we first deployed the application to the target environment which is in a different time zone – all of the timestamps looked wrong when viewed from our time zone.

So we modified the application to record all timestamps in UTC. This was easy to do – basically we used DateTime.UtcNow everywhere we had previously used DateTime.Now. However we still wanted the application’s UI to show all timestamps in the users’ local time. Amazingly there was no way of doing this in .NET Framework 3.0 – it could convert between UTC and server time, but not between two arbitrary time zones. When you think about this more, it’s actually a very difficult problem as it requires knowing exactly when daylight savings starts at ends in any given year, and governments have a nasty habit of changing this from year to year. So the solution is to have a large historical database of daylight savings dates in every location in the world. Since .NET Framework 3.0 didn’t have this, we experimented with a few open source solutions but couldn’t find one that was reliable. Eventually we solved this problem by moving to .NET Framework 3.5, which has a wonderful class called TimeZoneInfo that does exactly what we need.

This all happened a few months ago, and I thought we were out of the woods. However last week we found one (hopefully) last surprise. Our application also includes some daily batch jobs that do some additional processing on the last day’s events. Even though our events were being recorded in UTC time, the batch job was scheduled to process events for a 24 hour period ending at the same local time every day. For 363 days a year this was fine, but there is one day a year with 25 hours (in which case we would have missed an hour), and one day a year with 23 hours (in which case we would process an hour twice). Theoretically we could have added some smarts to the application to deal with these days, but we decided the path of least resistance was to change the jobs to process 24 hours of events ending at the same UTC time (meaning the cut-of time will appear to “float” in local time as we move in and out of daylight savings).

Even though I’m a big fan of having an extra hour of sunlight on summer evenings, I’ve learned that daylight savings time (and indeed local time in general) can have a significant impact on applications that need to be precise about recording and processing timed events – even if your users aren’t scattered across time zones. The moral of the story is to store dates in UTC wherever possible, and to move to .NET Framework 3.5 to make conversions a snap.

Protecting against XML Entity Expansion attacks

2009-05-21T12:25:00Z

One of the critical responsibilities of every developer and architect is to understand, and know how to prevent, as many kinds of security attacks as possible. While there are many types of attacks and many weapons at our disposal to thwart them, the most basic defence we have is input validation. The rule of thumb really needs to be to assume all input from uncontrolled sources is malicious, unless you can prove otherwise. This includes input from end users, as well as input from other systems.

Recently I worked on an application that received XML files from that most untrustworthy of sources, the Internet. Knowing the kind of people who lurk there, we took what seemed like a responsibly paranoid approach involving validating each parsed document against an XML schema, checking a digital signature to ensure it came from a known sender, and cherry-picking the values we needed out of the document.

So I was quite surprised to learn that there were was a class of attack which we had not mitigated. It turns out that you should never load untrusted XML content into a .NET XmlDocument class as a first step, even if you plan to do all sorts of checks on it afterwards. This is because there is a class of attack which can bring your server to meltdown just by getting it to parse some XML.

Consider this piece of XML:

<!DOCTYPE foo [ 

<!ENTITY a "1234567890" > 

<!ENTITY b "&a;&a;&a;&a;&a;&a;&a;&a;" > 

<!ENTITY c "&b;&b;&b;&b;&b;&b;&b;&b;" > 

<!ENTITY d "&c;&c;&c;&c;&c;&c;&c;&c;" > 

<!ENTITY e "&d;&d;&d;&d;&d;&d;&d;&d;" > 

<!ENTITY f "&e;&e;&e;&e;&e;&e;&e;&e;" > 

<!ENTITY g "&f;&f;&f;&f;&f;&f;&f;&f;" > 

<!ENTITY h "&g;&g;&g;&g;&g;&g;&g;&g;" > 

<!ENTITY i "&h;&h;&h;&h;&h;&h;&h;&h;" > 

<!ENTITY j "&i;&i;&i;&i;&i;&i;&i;&i;" > 

<!ENTITY k "&j;&j;&j;&j;&j;&j;&j;&j;" > 

<!ENTITY l "&k;&k;&k;&k;&k;&k;&k;&k;" > 

<!ENTITY m "&l;&l;&l;&l;&l;&l;&l;&l;" > 

]> 

<foo>&m;</foo>

This certainly looks like an odd bit of XML, but at first glance it doesn’t appear overly scary. It’s compact, well-formed and actually only contains one element: <foo>. But what’s in that element? It’s a single custom-defined entity, &m;. And how is that defined? Well, it’s 8 other custom &l; entities. So what’s an &l; then? Hmm, it’s 8 &k;s. You can see where this is going. The document will end up with 8¹² &a;s, where each &a; has 10 characters, so that innocent looking &m; will blow out to 10x8¹² or 687,194,767,360 characters. And on my reasonably well spec’ed developer machine, expanding that number of characters consumed all of my CPU for longer than I was prepared to put up with. A bad guy armed with this attack isn’t going to steal any data, but they could still cause a lot of damage through denial of service.

The good news is that it’s actually very easy to stop this entity expansion in its tracks. The key is to use an XmlReader before parsing the document into an XmlDocument (or instead of, if you can live without a fully-parsed document). It’s possible to validate against an XSD or other schema type using an XmlReader too, but here’s a minimalist example showing how you can check that a document is well-formed, contains no DTDs (and hence no entity definitions) and is less than 10K in size:

// Prepare text reader and settings for Xml Validation
StringReader textReader = new StringReader(unparsedXml);
XmlReaderSettings settings = new XmlReaderSettings();
settings.XmlResolver = null;
settings.MaxCharactersInDocument = 10000;

// Successfully parse the file, otherwise an XmlException is to be thrown
XmlReader reader = XmlReader.Create(textReader, settings);
while (reader.Read()) ;

If you get to this point without an XmlException being thrown, the document should be safe to parse. Of course, there could be all sorts of evil things lurking within the elements of the document, so you need to continue to use appropriate validation and encoding as you would for any untrusted input.

Your guided tour of the Microsoft Solutions Development Centre

2009-05-03T03:21:00Z

When I decided to leave the patterns & practices team to move back to Australia, one of my big concerns was whether I would be able to work on teams with the quality and dedication I experienced on projects such as Enterprise Library. It turned out that my fears were unfounded, as I’ve found myself working in the most high-performing team I’ve ever experienced, Microsoft Australia’s Solutions Development Centre (SDC). The SDC is basically a software engineering team available for hire, building complex solutions for external customers. We pride ourselves on excellence in process, technology and outcomes, using the latest Microsoft technologies and agile development practices to drive the best possible outcomes for customers.

About a month ago, the SDC opened its doors to a wider audience in an event we called the SDC Open Day. This consisted of a series of presentations describing all aspects of how we manage our projects, followed by a tour of our facilities with drinks, nibblies and networking. I had originally planned on blogging about this before the event so more of you could attend, but it turned out that we reached our maximum capacity through direct invitations and word of mouth.

The event went off without a hitch, and the feedback from the 150-odd attendees was overwhelmingly positive. However right from the start we wanted to find a way that this event would continue to provide value long after the projectors went dark. So we decided to video the entire event (over two hours of it!) for you to enjoy no matter where you are. The videos have just been posted live on microsoft.com.

If you’re in Australia (or are happy to move here :-), we’d obviously like you to consider how the SDC could help your team become more effective and your organisation to deliver critical solutions. However no matter where you are, I’m hoping you’ll find the videos interesting. Whether you’re a developer, tester, project manager or architect, the videos will show you some techniques that our team has found very successful, which you may want to apply to your own projects.

The event was divided up into 20 minute sessions presented by people playing different roles in the SDC team (including both Microsoft staff and the partners that work with us), which you can view individually in any order. There are also some introductory interviews and “vox pops” to add a bit of spice. The main sessions are:

An Introduction to the SDC, presented by Rob Mawston (SDC Lead at Microsoft). Rob provides some background into why the SDC exists and how we approach software development.
A Day In the Life of the SDC, presented by me (Solution Architect at Microsoft). I take the audience through a typical day, describing the key activities performed by the team and each individual role.
How we do: Project Management, presented by Prasadi de Silva (Senior Project Manager at Microsoft). Prasadi discusses how we use agile requirements management and metrics to ensure we successfully deliver what the customer actually needs.
How we do: Development, presented by Corneliu Tusnea (Senior Consultant at Readify). Corneliu describes some of the techniques our developers use to ensure ongoing quality and agility, including unit testing and refactoring.
How we do: Testing, presented by Sarah Richey and Bruce McLeod (Managing Director and Principal Consultant at Devtest). Sarah and Bruce describe why testing is critical to success, and how we use metrics and automation to give us great coverage throughout the project.
How we do: Build and Deployment, presented by Emma Hanna and Simon Waight (Senior Consultant and Lead Consultant at Avanade). Emma and Simon describe how we use daily builds, CI builds and automatic deployments to ensure the solution is always in a known state for use and testing.
A Customer’s Journey, presented by Fiona Boyd (COO at Ticketek Australia). Fiona describes Ticketek’s experience as a customer working with the SDC.

I hope you find the videos interesting. Please feel free to post any questions on anything you see in the videos to the blog and I’ll try to get them answered (if not by me, then by someone!).

@tomhollander just doesn’t get it

2009-04-19T03:46:00Z

I tried. I really did. For five months I’ve chronicled my daily thoughts and activities for all to see. But despite my efforts, this really only confirmed what I suspected right from the start: that Twitter is really pretty pointless.

Now before you all flame me, I realise that some people do get a lot of value out of Twitter. I’m just not one of them. When I first dipped my toes in the water it became clear that Twitter suffers from a typical “chicken and egg” problem – it’s no fun if you don’t have anyone following you, and it’s hard to get people to follow you when it’s no fun. But I stuck at it, eavesdropping on the veterans and occasionally replying when something interesting came up. Over time this approach has brought me a few followers and the odd interesting conversation – but not enough to justify the investment it’s taken to get this far.

Looking at the people who sing Twitter’s praises, I think I know why. These people get value because they have thousands of followers. And they get thousands of followers by tweeting all the time. I’ve got no problem with people doing this, but frankly I’ve got other things to do (and besides, while I am geeky, I’m not that geeky :-).

I’m curious about the much-discussed surge in Twitter usage over the past year – how many people are really finding it valuable, and how many are, like me, giving it a go but yet to find the point. In any event I’m not throwing in the towel just yet – I’ll keep an eye out on what’s happening and may even contribute the odd tweet. But from now on, it’s up to Twitter to prove its worth to me – I’m done with trying to prove my worth to it.

So for at least a little bit longer, @tomhollander is happy to continue the conversation, just as long as interesting and not too much hard work.

What do you want to see in EntLib 5.0?

2009-03-31T01:34:00Z

As you’ve probably noticed, I’ve been on a bit of a blogging vacation of late. Rest assured that I’m still here and I’ll try to get some good posts happening soon. But for now I just wanted to draw your attention to one of Grigori’s posts asking for your help in prioritising the features in the upcoming Enterprise Library 5.0.

There are a lot of cool things in the proposed list including overhauls to the config tool, support for using the Validation Application Block in WPF applications, a bunch of new training materials, and even a Resource Application Block to help with localisation.

But don’t let me (or anyone else) tell you what’s important – go and tell the team what you really want by completing the survey!

Enterprise Library 5.0 kick-off! Spend your $100 wisely!

2009-01-27T09:43:00Z

It’s that time again – the kick-off for the next major release of Enterprise Library. I can’t believe we’re already up to version 5.0 – it doesn’t seem like that long ago when we were planning version 1.0.

Anyway, you should know the drill by now. The patterns & practices team needs your input to decide what new features and scenarios go into each new release. Despite the tough economic times, Grigori has been kind enough to give each and every one of you $100 each to spend on the new release – please drop by his blog and let him know how much you want to spend on which features.

I’m going to have to think hard as to where I spend my cash. Unfortunately we’re still using .NET 3.0 (and hence EntLib 3.1) on my current project, and most of my wishes were already granted in EntLib 4.x. Still, there’s always room for improvements, and I’ll be sure to come up with something.

Just Released: Validation Application Block Hands-On Lab

2009-01-09T06:15:00Z

Reposting Grigori’s announcement in case you missed it:

Here’s a gift for the New Year’s. We have produced a new hands-on lab on validation with Enterprise Library. It contains 13 exercises that walk you through capabilities of the Validation Application Block in various application contexts:

The first 11 deal with a Windows Forms data processing application that takes the information entered by the user to populate and process business entities. The Validation Application Block is used to validate the created business objects before processing them in gradually more sophisticated ways.

Starting with Lab 7, the Windows Forms validation–integration feature is used to directly validate the input for the form's controls.

Labs 8 through 11 deal with the extensibility of the application block.

Lab 12 shows how to use the ASP.NET validation-integration feature of the application block to validate the ASP.NET control's values, using a Web forms version of the simple data entry application from the previous labs.

Finally, for Lab 13, the ASP.NET application works as a front-end for a Windows Communication Foundation (WCF) service while the WCF validation–integration feature of the application block is used to declaratively validate the service parameters on the server side.

The lab instructions are available as a CHM for easy navigation and as a PDF for printing.

There are two ways you can complete this lab set: you can manually complete it from start-to-finish or you can use the starter solutions to complete only the labs you want to. By using the provided starter solutions, you can complete any of the labs in the order you prefer.

Download Validation HOL

Next up is our new Interception hands-on lab. You may expect to see it released in January.

Thanks to the EntLib team for keeping the good stuff coming!

The Joy of Code Reviews

2009-01-06T10:08:00Z

As I mentioned in my recent post about how my team does agile, one of the core ingredients of our process is that nobody is allowed to check in without having gone through a code review and a test review. No other team that I’ve worked on previously has had such a rigorous process around code reviews – while we’ve had ad-hoc pair programming and occasional code walkthroughs, there were no rules about all code being reviewed before check-in. So when I first joined my new team at the SDC, I was unsure what to expect or if I’d like it. But as you might guess from the title of the post, I’ve become a convert.

First, let me go into a bit more detail about how the process works. A developer prepares a change set which normally consists of one completed requirement, or one or more bug fixes. Once they believe they are ready to check in, they will shout out “Code Review!”, at which time any other developer who can spare some time will volunteer to be the reviewer. In some cases the “reviewee” will seek out a specific “reviewer” if they know them to be best qualified in the relevant technology or component.

A code review will typically take around 15 minutes, but they may be considerably longer or shorter. It takes place at the reviewee’s computer (we normally have our entire team working in the same room. For a while we did have one developer in another location - in this case we mimicked the “same computer” approach by using desktop sharing and speakerphones). Normally there isn’t any need to walk through the functional requirements or the high-level design in any detail, since the entire team is involved in the planning sessions and generally knows the critical details. However in some code reviews there may be some use of the whiteboard to communicate any design details that are needed to provided context to the code.

The review is performed by looking at the list of changed files in Visual Studio’s “Pending Changes” window, going through them one-by-one (sometimes from top to bottom, sometimes in whatever order the reviewee thinks is most logical), and performing a “Compare with Latest” operation on each file. Most of us have Beyond Compare or something similar installed to make this easier, but the Visual Studio text comparer works OK as well. We don’t have a specific checklist of things that need to be reviewed, but some typical areas for discussion include:

Quantity and quality of unit test coverage
Code readability, method and line length
Opportunities for reusing existing functionality, or merging new code with existing functionality
Naming conventions
Consistent application of patterns
Globalisation (appropriate use of culture classes, resource files etc.)
Hacks, shortcuts or other “smelly” code

If the reviewer is happy with everything in the change set, it’s ready for a test review (or if that happened first, it’s ready to be checked in). Alternatively, the reviewer can veto the check-in and insist that any uncovered issues are fixed first. In rare cases the reviewer may allow the check-in even if there are known issues, with TFS bugs or tasks created for tracking purposes. This option is most commonly used when the issues are minor and there are other developers waiting for the check-in before they can complete their own tasks.

So why did we choose to impose this additional workload across the development team? Well, it’s certainly not because the developers make a lot of mistakes or need close supervision – the team is as experienced and capable as any I’ve worked with. And in fact it is quite rare for a code reviewer to veto a check-in – I don’t have hard statistics but it probably only happens 1 time out of 10. Nevertheless I think the process extremely valuable for the reviewer, the reviewee and the quality of the codebase. First, each developer writes code with full knowledge that it will be scrutinised, so they take extra care to follow established patterns and avoid ugly hacks. Second, it helps “share the wealth” around who understands different parts of the solution. And finally it provides a very personal way for developers to learn from one another, whether it be a new Visual Studio shortcut key, a .NET API they didn’t know existed, or a new architecture, design or testing technique.

One more interesting observation about how this process works in my team: at our “retrospective” meetings that we run at the completion of each iteration, there have been a number of occasions where people have called out that it takes too long to check in code. However I’m not aware of any situations where anyone in the team has suggested abolishing (or even streamlining) the code review or test review processes to achieve this outcome. And having the support and confidence of the team is the ultimate measure of the success of any process.

StickyNotes for Visual Studio

2008-12-23T03:04:00Z

Pablo Galiano, one of my friends and colleagues from my patterns & practices days, has just released a very cool extension for Visual Studio 2008 called StickyNotes. As the name suggests, it allows you to attach sticky notes to your code, with a lot more richness and less intrusiveness than regular code comments. You can also choose between personal notes (visible only to the person who created it) and team notes (visible to the entire team).

Unfortunately my team is still using Visual Studio 2005 (we started before VS2008 was released and haven’t had time to upgrade yet!), so I haven’t been able to use this on my project yet, but I have played around with it and it looks very nice. And while this tool isn’t free, at $US9.99 (about what I pay for lunch) it’s practically free. Give it a try!

How my team does agile

2008-12-16T03:02:00Z

As you know, I’m a big fan of agile software development. But what exactly does “agile” mean? If you ask a room full of software engineers that question, you’re sure to get as many different answers as there are people. I’m not going to try to tell you what agile is, or what it should be – but a lot of people ask me how our team goes about implementing agile. We don’t practice any particular official brand of agile – like most good teams we’ve combined all of our past experiences to come up with something that works well for our project and team. So I’m definitely not claiming that this is the “right” way to do agile or that it will work for your team. However I’m also not going to accept any arguments that what we’re doing is “wrong” (since it works very well for us) – although constructive suggestions are always welcome!

To provide a bit of context, I’m working with a dedicated software engineering team that is part of Microsoft’s Solutions Development Centre in Sydney. We have been building a large new system for an external customer for the last 12 months or so. It’s largely a greenfield project, but there are a handful of external systems we need to integrate with. The application is in a new business area for the customer so their requirements have been evolving constantly, especially after development began. The project team consists of 2 project managers, 1 solution architect (me), 5-9 developers (the numbers have changed a bit over time), 4-7 testers, a build manager, a release manager (just for the last third of the project), and part-time SMEs focusing on infrastructure, security and database. We also have a full-time product manager from the customer, and around 3 other customer representatives that spend at least a day a week with our team.

The project began with an end-to-end analysis and estimation phase, which lasted a couple of months. While this does not fit in with many people’s idea of agile, I believe it’s a necessary step for commercial projects (and probably worthwhile even for internal projects). The goal of this period was to get as good an understanding as possible of the business requirements (as they were understood at the time) in a relatively compressed time period. During this period the team produced UI wireframes, flowcharts and very high level requirements and design diagrams. Doing this estimation allowed us to provide a reasonably good estimate of the schedule and budget for the overall project (even with the full awareness that many, many things would change throughout the project). It also helped us identify the most important and most risky areas so we could schedule them in the most appropriate iterations.

Once the estimation phase was out of the way we started our development iterations. We settled on 4 week iterations, involving 3 weeks of development and one week of stabilisation. Here’s how we tend to approach each iteration:

During the second half of the previous iteration, the project managers, architect and customer start planning what requirements should be candidates for the next iteration. For a large project like ours, strict stack-ranking of requirements or stories was not found to be practical. Instead we broke the project into larger “modules” of functionality that each fit roughly into a development iteration and prioritised those. This allows each iteration to have a single theme and vision for the entire team to focus on. Within each module we will prioritise the requirements to ensure we work on the most important features first, and often it will turn out that we don’t get time to complete the bottom-ranked requirements (although they may still be resurrected in future iterations). We also find that new, related requirements tend to emerge as we begin development. As long as they are higher priority than other candidates, we try schedule them for the current iteration.
As the architect, I will spend time in the last week of the previous iteration looking through the candidate requirements for the next iteration and come up with a high-level design. This normally involves a first-cut at a data model, 5-10 pages of documentation (usually with lots of pictures!) and often a “spike” or prototype. The project and product managers will also spend time updating or creating UI wireframes and clarifying the requirements stored in Team Foundation Server. The design documents and wireframes produced during this time are never final, but having a documented starting point has proven to help the development team get on the same page quickly and hit the ground running when the iteration begins.
On the first day of the new iteration, the entire team gets together to discuss the proposed requirements in detail. This will typically involve reviewing the UI wireframes and design documents on a projector as well as less formal whiteboard discussions. This is also the time when the development team loves to point out problems in my design and how they intend on fixing them :-). Once everyone has a good understanding of the requirements, we’ll start assigning work to the developers. For a while we tried variations of “planning poker”, but we found it rather time consuming and not overly helpful. So we’ve settled on a simpler system where developers will volunteer for requirements until they believe they have at least a week’s worth of work. They will then go away and break it into finer-grained tasks with estimates. We found that 3 weeks of development is too long to plan for in one go, so we do small planning sessions at the start of the next two development weeks where we “top up” any developers who have less than a week’s worth of work with new requirements. In essence, the development phase of each iteration is broken up into three one-week mini-iterations all with a consistent theme.
We don’t have any official design period within the iteration, but for the first couple of days the developers will usually spend a lot of time at whiteboards figuring out how best to approach their requirements, with actual development ramping up quickly after that. As soon as a developer (or sometimes several developers) is finished with a requirement, it is marked in TFS as “ready for test”.
Testing obviously goes on throughout the iteration, but for the first week the testers typically focus on analysing the requirements and writing test cases. By the second week things are usually in full swing, with the testers developing automated test cases as well as performing manual testing. Any bugs that are discovered are logged in TFS. Bugs are considered “blocking” if they preclude the requirement from being effectively tested, but lower priority “non-blocking” bugs are logged too. Any requirements without blocking bugs are marked as “tested”.
The product manager (who represents the customer to our team) gets the final say on whether a requirement is complete. They will go through each of the “tested” requirements and confirm that it meets their needs. If it does, the requirement will be closed. If not, they may raise bugs (if it is not implemented as requested) or new requirements (which generally means “that’s what I asked for, but not what I want” :-).
At the end of the third week we hit our “code complete” milestone for the iteration. This doesn’t mean all candidate requirements were completed (in fact we try to have more candidates than we can complete), but it means we have completed all of the highest priority requirements in the time available.
The final week is for stabilisation. This means the developers will not start any new requirements, and will work solely on fixing bugs. In order to prevent the bug count from getting unmanageable by the stabilisation week, we also impose a “bug cap” on each developer during the development weeks – once a developer’s personal bug cap exceeds this value they will need to temporarily stop work on new features to fix bugs. However even with the bug cap in place, there are always enough bugs to keep the team busy for the last week. Testing continues during the stabilisation week as well (in fact it’s usually the busiest week in the iteration for the testers). To be completely sure we don’t run out of bugs, we also schedule a “bug bash” event at the start of the stabilisation week. This involves the extended team (usually involving people from the customer who are not normally involved with our team day-to-day) spending an hour playing with the application with a goal to discover as many bugs as possible. To make this fun we put on food and drinks, and hand out prizes for the most bugs, the most severe bug and the most obscure bug.
At the end of the fourth week, the iteration comes to an end. We strive to exit each iteration with zero P1 and P2 bugs, and an agreed limit to the number of P3 and P4 bugs. This effectively means that everything we have built in that iteration is complete and stable and ready to be deployed. That said, for this project we don’t actually deploy the application to real users after each iteration. As we get ready for our final release there will be a “final stabilisation” iteration, as well as a number of processes that can only be completed at the time of the final release such as end-to-end security and performance tests, deployment and failover testing.

Hopefully this gives you an idea of the rhythm of our iterations. However we also have a daily rhythm within our team room:

At 9am we have our daily stand-up meeting. We’re lucky enough to have our entire team working in the same location, so this doesn’t involve any fancy technology like conference calls. The goal of the stand-up meeting is to make sure everybody knows what’s going on in the team. Each team member gets up to 2 minutes to describe what they did yesterday, what they are going to do today, and what impediments (if any) they might have.
After the stand-up everyone starts work – although it’s normally quite a noisy affair (well it is when I’m involved!). We have our own dedicated team room so we can make as much noise as we like, and we encourage face-to-face collaboration on problems as much as possible. Many times a day, someone will raise interesting and difficult questions that may require the customer to clarify requirements, project managers to reprioritise or modify requirements and plans, and surprise changes to the design and test cases.
We encourage our developers to check in at least once a day to prevent painful integration and long periods where the testers can’t make progress. Before anyone in the team can check in (yes, me included!) they must go through a code review and a test review. Any developer or tester can perform these roles for anyone else, and we encourage as many combinations as possible. For the code review, the reviewer and the reviewee will share a computer and go through each of the changed code files and discuss the changes. The reviewer has the right to veto the check-in if they find problems, which can include inadequate unit test coverage. The test review is much the same but normally involves running the application through whatever obscure scenarios the tester can think of to see if it holds up. Once both the code and test reviewer are happy, the developer can check in.
We run a continuous integration build after each check-in, where the application will be compiled and all of the unit tests run on the build server. If the build breaks, noisy sirens will alert everyone in the team to the problem, and whoever caused the problem needs to fix it immediately as everyone else is blocked from checking in. A good CI build will result in a much more cheerful “Yippee!” sound.
At 4pm we do our daily bug triage. This involves the test lead, one of the project managers, the customer product manager and the architect (me). We will go through each of the bugs raised in the last 24 hours, debate its priority, decide if and when it needs to be fixed, and (if it’s not closed) assign it to the most appropriate developer to fix.
Also at 4pm we commence our daily build. In addition to compiling the code and running the unit tests, this involves deploying the application to our test server and running a suite of automated Build Verification Tests. Again, if any of this fails, the appropriate people need to down tools and get it all fixed.
Once the build is declared “good”, we will start our nightly run of automated test cases. The next morning the testers will analyse the results of the previous night’s run and raise bugs for any regressions, or sometimes update the automated tests to deal with failures caused by changing requirements.

I hope this gives you some insight into one way of using agile methods to deliver a complex project. As I mentioned at the start of the post, this isn’t textbook agile, and we’ve needed to constantly evolve our process to address various challenges and to suit the personalities and experiences of our team. However I’m very happy to say that this is the most passionate, energetic and productive team I’ve ever worked with, and despite the fact that the requirements continue to evolve at what often seems like a frightening pace, we’ve continued to hit all of our milestones and targets. So I hope some of this will be of use to your team as well, and of course I’d love to hear about what you do differently in your team and how it’s working out for you.

Windows Live finally makes sense

2008-12-13T11:31:00Z

Those who have followed my blog for a while should know that I’m not in the habit of using this space to blindly promote Microsoft products – although if I find something genuinely cool or useful I’ve been known to give it a quick plug. This is why I’ve never posted about Windows Live before. It’s not that the various sites and apps were bad – it’s just that they were obviously a random collection of rebranded MSN assets that didn’t make a lot of sense as a unit.

But thankfully this has all changed with the latest incarnation of Windows Live. This has been out for a couple of weeks now, but I’ve only started looking at it properly in the last few days – and so far I’m extremely impressed. While most of the old sites are there in some form, there is now a clear common theme: sharing and discovering information about your social network (which is built from your Messenger contacts). However rather than competing with other social networking sites, Windows Live is able to link to other sites (yes, even non-Microsoft sites such as Twitter, Flickr and StumbleUpon) and aggregate activities from those sites into a single view.

Aside from the new home page that shows all of your network’s activities, the coolest new feature I’ve discovered so far is the new Photos site and the corresponding Windows Live Photo Gallery downloadable app. You can now store all of your photos online in your SkyDrive (with 25Gb of free space!), share them with as many or few people as you want, and view them online in an awesome new Silverlight-based slideshow view. The offline Photo Gallery app has the expected integration with Live Photos and Flickr, plus some fun new features such as People Tagging and Photosynth stitching.

I’m sure there are much more comprehensive reviews out there, but I wanted to draw it to your attention as so far it doesn’t seem to be getting a lot of web love – and it finally deserves some!

Constructors and Inheritance – Why is this still so painful?

2008-11-18T12:32:00Z

Recently my team discovered a limitation in the RelativeDateTimeValidator that ships with the Enterprise Library Validation Application Block. This validator is used to check if a DateTime object occurs within a configured time before or after Now. It’s a useful validator for checking things like birth dates and expiry dates. However we discovered that it assumes the date being validated is in local time – if the date is specified in UTC (which we do in our app) the calculation will be wrong. Fixing the logic was no big deal – it just involved checking if the DateTimeKind is UTC and if so, converting it to a local time before doing the validation. However like many people we didn’t want to modify the original EntLib codebase, so instead I built a new class imaginatively titled RelativeDateTimeValidatorEx that inherits from RelativeDateTimeValidator and corrects this problem. But while the important code change was only a couple of lines, I was forced to manually implement every one of the 14 constructors from the base class (with no code beyond calling each base constructor) to ensure equivalent and compatible functionality. In my case I didn’t need to change the public interface at all. But even in the (probably more common) case where a derived class includes new functionality that warrants new constructors, it’s very common to want to include all of the base class’s original constructors as well.

To provide a quick example (but thankfully with fewer constructors), imagine the following base class:

public class BaseClass
{
    public BaseClass(int a) : this(a, "Foo")
    {
    }
    public BaseClass(int a, string b) : this(a, b, DateTime.Now)
    {
    }
    public BaseClass(int a, DateTime c) : this(a, "Foo", c)
    {
    }
    public BaseClass(int a, string b, DateTime c)
    {
        // Shared initialisation code
    }
}

Now if I want to build a new class that provides equivalent functionality and supports a new optional parameter, I’m left with this at a minimum:

public class DerivedClass : BaseClass
{
    public DerivedClass(int a) : base(a)
    {
    }
    public DerivedClass(int a, string b) : base(a, b)
    {
    }
    public DerivedClass(int a, DateTime c) : base(a, b, c)
    {
    }
    public DerivedClass(int a, string b, DateTime c) : base(a, b, c)
    {
    }
    public DerivedClass(int a, string b, DateTime c, long d) : base(a, b, c)
    {
        // New initialisation code
    }
}

…and if I want to offer more permutations, the number of constructors may grow exponentially.

The good news is that C# 4.0 will finally include support for optional parameters (a very useful feature that VB programmers have enjoyed for years), which will partly mitigate this issue by providing an alternative to writing a crap-load of constructors. However even with optional parameters, it would be nice if you could tell C# that you want your derived class to inherit all of the base class’s constructors.

On a somewhat related note, it’s often bothered me that there is no way to include constructors as a mandatory part of a class’s contract. This comes up every time you use a plug-in pattern where you create instances of a class at runtime using Activator.CreateInstance and you require all of your plug-ins to be initialised with the same information. The most logical approach is to require each plug-in to implement the same constructor. While you can make this work by passing the constructor parameters to Activator.CreateInstance, this is completely untypesafe as the compiler can’t enforce that all your plug-ins have the required signature (this is a common problem for people implementing custom EntLib plug-ins that receive their configuration properties via a NameValueCollection passed to the constructor. People who forget (or don’t know) to implement this constructor often have a hard time figuring out what is wrong).

The other approach you sometimes see is to create the instances using a default (parameterless) constructor and set the initialisation parameters through a public interface method like Initialize(NameValueCollection properties). But this approach has its own downsides. First, it means your plug-in classes are most likely constructed in an unusable state. Second, it still suffers from the same type safety problems, as there is no guarantee that every plug-in class implements a parameterless constructor.

So why isn’t it possible to specify constructors as a part of a class (or interface) contract? And to make it useful, there should be a compiler-verifiable way to create instances of unknown types that call the said constructor. I’m no language designer, but I’m thinking of something like this:

public class PlugIn
{
    public virtual PlugIn(NameValueCollection properties)
    {
    }
}
// Create an instance
Type plugInType = Type.GetType(plugInTypeNameFromConfigFile);
NameValueCollection properties = LoadPropertiesFromConfigFile();
PlugIn myConcretePlugIn = Plugin.New(plugInType, properties); // Compiler-checked

This is probably a bastardisation of the virtual keyword but you get the idea. Another benefit of this approach is that you could safely new-up instances of generic types in a lot more situations. Currently you can specify a where T : new() constraint for a generic parameter that specifies that the generic type must have a default constructor. I’ve often thought this was overly restrictive, and that it should be possible to do something like where T : new(int, string). This may still be desirable, but if constructors could be specified as a part of a construct then the type alone could allow the code to create new instances. For example, if you had a generic parameter T where T : PlugIn then your code could safely execute T myPlugIn = new T(properties).

Anyway, end of rant. I’m interested in hearing whether any other programming languages have either of the features I’ve proposed, or if there are any good reasons I haven’t thought of which would make these features undesirable or impossible.

What did the other 37.5% want?

2008-11-06T12:49:00Z

And so another US election is behind us. Despite the fact that I no longer live in the United States, the presidential race was still watched with interest by me and many, many others around the world. And like many others in the US and around the world I am very happy with the outcome, and look forward to a better world with a President Obama in the White House.

There have been a number of articles in the media calling out that the voter turnout was the highest in decades. Given how much was at stake, this didn’t surprise me. However I was shocked to discover that the highest turnout in decades has been estimated to be around 62.5%, or less than 2 out of 3 eligible voters.

For Americans (and most likely people from a lot of other countries) this may seem to be an impressive, or at least respectable figure. However having grown up in Australia, I’m used to seeing voter turnouts averaging 95%. While it’s possible that Australians are fundamentally more interested in politics than Americans, the real reason for such a high figure here is because this is one of a handful of countries that has (and enforces) compulsory voting.

Due to my cultural upbringing, I find this policy completely familiar and uncontroversial. However whenever this topic came up in conversation in the US most people were horrified, considering it a major violation of rights. There are a couple of reasons why I disagree with this view.

Quite a long time ago, I participated in a training course to learn how to conduct door-to-door market research (although as it turned out I never actually did the job!). In the training we learned that at any given time that you might knock on doors, some people may not be at home so obviously you can’t ask their opinion about whatever product you are researching. However rather than just discounting that person’s opinion, you had to come back on a different day and try again. The reason for this is that a sample of people that only includes those that are home at a particular time (like Saturday morning) is not considered statistically representative. Maybe the people who aren’t home were busy playing sport, and these lifestyle choices are likely to significantly shift their opinion on the product in question. Such practices are apparently common in the world of market research. So if these practices are important enough to guide research into views on fizzy drinks, shouldn’t they be used to decide who should be in government? There is no reason to believe that the group of people who feel uninterested or disenfranchised enough to not want to vote would vote in precisely the same ratio as those that do turn up. Governments must represent all of their constituents, so any election that only includes a self-selecting subset is inherently flawed.

Just to ram this point home, I’m sure you’ve all seen self-selecting polls on all sorts of topics that are frequently run by websites and TV news shows. Most of them contain a disclaimer something like this:

DISCLAIMER: These polls are not scientific and reflect the opinions of only those Internet users who have chosen to participate. Poll results cannot be assumed to represent the opinions of Internet users in general, nor the public as a whole.

The disclaimer is hard to argue with. So why do people believe that voluntary elections, which run on the same principle, are an accurate reflection of the public as a whole?

One common argument against compulsory voting is that some people genuinely have no opinion or are fundamentally opposed to voting on philosophical or religious grounds. And to go back to my market research example, even if the researcher keeps coming back until somebody is home, there is no way to force them to give an opinion if they don’t want to. While I find it hard to understand how someone can be completely indifferent to something that can have such a big impact on their lives, the Australian system does deal with this well. “Compulsory voting” is not really an accurate description of the system. It’s compulsory to turn up to the polling booth on election day and get your name ticked off. What you do with your ballot paper after that is really up to you. And while there are a percentage of people who will lodge an “informal vote”, the overwhelming majority of people will vote properly – after all, they’re already at the polling place with a ballot paper in hand.

I’m not expecting to convince everyone that this is the better way, nor am I saying that compulsory voting would have changed the result in the latest US presidential election. However I’m sure that there have been many elections over the years in many countries where the result would have been different had voting been compulsory. And I find it hard to see how things wouldn’t be better if the outcomes of elections were based on what the people really want, regardless of their views on the political process.

Speaking at WDNUG

2008-10-10T07:56:12Z

It’s been a little while since I’ve presented at a public forum – particularly since I missed TechEd Australia after it clashed with my Great Barrier Reef holiday. However this is all going to change with my next live appearance in the centre of the known universe, Wollongong! More specifically I’ll be talking at the Wollongong .NET User Group (WDNUG) next Wednesday, October 15. That should leave plenty of time for you all to book flights from all over the world to attend this monumental event.

I’m dong a talk with the catchy title of Aspect Oriented Architecture meets Service Oriented Architecture using the Policy Injection Application Block and WCF. Here are the official details:

The October WDNUG meeting is on Wednesday 15th October and this month we have Tom Hollander from Microsoft visiting to talk about Aspect Oriented Architecture within a .NET environment.

Abstract:
Service Oriented Architecture provides a way of looking at systems as black boxes, while Aspect Oriented Architecture allows separation of concerns within each of those boxes. In this session Tom will discuss how these approaches have been brought together on a customer project using Windows Communication Foundation and Enterprise Library’s Policy Injection Application Block, resulting in more predictable behaviour with less code.
Bio:
Tom Hollander is a Solution Architect in Microsoft’s Solutions Development Centre in Sydney, responsible for driving the technical design and delivery of complex customer projects. Prior to joining this team, Tom spent over three years in Microsoft’s headquarters in Redmond working as a product manager in the patterns & practices team. In this role Tom helped deliver many patterns & practices deliverables including Enterprise Library, the Guidance Automation Toolkit and Web Service Software Factory. Tom is a frequent blogger on patterns & practices and architecture topics, at http://blogs.msdn.com/tomholl.

Topic: Aspect Oriented Architecture
Date: Wednesday 15th October
Time: 6:30pm
Location: CSC Offices
Edney Lane
Mt St Thomas

Please RSVP so we know how many people will be turning up.

The attendance of all readers of my blog is mandatory. Looking forward to seeing you all in sunny Wollongong!