Espresso Fueled Agile Development : Testing

If you missed the patterns & practices summit....

mpuleio — Thu, 04 Dec 2008 12:37:00 GMT

Ajoy has posted that three of the sessions have been made available online at pnp Summit videos online. Since one of the three sessions is one that Grigori and I did about the Acceptance Testing Guide, Driving Development with Acceptance Testing, I figured I'd share the information. The other two sessions are Ade talking about distributed agile development, and a discussion with Ajoy on the new SharePoint guidance from p&p that I have mentioned before (Guidance on SharePoint, Unit testing SharePoint). If you missed the Summit and want to attend next year in Redmond (or elsewhere), keep an eye on http://www.pnpsummit.com/, which will eventually be updated with information on the next conference.

Acceptance Test Engineering guide Alpha Available

mpuleio — Wed, 27 Aug 2008 21:00:00 GMT

Last night I posted the Alpha version of Acceptance Test Engineering guide - How to Decide if Software is Ready for You or Your Customer to the Acceptance Test Engineering CodePlex community site. There are 300+ pages of content. It is still a bit rough, has some gaps in it, and needs some work, but there is a lot of good information too.

I would ask for some help from the public (developers, testers, PMs, and folks who act as customers or sponsor software projects): Read the Alpha (or just part of it) and reply to the Feedback Discussion with anything you think we got wrong, anything we got right, or anything we missed entirely. Thanks.

Unit testing SharePoint

mpuleio — Sat, 23 Aug 2008 00:38:00 GMT

Francis has an article on Unit Testing SharePoint 2007 Applications

He also did a brownbag talk on this yesterday, which was very interesting and brought up a few big challenges and ways to work around them.

Enjoy

Acceptance Testing Guide Community Preview 2 Available!

mpuleio — Mon, 04 Aug 2008 20:16:38 GMT

As I have mentioned before in a few posts (Acceptance Test Engineering Guidance, Acceptance Testing Guidance Survey, and Acceptance Testing Guide Update), I am working with a team on guidance around acceptance testing. We still have a lot of work to do our the guide, but we have made available a second community preview to show you how things are shaping up and to give you a chance to provide feedback.

Check these out:

The Acceptance Test Engineering Guidance community site on CodePlex (so you can follow the project)
An announcement about the guide (which is being handed out at the Agile 2008 conference this week)
The second community preview of the guide (it is a bit rough, but you should be able to see where we are headed with it)
Provide Feedback on Community Preview Aug 4, 2008 (so we can make changes, add topics we missed, etc).

Acceptance Testing Guide Update

mpuleio — Fri, 11 Jul 2008 03:27:48 GMT

The team has been writing, editing, re-writing, and working hard on putting together a guide on Acceptance Testing. We have made quite a bit of progress, decided on an organization scheme, the basic format of how we want to write each chapter, and have re-worked a lot of content to fit this model. We recently posted two things on the Acceptance Testing Guidance CodePlex site:

A survey on What is Hard To Test?
A preview release of a few chapters/snippets/etc.

So, please answer our survey. It only has ONE question.

Also, send us feedback on the draft sections of the guide we posted by starting up some discussions in the Acceptance Testing Guide Discussions group.

Acceptance Testing Guidance Survey

mpuleio — Sun, 18 May 2008 21:52:00 GMT

We have put together a survey to get a bit more input from the community about Acceptance Testing. If you are at all involved in software development, deployment, or operations acceptance testing proabably impacts you in some way, shape, or form, so you may want to add your insight to what we are working on.

You can find it here: Acceptance Testing Survey

Thanks for the feedback and input.

Acceptance Test Engineering Guidance

mpuleio — Sat, 19 Apr 2008 00:48:23 GMT

As I mentioned before (Acceptance Testing Guidance, Grigori's Take on the Acceptance Testing Project), I am working on guidance around acceptance testing with Grigori.

We now have alive CodePlex site that we can work from: patterns & practices: Acceptance Test Engineering Guidance

We will be posting more as we really get ramped up. We have a LOT of ideas that we will post and talk about. We will also post bi-weekly (or close) drops, probably in the form of Word documents, showing what we have so far, and asking for input.

Grigori's Take on the Acceptance Testing Project

mpuleio — Wed, 09 Apr 2008 00:42:17 GMT

The other day, I mentioned a new project here at p&p, Acceptance Testing Guidance,

Grigori has his own take on the project, Acceptance Test Engineering Guidance - project kick-off.

Check it out.

Acceptance Testing Guidance

mpuleio — Fri, 04 Apr 2008 00:52:25 GMT

On Monday, Grigori and I kicked off a new project: Acceptance Testing Guidance

Actually, we kicked off a pre-project planning project. In two to three weeks, we need to decide if there is enough demand for guidance on acceptance testing to warrant a real project, determine the scope of the project, generate a preliminary backlog for the project, and a rough release plan, and ensure we understand the space enough to move forward, and estimate a budget and timeframe.

We have some ideas, but we are open to input. I have a few questions for the world to help us get started:

What kind of guidance around testing, and particularly acceptance testing, would you like to see?
What questions do you want us to answer around testing and acceptance testing?
Do you think there is a need for guidance in this area?

Please use the Comments feature to reply, and I will consolidate responses. We will also have a survey (soon I hope) that I will post links to.

Weird COM Exceptions in SCSF/CAB Unit Tests

mpuleio — Wed, 19 Mar 2008 22:17:28 GMT

The SCSF project was getting a weird error with the unit tests that were brought forward from VS2005 to VS2008.

All the unit tests would pass, however the test run would fail with the following error in a special log file:

One of the background threads threw exception: System.Runtime.InteropServices.InvalidComObjectException: COM object that has been separated from its underlying RCW cannot be used.
at System.Windows.Input.TextServicesContext.StopTransitoryExtension()
at System.Windows.Input.TextServicesContext.Uninitialize(Boolean appDomainShutdown)
at System.Windows.Input.TextServicesContext.OnAppDomainUnloaded(Object sender, EventArgs args)

Of course, we had hundreds of these messages, and no way to correlate them with a specific unit test.

After some work, I was able to find a single repro case. After some research, it looks like any unit test that creates a Window of any type will do this. The fix is to add a TestCleanup method to your fixture like this one:

// Added a TestCleanup method to deal with the fact that the code was throwing an InvalidComObjectException
// with the information "COM object that has been separated from its underlying RCW cannot be used."
// Fix is based on this bug logged on Connect.Microsoft.Com:
// http://connect.microsoft.com/VisualStudio/feedback/ViewFeedback.aspx?FeedbackID=318333
[TestCleanup]
public void CleanUp()
{
System.Windows.Threading.Dispatcher.CurrentDispatcher.InvokeShutdown();
}

I know that unit tests that create Windows are not ideal, but for things like CAB and SCSF ( code bases I inherited) it may be necessary.

I hope this helps anyone else who runs into this issue.

Web Client Software Factory February 2008 Ships!

mpuleio — Thu, 28 Feb 2008 22:25:48 GMT

I know that folks have been reading about the Web Client Software Factory on Blaine's blog and on Glenn's blog. Blaine posted about what we planned to include, back in December with Next version of Web Client Software Factory (WCSF). Glenn had the post at the end of January, Web Client 2.0 closer than you think, and Web Client 2.0, what's the hold up?

So, what is in the latest version of the Web Client Software Factory?

Here is a quick blurb from the front page of our docs, with my comments in Blue:

The February 2008 release of the Web Client Software Factory is an update to the June 2007 release. The following are the major changes:

Added user interface responsiveness guidance. The guidance includes documentation, Web controls, QuickStarts, and a new reference implementation that demonstrate how to incorporate Microsoft ASP.NET AJAX technologies in your Web applications to provide a richer user interface experience.
Added support for the Model-View-Presenter pattern in user controls and master pages. The Composite Web Application Block includes a new Dependency Injection mechanism that facilitates the implementation of the Model-View-Presenter pattern in Web controls and master pages. The guidance package also includes new recipes that help developers create master pages and user controls that implement the Model-View-Presenter pattern. By using the Model-View-Presenter, developers can extend the testability surface to user controls and master pages.

Dependency Injection also works in ASMX Web Services hosted in a WCSF web application project

User controls can be reused across modules. Developers can build Web pages made up of user controls from different modules.

There is an example of this in the Order Management Reference Implementation

Updated the Composite Web Application Block. The main changes include the following:

Improved performance
Support for services registration through configuration
Support for type mapping for dependency injection

Type mapping was a no-brainer to add. We also needed it to show a few of the deeper concepts around MVP.

The performance improvements are the part of how the new Unity project works deep under the hood. Chris Tavares and worked with the WCSF team on the performance enhancements. Later, Chris went to start the Unity project, which has evolved into a great tool from what I have seen. I know the next question is "When will WCSF support Unity?" My answer is as soon as we can do a complete re-write. Out of the box, Unity provides a lot of functionality that CWAB provides. To properly use Unity, we would need to throw out a lot of the CWAB code, and re-do it.

The ability to add a service via config was requested by the community, and is shown in the Order Management Reference Implementation.

Updates to the Add Business Module and Add Foundational Module recipes. These recipes now include a new option to create a separate project for the modules’ public interface.
Updated the patterns documentation topics. The main changes include two new pattern description topics, Inversion of Control and Module Interface Separation, and it updates to the Model-View-Presenter topic.
Included additional guidance for several technical concepts. The technical concepts covered are views testability, modularity, autocomplete, validation, and search. The guidance consists of documentation, QuickStarts, Web controls, and How-to topics.

These concepts were shipped previously as "Bundles" on our CodePlex community. We will be releasing new, updated VS2008 versions of these bundles on MSDN in the near future, but all the content is in the factory. The bundles are so folks can get a small taste and look at a single feature or concept in isolation.

Added support for Visual Studio 2008 to the guidance package.

VS2008 and and .NET 3.5 support was asked for very loudly by customers.

Testing Guidance

Something we do not talk about enough is the fact that we are releasing some guidance on how to test web applications. First, since we do most development via ~~TDD~~ EDD (Example Driven Development), we have quite a bit in the way of unit tests. There are a few exceptions, where we intentionally do not include unit tests as we do not want to clutter a QuickStart example with another concept that may hinder understanding of what the QuickStart is about.

We are also shipping acceptance tests. In a few projects, these acceptance tests are manual test scripts in the form of a Visual Studio Manual Test. In others, we have completely automated the acceptance tests using

<TheLawyersMadeMeSayIt>

Note: this is a link to an external Web site that provides software that can be used to write acceptance tests. Please note that Microsoft is not responsible for the content of external Internet sites.

</TheLawyersMadeMeSayIt>

the WatiN (Web Application Testing in .NET).

To compile the test projects and run the tests, you will need VS2008 Professional or Team System, and to download the WatiN binaries, and copy the DLLs to the Lib folder (which our help describes with more info). The upside is that anyone can see how we are doing functional/acceptance testing of our sample applications. Of course, there is a lot more to testing than just acceptance testing, but it is a start.

Other resources

The WCSF page on MSDN
The WCSF Community on CodePlex
If you just want to peruse the docs, check out the WCSF Documentation download page

You will also want to check out what Blaine and Glenn say about the release:

Blaine's post: Web Client Software Factory Shipped
Glenn's post: Web Client Software Factory 2.0 Shipped

WCSF: Insights and Plans

mpuleio — Mon, 15 Oct 2007 21:36:57 GMT

There have been a lot of posts recently about WCSF, spurred on by our recent releases. Here are a few highlights with my comments about each:

Developer Hands-on Labs available for WCSF June 2007 Release - Blaine

These have been on the back burner. They had been "almost" done for quite a while, and are finally out the door.

Web Client Bundle Quality Checkpoints - Blaine

I worked with Blaine on drafting this. (I actually threatened to post it if he did not do so soon). It gives some insight into what we do around testing each and every one of our releases.

More Web Client goodness shipping, WCSF 2.0. - Glenn

A re-cap of what we have shipped recently, as well as future plans for the team.

How-To: Web Client Software Factory (WCSF) June 2007 & Visual Studio Orcas Beta 2 - Ezequiel

Necessary reading if you are running Orcas and want WCSF to work.

Unit testing is evolving

mpuleio — Fri, 21 Sep 2007 02:01:19 GMT

Jim Newkirk (of nUnit fame and one of the Microsoft folks behind www.CodePlex.com ) has been blogging about unit testing in forever ( most recently: Why you should not use Setup and Teardown in NUnit ).

Now, Jim and Brad Wilson are putting action to his words and codifying lessons learned over the years by developing xUnit.Net. Check out Jim's intro to this new tool at "Announcing xUnit.net".

Personally, I'll try using the shipping version on my own personal projects. At this point, it will probably not effect what I do at p&p.

Check it out and enjoy.

[Note: I added the testing flag since xUnit.NET claims to be extensible into an acceptance testing framework. Not that I consider EDD testing.]

Enterprise Library 3.0 will include a new block

mpuleio — Tue, 27 Feb 2007 03:26:00 GMT

Tom and Ed have complimentary articles on the Policy Injection Application Block:

This block is really impressive. You can do things like: "Every time I call a method on class foo, call this method before hand, invisibly". Very useful if you ever had to add auditing after the fact on a data access layer, or a requirement for perf counters came up at the last minute. (Not that I have ever had to deal with either of those....)

How could you use this for testing? Create a test interception handler, and :

mock out (either for acceptance or unit tests) the data acces layer.
Introduce faults anywhere in the program you want to (I want to throw an OutOfMemoryException .... there, OR no, that file no longer exists. bye bye.)

Another name for this block is the Give-me-enough-rope-to-hang-myself-and-everyone-else-working-on-this-project-Block. It is very powerful, but if you write your own handlers, you must be careful. "With great power, comes great responsibility."

Check it out in the Enterprise Library CTP and enjoy.

Thoughts on Code Coverage

mpuleio — Tue, 27 Feb 2007 02:44:00 GMT

Jim Newkirk and Brad Wilson have an interesting blog post on code coverage and what it means.

They have some interesting ideas to think about.

On my last project, I ran covereage numbers at fairly regular intervals. At one point, the trend sharpy dropped, and I had a conversation with the dev team, resulting in us fixing the problem. I guess I did what they are advocating, not with intent, but on accident.

On my current project, I want this reporting to become automatic, and not need to worry about communicating the problems with the team, but have them see the problem and take responsibility without intervention... We'll see how it turns out, if I can get the automation and reporting to work.

[Note: I know that TDD is not testing, and a better description is Example Driven Design. I threw the Testing tag in anyway, because this can apply to acceptance test code coverage as well as unit test coverage.]