Dominic Hopton's Blog : General

Where Have I been? Helping make Team System Rock!

dhopton — Sat, 15 Jan 2005 05:53:00 GMT

Well, It's been a long time since I blogged - actually, I've not blogged since we shipped our very first release just after Beta 1.

So, what’s been happening with the Testing tools in Team System? Well, I've still got Test View, Manual Testing and Test Project but I now also own Unit Test Debugging, Generic Testing (very useful) and Check-in policy support for Hatteras Source Code Control. But, I've lost Solution Integration - Michael (who owns Code Gen testing) now owns that.

In that time we've released a couple of CTPs that have included our bits - the most recent being The December CTP.

As you'd expect, all of the issues that I talked about in previous blogsicles have been fixed - we're looking in great shape right now. Some of the new functionality we have now in my feature areas are really compelling. Taking debugging as an example - enabling this has made my job of writing tests a lot easier. Just being able to set a break point in the test code and its hit - It's fantastic.

Generic testing enables you to integrate pre-existing test harness & result reporting into our infrastructure. For the full effect, you need to add some code to your test result reporting to output xml (we supply an XSD) with greater detail. Very powerful if you want to take advantage out our integrated solution but don’t want to port all your tests.

Check-in policy is quite possibly my favourite from a what-it-enables aspect - the ability to have a set of tests that must pass before a developer can check-in his code is fantastic. It's such a great way to help ensure that quality of code in the source repository is of a high standard.

I hope to blog more - it's one of my new years resolutions! I look forward to hearing comments from you, and also make sure you check on MSDN Chats for EDT chats.

No! My Run config got toasted!

dhopton — Tue, 25 May 2004 00:46:00 GMT

Another problem with the run configuration is that the template runconfig (The one that gets added to new solutions) can easily become overwritten with your user one, thus potentially screwing up test execution for all projects AND new projects. This only happens in a specific situation, but sadly this is a very common situation.

So, what happens? Well, when you create a new Test Project (not Empty Test project) you are prompted in the wizard to add a default run config to the solution. If you select this option the run config is added to the solution, and you’re good to go. The only problem here is the bug that adds the run config as a link rather than copying to the solution directory. Because of this, all changes to the run config made in this situation will result in those changes being reflected across ALL solutions that were created in the same way, and ALL new projects where you add a run config in any manner to the solution.

You can work around this simply by not using the wizard to add a run config to the solution, and instead letting the Run Config Dialog add the run config when you first run a test.

When you first go to execute tests when there is no run config associated with the solution, the dialog will not list anything in the “Run Configuration” Drop down – this is fine. Click the browse button, and select localtestrun.rcg. Click Open, then click OK. When prompted “Do you want to add this to the solution” select Yes, and all will go fine and dandy. If the file is not listed, navigate to %ProgramFiles%\Microsoft Visual Studio 8\Enterprise Developer Tools\Testing Tools\TestProjectCommon\Templates\RunConfig\1033 and the file should be located there.

If you manage to corrupt your localtestrun.rcg that resides in the project directory, you can just copy it off the install media again and you’ll be up and running again. This can be found in the %ProgramFiles%\Microsoft Visual Studio 8\Enterprise Developer Tools\Testing Tools\TestProjectCommon\Templates\RunConfig\1033 directory on the media – just copy this to the same path on the hard disk.

Setup Failure – It’s just throwing a tantrum

dhopton — Tue, 25 May 2004 00:43:00 GMT

One of the problems we came across during the last moments of testing for the CTP was a very rare situation where setup will tell you that it has failed, implying your screwed and will have a very messy setup. This is not what it seems at first :)

Specifically, this situation will only occur if you are asked to reboot after the CLR has been installed – something that doesn’t happen very often. When it does reboot there is a bug in setup that accidentally deletes some of the files setup needs to continue (Specifically the J# runtime installer), so it thinks that’s something has gone horribly wrong and errors out – DON’T PANIC!

When setup tells you it’s failed, ignore it. Click out of the failure and return to the 3-step step setup dialog. Click setup again and keep working through setup as before and it will install fine and dandy this time.

This will be fixed for Beta 1 of Whidbey I’m told – we caught it a little too late and didn’t want to put the CTP at risk from code churn. Additionally, it seems to rarely happen, so shouldn’t effect too many people. If it does, let me know (comments, mail, IM)

I turned off code coverage and I can’t run my tests anymore!

dhopton — Tue, 25 May 2004 00:43:00 GMT

This is a problem we only found out after we’d sent off the DVDs to be pressed, and turns out to easy to repro, but also easy to workaround. It’s also got a slightly strange workflow compared to the way we think a lot of people will be using run configs initially.

If you enable code coverage and then subsequently disable it – all through the Edit Run Configuration dialog (or opening & saving from the Run test run config dialog), you will possibly find that when you run your tests you will see that the test result is error/fail and the message is “FileNotFoundException” or “MethodNotFound”.

How can you workaround this problem and get back to running your unit tests? Well, there are two ways:

a) Copy the localtestrun.rcg from the product directory over the one in your solution

b) Edit the .rcg file for your solution so that the XML element isNewLocalDeployment is set to false rather than true.

If you do either of these you will need to close your solution and re-open it to see the changes.

There is a simple way to avoid this happening. When working with run configuration, don’t use the “Edit Run Configuration” button on the toolbar to enable/disable code coverage. This is because using this dialog saves the changes to disk when you click OK & keep the changes. When you change settings in the Run Configuration dialog shown when you execute tests, the changes are only saved if you explicitly save them. Because of this subtle difference, you can change settings at runtime without screwing anything up.

Another option of source is to save the code coverage enabled runconfig to a separate file and just swap between them when you need to have or not have code coverage information.

One last thing on this topic – you may have a problem with a) because of another bug, which I’ll cover in another blogsicle.

Test View – your new most used window

dhopton — Tue, 25 May 2004 00:42:00 GMT

Let’s take a look at the Test View tool window – the place where you will be running your tests from as you develop and debug them. Test View provides you with a flat list of all your tests, with each test only listed once. From this window you can easily manipulate the properties of your tests, easily edit the test themselves, and execute one or more tests under a specific run configuration.

Let’s take a quick tour round the window.

At the top of the window we have the toolbar:

Group By Drop-down: This allows you to group your tests by the value of certain properties. With only 10-20 tests this isn’t really an issue, since you can navigate and find most of the items by sight. but when you’ve got 100’s of tests it’s must harder. With Group By, the tests are (duh) grouped by their property and those groups can be collapsed to just their header to save space. The properties listed are a union of all the properties available for all test types.

Run Tests: ‘nuff said. This will run your tests. Depending on your configuration this may show the Run Configuration dialog.

Customise: This allows you to add/remove columns from the test list, and also arrange the order of those columns. You can also arrange the order of the columns through drag & drop of the header columns in the test list.

Edit Run Config: This allows you to edit your run configuration settings and save those changes to the solution run configuration. This is a convenient way to change settings without running your tests.

Column Header: Exactly as you’d expect.

Test List: This is where you will see all of you your tests of all types listed. You can select one or more tests (of multiple types) and interact with them (run, edit, etc). From this window it is possible to drag & drop tests onto Test Explorer for easy categorization.

From this list, its possible to display a context menu. The items on this context menu can vary from test type to test type, but I’ll cover the basic items here:

Edit: Allows you edit a manual test. This will bring up word if you are editing a MHT test.

Goto Source Code: This will allow you to right click a test in test view and jump straight to the source code without hunting it down. Very useful.

Run Selection: As per the run button on the toolbar.

Delete: This only works for Manual Tests at the moment and removes (but doesn’t delete) the manual test from the project.

Open Test Explorer: Shows test explorer or brings to the front if it’s already open.

Who the heck are you?

dhopton — Tue, 25 May 2004 00:39:00 GMT

I’m Dominic, and I’m an SDE/T on the Enterprise Developer Tools team here at Microsoft, which is part of the bigger Visual Studio Team System product. Specifically within EDT, I work the quality tools team – we own the unit testing, manual testing, and general testing infrastructure. Within EDT there are also dynamic analysis and static analysis teams (Profiler, Prefix et al).

Day to day I own the testing of Test View, Manual Testing, Solution integration, Test Project and the Properties Window integration. Of all of those, Test View is the one that most people are gonna be using the most, but within all of those areas there is a lot of functionality. When I say solution integration I mean all that funky stuff that goes on under the hood inside VS to make our windows, and other bits seamlessly work.

I’m not sure what I’m gonna put on this blog yet – But I do know it’s doing to be almost entirely work related. I’ve got my own personal blog for non-work related things, and I like (or want to try) to keep my two lives separate :)

Anyway – if you want to chat about the product, leave comments, track back etc. If you feel like the need for a bit more interactive communication I’m on MSN as dominichopton@hotmail.com.

We want your feedback!

PS Please be gentle – this is the first public showing of our stuff. It’s only a technical preview so the edges are a little rough & things maybe broken – they won’t stay that way because we’re gonna make these tools ROCK!