There was a interesting question posted in my comments on the TRX file format, about being able to republish results to TFS Server. This is an oft requested behavior, and the short answer is "no, you can't". Once you've published it, it's there forever. You can, technically, delete an re-publish, but in the scenario DmytroL was looking at, it ain't gonna help. This is resolved in the next big release, which one can see a preview of this in the April Rosario CTP, but in that release, is focused on using the new manual test tool to republish results.

However, it is possible to create a solution with some coding to merge the TRX files (something we don't have in the product).

Lets work through DmytroL's questions:

1. Is it possible to publish several TRX files under the same test run (of course provided all the TRX files in question specify the same correct test run GUID in the TestRun tag)?

No. You can't. If you try to publish a run that was already published (irrespective of the actual contents), we will block you from publishing -- you'll receive an error that the run has already been published.

2. Is it permitted to overwrite the outcome of a certain test within the same test run?

No. There needs to be one outcome. While (With some simple testing i just did), it is possible to have the same test in the results file, with different executionId's, this is completely unsupported. I would expect things to become pretty sticky on the server if you published a file like this (if you have a test server, you could try... YMMV, everything may break, no I won't help you if it takes down your production server...)

3. Are there any potential problems if these TRX files are published by different people?

See (1) & (2) :)

The scenario behind all this is: a test run is a collection of manual tests put together to verify certain functionality (I know there's a notion of test list for this, but one can't slice the TFS warehouse cube by test lists). Now, subsets of the test run are assigned to different QA engineers for execution. So, each QA engineer goes through her "share" of tests and publishes the results - but the results from all QA Engineers involved should go under the same test run.

Write a tool that loads the results from each run, and then sorts them by time, and pick the post recent result (or whatever criteria you decide on), and output a single TRX, and publish

Also, it's important to note that you can slice the warehouse the way you want to -- when you publish multiple runs against one build (which is allowed, supported and expected), the roll up data goes into the warehouse with the most recent result being the "roll up" result for that test.

One of the questions that comes up internally, and externally is "What are these magic GUID attributes in the file? How can I generate them?". This is often from people who are creating tools to export TRX files from some sort of private test hardness, or generate VSMDI files in an automated fashion.

Note, that the XSD schemas are available with all visual studio installs in the:

%VSINSTALLDIR%\xml\Schemas\vstst.xsd file

directory, along with many other schemas.

TRX Files

Lets take a look at TRX files first. I'm not going to go through the whole file line by line -- we put a lot of info into the file, and some of it can be easily guessed. Things in the TRX file other than the obvious "results":

• Runconfig - we just dump in the whole run config that was use as part of that run
• Test List info - The test list info etc that was associated with the tests as they were run. If you just run the test from test view, or use the keyboard shortcuts (Ctrl-R, T), you won't see the test list info.

Neither of these pieces of information are required per-se, we just dump them in there for reference reasons (mostly for re-running without having all the original information).

Interesting lines:

<TestRun id="774eb9dc-b311-4cdd-9987-48dd12f1e5f4" />

The GUID here is just a randomly generated GUID -- Guid.New().ToString() is good enough here, it's just used to allows us to identify the run compared to any other arbitrary run.

<TestRunConfiguration name="Local Test Run" id="3b7e71cb-2506-4724-9b77-b3f5d5c4dd70"/>

This GUID is ID of the Test Run Config that was copied from the one specified at the time the run was created. This should match, but there's no specific need -- we don't go and look up that GUID later.

<UnitTest id="59863143-3238-122e-4d8a-637b491cc755"/>

This is where it gets interesting. This element, which is found under the TestDefinitions element enumerates/declares all the tests that are part of this run. Tests have a name, which is used to provide a friendly way for users to identify test cases from each other. However, the tests themselves need to be uniquely identified within the test system. Most test types (all but unit, for the ones that are shipped in-box) are identified using a random GUID, which is stored in the file the test is stored in. Example: Open up a word based manual test, and look on the properties of it -- you'll see an item which has a GUID in it, and is called "ID". The ID here on these child elements (children of TestDefinitions) is that same GUID as found the test. However, this is not the case for unit tests. Unit tests are generated using a hash fully qualified class name, and the method name [1] (i.e. Company.Product.TestClass.TestMethod).

<Execution id="958d14e2-2f82-4f2c-8e91-84543ca85428" />

This is another "random" GUID, however it is referred to later in the file, specifically as part of the results and tracking which tests were executed, so make sure that if you're generating here, your not just throwing the GUID out of the window after you generate it.

<TestLists>
<TestList name="All Loaded Results" id="19431567-8539-422a-85d7-44ee4e166bda" />
</TestLists>

This is similar to the test run configuration section, where it's a dupe of the actual test list file. This contains all the test lists that are in this run, with the name (as we can see), and the GUID -- again, this really is just a random GUID. But as per <Execution />, it's used later in the file.

<TestEntry testId="59863143-3238-122e-4d8a-637b491cc755" executionId="958d14e2-2f82-4f2c-8e91-84543ca85428" testListId="8c84fa94-04c1-424b-9868-57a2d4851a1d" />

Test Entry provides a match between test lists, the test case, and the test result. Note that these GUIDs here refer to the IDs previously seen throughout this post. Make sure these GUIDs match up, and your results will all be peachy.

<UnitTestResult executionId="958d14e2-2f82-4f2c-8e91-84543ca85428" testId="59863143-3238-122e-4d8a-637b491cc755" />

These are the results (whose elements are determined by the test type that was executed, hence the 'UnitTestResult'), and as one can see, there are some guids here, which tally back to elements & ID's that were declared earlier in the file -- specifically the execution ID, and the test ID (since this is a unit test, the computed ID).

VSMDI Files

The VSMDI file format is much simpler, although equally filled with GUIDs :). The basic format here is the runconfig (don't ask why it's in there, it's got long historical reasons, and does allow for an... interesting scenario [2]) attached to the of the test lists (which is the same GUID no matter what), and then all the other test lists in the file, with their own GUID, a guid linking to a parent test list, and the contents of that list itself. Lets look at an example.

<TestList name="Lists of Tests" id="8c43106b-9dc1-4907-a29f-aa66a61bf5b6">

This is the "Root" test list, the one you see at the root of test list editor window - an immovable, unchangeable test list. This GUID is constant and the same in all VSMDI files, and all machines. It's been the same GUID since we shipped the first beta of VS 2005 (amusing to think that was over 4 years ago). If you don't have this, then things are going to go a bit pete tong.

Other test lists take the same form, but with there own GUID, which is merely a randomly generated GUID:

<TestList name="SampleList" id="e3d327f8-49dd-4ee0-b8c0-57e9b47d6cff" parentListId="8c43106b-9dc1-4907-a29f-aa66a61bf5b6">

Here, we can see that this node is a child of the parent node (Lists of Tests) by noting the parentListId GUID is the same as the ID guid of the "Lists of Tests" test list. This allows you to establish many levels of nesting of test lists to create the right grouping and categorization you need. However, for tests to appear in these test lists, you need to add child elements of this TestList element:

<TestLink id="59863143-3238-122e-4d8a-637b491cc755" />

Here, the ID refers to the test id of the test being linked in. If you look back through this article you can see the ID of this test showing in the TRX file section -- these IDs really do identify the test across multiple stores (even into the TFS warehouse if you push your results there).

Conclusion

There you have it, a whirl wind tour of the TRX and VSMDI file formats -- which should provide some insight for understanding the rest of our file formats. Note that while some of the information here is still relevant to VS 2005, the file format in that version is radically different, and requires some what unnatural acts to be able to generate it from outside of VS -- I would recommend upgrading to 2008 if you wish to do that :)

[1] We hash the fully qualified test method with code like this:

using System;
using System.Diagnostics;
using System.Security.Cryptography; 

namespace TeamTestSample
{
    internal static class UnitTestIdHash
    {
        private static HashAlgorithm s_provider = new SHA1CryptoServiceProvider(); 

        internal static HashAlgorithm Provider
        {
            get { return s_provider; }
        } 

        /// 

        /// Calculates a hash of the string and copies the first 128 bits of the hash
        /// to a new Guid.
        /// 
        internal static Guid GuidFromString(string data)
        {
            Debug.Assert(!String.IsNullOrEmpty(data);
            byte[] hash = Provider.ComputeHash(System.Text.Encoding.Unicode.GetBytes(data)); 

            // Guid is always 16 bytes
            Debug.Assert(Guid.Empty.ToByteArray().Length == 16, "Expected Guid to be 16 bytes"); 

            byte[] toGuid = new byte[16];
            Array.Copy(hash, toGuid, 16); 

            return new Guid(toGuid);
        }
    }
}

[2] One little know feature of the testing tools inside visual studio, is that you can re-run a test run without sources, just by picking up the test results directory, and opening the TRX file, and clicking the run button. What is even less known is that if you have the Test storages (the dlls, webtest, etc), and the VSMDI, you can open the VSMDI directly (no project, or solution), and select the tests to run. The test run configuration that will be applied will the last test run configuration that was active when the vsmdi was last edited.

I was writing some unit tests yesterday, and needed to test a private method. As I was gleely tapping in the code to call my static method using PrivateType, I noticed that two of my parameters were "out" parameters. I immediately went 'Hmm, I bet private type doesn't do anything special for those parameters'. I cast my mind back to a bug I fixed with the new Publicize tool in VS2008 where by ref/out parameters that were actually accessor types were not being 'unwrapped' and passed back out to the test code. Turns out we weren't doing the right thing here, but in fact reflection was helping us.

Reflection, when you give it the params array to Invoke, it actually automatically unpacks (although, in reality, I don't think there is any unpacking going on) the out/ref's into the same array, so you can get back at them. This means that since Private[Type|Object] are just simple wrappers around reflection, their Invoke methods do the work for you, and in fact everything works for free.

In the case of accessors, the experience is (now) the same. But it wasn't a complete no-brainer to get it to work. See, when we call the private method from the accessor, we have to unpack all the accessor types into their raw type, to pass to the actual method. This means that when the invoke returns, the 'raw' array has those ref/out values. But before we can return to the test code (the real caller), we have to 'repack' the instances into accessor types using the attach functionality of PrivateObject. It's a bit hairy when it comes to value types rather than ref types, but ultimately it's all good.

Some of the team are at the STARWest conferance in Anaheim this week. We're here for the two "Tutorial" days, and the full conference itself. While we're not here to demo, or hang out in the expo hall (We're just regular attendies), we'd love to meet up with any customers and users out there to talk about Team Test and Team foundation.

Is there any interest in this? If there is leave a comment on the blog, and if theres enough response we'll sort a time and place to meet up.

We can talk about the product, any problems you might have had, and if we're lucky even demo some of the team foundation features.

Let us know!

Dominic

[X-Posted from the VSTS Quality Tools Blog]

Ok, So I’d thought about this blog yesterday but discounted it ‘cause its not really my area. But what the heck :)

There is a very cool feature, which isn’t fully surfaced yet in the UI. This allows you to define new properties for your tests, which can be shown in the property window. Under test properties you’ll see “Test Specific” group, which shows your new property.

Right now, the bit that is missing is the ability to group-by & filter on these properties rather than just the core set. I just talked to the Dev who owns Test View (since our PM is at TechEd), and I was remembering right – our current plan is to allow you to filter, group by and add columns for those properties to the test view. Also, the property window is going to allow you to edit these properties and round trip them to code – so even if you don’t know the language your tests are in you can manipulate the custom properties.

Cool, isn’t it? :)

Another problem with the run configuration is that the template runconfig (The one that gets added to new solutions) can easily become overwritten with your user one, thus potentially screwing up test execution for all projects AND new projects. This only happens in a specific situation, but sadly this is a very common situation.

So, what happens? Well, when you create a new Test Project (not Empty Test project) you are prompted in the wizard to add a default run config to the solution. If you select this option the run config is added to the solution, and you’re good to go. The only problem here is the bug that adds the run config as a link rather than copying to the solution directory. Because of this, all changes to the run config made in this situation will result in those changes being reflected across ALL solutions that were created in the same way, and ALL new projects where you add a run config in any manner to the solution.

You can work around this simply by not using the wizard to add a run config to the solution, and instead letting the Run Config Dialog add the run config when you first run a test.

When you first go to execute tests when there is no run config associated with the solution, the dialog will not list anything in the “Run Configuration” Drop down – this is fine. Click the browse button, and select localtestrun.rcg. Click Open, then click OK. When prompted “Do you want to add this to the solution” select Yes, and all will go fine and dandy. If the file is not listed, navigate to %ProgramFiles%\Microsoft Visual Studio 8\Enterprise Developer Tools\Testing Tools\TestProjectCommon\Templates\RunConfig\1033 and the file should be located there.

If you manage to corrupt your localtestrun.rcg that resides in the project directory, you can just copy it off the install media again and you’ll be up and running again. This can be found in the %ProgramFiles%\Microsoft Visual Studio 8\Enterprise Developer Tools\Testing Tools\TestProjectCommon\Templates\RunConfig\1033 directory on the media – just copy this to the same path on the hard disk.

This is a problem we only found out after we’d sent off the DVDs to be pressed, and turns out to easy to repro, but also easy to workaround. It’s also got a slightly strange workflow compared to the way we think a lot of people will be using run configs initially.

If you enable code coverage and then subsequently disable it – all through the Edit Run Configuration dialog (or opening & saving from the Run test run config dialog), you will possibly find that when you run your tests you will see that the test result is error/fail and the message is “FileNotFoundException”  or “MethodNotFound”.

How can you workaround this problem and get back to running your unit tests? Well, there are two ways:

a)      Copy the localtestrun.rcg from the product directory over the one in your solution

b)      Edit the .rcg file for your solution so that the XML element isNewLocalDeployment is set to false rather than true.

If you do either of these you will need to close your solution and re-open it to see the changes.

There is a simple way to avoid this happening. When working with run configuration, don’t use the “Edit Run Configuration” button on the toolbar to enable/disable code coverage. This is because using this dialog saves the changes to disk when you click OK & keep the changes. When you change settings in the Run Configuration dialog shown when you execute tests, the changes are only saved if you explicitly save them. Because of this subtle difference, you can change settings at runtime without screwing anything up.

Another option of source is to save the code coverage enabled runconfig to a separate file and just swap between them when you need to have or not have code coverage information.

One last thing on this topic – you may have a problem with a) because of another bug, which I’ll cover in another blogsicle.

I’m Dominic, and I’m an SDE/T on the Enterprise Developer Tools team here at Microsoft, which is part of the bigger Visual Studio Team System product. Specifically within EDT, I work the quality tools team – we own the unit testing, manual testing, and general testing infrastructure. Within EDT there are also dynamic analysis and static analysis teams (Profiler, Prefix et al).

Day to day I own the testing of Test View, Manual Testing, Solution integration, Test Project and the Properties Window integration. Of all of those, Test View is the one that most people are gonna be using the most, but within all of those areas there is a lot of functionality. When I say solution integration I mean all that funky stuff that goes on under the hood inside VS to make our windows, and other bits seamlessly work.

I’m not sure what I’m gonna put on this blog yet – But I do know it’s doing to be almost entirely work related. I’ve got my own personal blog for non-work related things, and I like (or want to try) to keep my two lives separate :)

Anyway – if you want to chat about the product, leave comments, track back etc. If you feel like the need for a bit more interactive communication I’m on MSN as dominichopton@hotmail.com.

We want your feedback!

PS Please be gentle – this is the first public showing of our stuff. It’s only a technical preview so the edges are a little rough & things maybe broken – they won’t stay that way because we’re gonna make these tools ROCK!