random bits, rants and raves

  • allen@msft

    Remote Test Execution using Team Foundation Server 2015 RC2 and beyond



    The Agents for Visual Studio suite containing the test controller and agents allowed you to execute your tests on a set of remote machines by providing the right configuration in your .testsettings file. Based on the feedback we have received we are providing a new and simpler way for you to do your remote automated testing starting Team Foundation Server (TFS) 2015 RC2.

    In this blog post we will quickly setup a test rig for remote test execution using TFS 2015 RC2



    1. TFS 2015 RC2 installation with a Build vNext agent configured on a machine with internet access.
    2. A (virtual) machine to be used for testing with following requirements


    Step-By-Step Guide

    Head to your TFS homepage and navigate to the team project you want to work with. The below slides will guide you through this entire scenario.



    So what has changed

    There have been a lot of changes to get the stability and resiliency that you demanded in test execution. To summarize the top few improvements

    1. The test controller has been removed as the intermediate hop and the agents now directly communicate with the TFS server using HTTP.
    2. You can now run tests using 3rd party test frameworks like nunit/xunit/cpp using the relevant unit test adapter, similar to Visual Studio.
    3. Automated installation and configuration of your test agent on your test machines as a part of the Build workflow using our tasks in Build vNext.
    4. Improved reporting for automated runs in Test Management.


    What are our limitations

    1. We do distribution on a per test container basis ie if you have 2 machines and 2 test dlls we will execute all tests from testdll1 on machine1 and the rest from testdll2 on machine2.
    2. We don’t have support for workgroup/isolated/azure machines as yet. As long as you have http connectivity from the agent to TFS/Visual Studio Team Services and your build machine you will be good to go.
    3. We do not support Azure VMs as machines in the machine group used for testing
    4. Yes the scenario can be used in TFS/Visual Studio Team Services as well. However we do not support hosted build pools when using on-prem machines. For this case you would have to create a new build pool and register a new build agent with it.

     More up-to-date details @


    Why do we have a test agent and a build agent? Why can’t there just be a common agent?

    The build agent does provide you with unit testing facilities but only after you install Visual Studio on your build machine. The test agent has no such prerequisite. This is the sole reason why we have two agents instead of one. We are definitely looking at ways in which we no longer need to make such a distinction.

    Will I be able to run my java tests on my remote Linux/OSX machines using the test agent?

    Today the cross platform functionality is available in Build vNext. We are looking for the right data points before we build the remote testing scenario for other platforms.


    We hope that this post will allow you to get started with the new remote test execution workflow on your on-prem TFS installation. Try it out and let us know what you think


  • allen@msft

    Test controller/agent usage recommendations


    Over the years we have had many people ask us on what is the best way to configure and use the test controller and agent for remote test execution. Here is a summary of the best practices that we advocate.

    Setup and Settings

    If you are on the latest VS2012U1+ or VS2013 you can ignore step#1&2

    1. Ensure that Visual Studio 2010 SP1 is installed on all machines
    2. Dev10 SP1 -

    3. Next install the latest patch available

    5. Open "<VSINSTALLDIR>\Common7\IDE\QTControllerService.exe.config" and add the following entry under
    6. <add key="ControllerJobSpooling" value="false"/>

      • Restart Controller
    7. Agent setup
      • Install the latest patch available
      • Open "<VSINSTALLDIR>\Common7\IDE\QTAgentService.exe.config" and add the following

      <add key="RestartTestExecutionProcessForEachRun" value="true"/>

      • Restart Agent
    8. Client Setup
      • Install the latest patch available
      • Open "<VSINSTALLDIR>\common7\ide\mstest.exe.config and add the following key

      <add key="DeleteTestDeploymentFiles" value="yes" />

    9. In your test settings
      • Keep bucket size as 1000. This will result in a test run going to smaller set of agents.


          <Buckets size="1000"/>

    10. Set test run timeouts to ensure no runs go beyond acceptable limits
    11. <Execution>

      <Timeouts runTimeout="7200000" />

      7200000 = 2hrs specified in milliseconds

    12. Run your tests in parallel if your agents are on a multiproc system
    13. <Execution parallelTestCount=0>

    Other Recommendations

    1. Test Controller keeps all results for a run in memory. Therefore it is necessary that test runs be split into logical groups. We recommend you split tests into assemblies so that each test dll has <=5000 tests and/or run the TC in 64bit mode
      1. testcontrollerconfig.exe configure <… configuration parameters… > /platform64

    2. When any component of the Test Rig (any of the agents or controller) faces an issues (network disconnect, deployment failures, test crashes), the current test run is aborted. The probability of the test run getting aborted due to failure in agents reduces when we send one test run to a smaller set of agents. Keep a run limited to minimum number of test assemblies
    3. To scale out you should be using multiple such rigs and breaking up your test runs amongst these rigs as required.
  • allen@msft

    Empty .coverage file with profiler related errors in the event logs


    ​If you find yourself with a an empty .coverage file and see errors similar to the below in your event logs you most probably have a corrupt install

    (info) .NET Runtime version 4.0.30319.17929 - The profiler has requested that the CLR instance not load the profiler into this process. Profiler CLSID: '{b19f184a-cc62-4137-9a6f-af0f91730165}'. Process ID (decimal): 12624. Message ID: [0x2516].

    (Error) TraceLog Profiler failed in initialization due to a lack of instrumentation methods, process vstest.executionengine.x86.exe



    a) Environment variable VS110COMNTOOLS is set to <vsinstalldir>\common7\tools

    b) Regkey HKLM\SOFTWARE\Microsoft\VisualStudio\11.0\InstallDir is set to your <vsinstalldir>\Common7\IDE\

    c) covrun32.dll and covrun64.dll exist in "<vsinstalldir>\Team Tools\Dynamic Code Coverage"

  • allen@msft

    Test Controller 2012 Update 3+


    New features introduced

    1. Visual Studio Test Controller 2012 Update3+ now supports backward compat with TFS servers. So you can connect your 2013 test controller to TFS 2012+
    2. The test controller now works with hosted builds having server drop locations
  • allen@msft

    Getting the status of your test agents


    Users routinely want to check the status of the test agents against which they are scheduling their remote test execution. The existing way of doing this was to fire up VS and open the “Manage Test Controllers” dialog

    With Visual Studio 2013 you can now check the status of your agents on the controller itself using the commandline

    TestControllerConfig.exe status [/testController:<testControllerUri>]

    testController              URI of the test controller. Default is localhost:6901

    Sample output

    Microsoft (R) Visual Studio Test Controller Configuration Tool
    Version 12.0…. for Microsoft Visual Studio 2013
    Copyright (c) Microsoft Corporation.  All rights reserved.

    Total number of agents         : 1

    Status of agents :
    Agent name                     : vstfs:///…
    Agent status                   : Disconnected

    Summary of all agents :
    Ready                          : 0
    Running tests                  : 0
    Offline                        : 0
    Deploying build                : 0
    Disconnected                   : 1

    The agent status correspond to the existing known states listed here

  • allen@msft

    Getting started with Fakes using Visual Studio


    These are a good set of references to anyone looking into getting started with fakes

  • allen@msft

    Important timeout tweaks for controller agent


    Based on your network you may run into test run failures associated with the agent/controller communication. This primarily happens due to network latency, high data transfer between controller & agent or high cpu utilization on the agent. In such cases you can tweak some of the timeouts to improve the resiliency of the system

    Increase the timeout setting for Agent timeout:

    [Controller machine] In QTController.exe.config: <drive letter:>\Program Files (x86)\Microsoft Visual Studio <Visual Studio Version>\Common7\IDE\.

        <add key="AgentConnectionTimeoutInSeconds" value="120"/>

        <add key=" AgentConnectionTimeoutInSeconds" value="120"/>

        <add key="AgentDeploymentTimeout" value="300"/>

        <add key="AgentSyncTimeoutInSeconds " value="300"/>



    Increase the value to triple.

    [Agent machine] In QTAgentService.exe.config <drive letter:>\Program Files (x86)\Microsoft Visual Studio <Visual Studio Version>\Common7\IDE

                        Set ControllerConnectionPeriodInSeconds to 90 (making this 3 times too).

    Restart the services.

    Apart from this increasing the bucket size will help reducing communication too.

  • allen@msft



    Am a developer with the Lab Management team and work in the areas related to the setup and administration of the product. The intention is to publish out some random bits that could help other developers.

    Since I don't want my first post to be a total dud I'm adding a slighlty modified version of jmanning's infamous get-tfs.ps1 script. This introduces you to our namespace and allows you to access the public lab services. There are only 3 new lines added in here. HTH for an appetizer.

        [string] $serverName = $(throw 'serverName is required')

        # load the required dll

        $propertiesToAdd = (
            ('VCS', 'Microsoft.TeamFoundation.VersionControl.Client', 'Microsoft.TeamFoundation.VersionControl.Client.VersionControlServer'),
            ('WIT', 'Microsoft.TeamFoundation.WorkItemTracking.Client', 'Microsoft.TeamFoundation.WorkItemTracking.Client.WorkItemStore'),
            ('CSS', 'Microsoft.TeamFoundation', 'Microsoft.TeamFoundation.Server.ICommonStructureService'),
            ('GSS', 'Microsoft.TeamFoundation', 'Microsoft.TeamFoundation.Server.IGroupSecurityService'),
            ('LS',  'Microsoft.TeamFoundation.Lab.Client', 'Microsoft.TeamFoundation.Lab.Client.LabService'),
            ('LAS', 'Microsoft.TeamFoundation.Lab.Client', 'Microsoft.TeamFoundation.Lab.Client.LabAdminService')


        # fetch the TFS instance, but add some useful properties to make life easier
        # Make sure to "promote" it to a psobject now to make later modification easier
        [psobject] $tfs = [Microsoft.TeamFoundation.Client.TeamFoundationServerFactory]::GetServer($serverName)
        foreach ($entry in $propertiesToAdd) {
            $scriptBlock = '
                [System.Reflection.Assembly]::LoadWithPartialName("{0}") > $null
            ' -f $entry[1],$entry[2]
            $tfs | add-member scriptproperty $entry[0] $ExecutionContext.InvokeCommand.NewScriptBlock($scriptBlock)
        return $tfs

  • allen@msft

    Lab Management Dictionary


    This is an attempt to shed some light on the variety of nouns introduced by Lab Management, as is common with every new product. The list below primarily contains terms that you would encounter while accesing the sdk exposed by Lab Management ie this targets developers.

    SCVMM : System Center Virtual Machine Manager aka VMM

    Host Group [HG] : A logical entity which groups a set of hosts. Host groups can be hierarchial in nature and hosts cannot be shared across disjoint host groups.

    Library Server : A host which exposes UNC shares that would be used as resources to be managed by the SCVMM server.

    Library Share [LS] : A subset of the UNC paths on library servers that are explicity added to the SCVMM server

    Integration Components : A set of drivers installed on virtual machines to provide better integration with the host housing the virtual machine.

    Virtual Network : A virtual network allows you to configure various network topologies for virtual machines and hosts. Virtual networks can be external/internal/private

    Network Location : Each physical network adapter has network location associated with it eg. This is something new introduced in Vista.

    TeamProjectCollectionHostGroup [TPCHG] / TeamProjectCollectionLibraryShare [TPCLS] : Logical lab entities at the TeamProjectCollection level which map onto related SCVMM Host Groups and Library Shares. You would want to have a 1-1 mapping of a library share or host group on SCVMM to a TPCLS and TPCHG. It is not recommened to have multiple TPCLS/TPCHG objects mapping to the same LS/HG objects on SCVMM.

    For a given project collection you can add multiple such TPCHGs and TPCLSs

    TeamProjectHostGroup [TPHG] / TeamProjectLibraryShare [TPLS] : Logical entities at the TeamProject level which map onto TPCHG and TPCLS objects. Think of these objects being contained within their corresponding TPC objects

    For a given project you can add multiple such TPHGs and TPLSs

    Bird's View : The above objects are essentially a hierarchy of logical objects, one contained within another at most times, as shown here: 

    MyCollection {TeamProjectCollection}

    MyTPCHG1 {TeamProjectCollectionHostGroup -> internally maps to HG1 on SCVMM}

    MyTPCLS1 {TeamProjectCollectionLibraryShare -> internally maps to LS1 on SCVMM}

    MyProject {TeamProject}

    MyTPHG1 {TeamProjectHostGroup -> internally mapping to MyTPCHG1 and therefore to HG1 on SCVMM}

    MyTPLS1 {TeamProjectLibraryShare -> internally mapping to MyTPCLS1 and therefore to LS1 on SCVMM}


  • allen@msft

    Using the TFS - Best Practice Analyzer (BPA) for Lab Management


    As announced earlier the TFS Best Practice Analyzer (BPA) has been released for RC. You can install those bits from here.

    One of the new and useful features in the RC release is the ability to exclusively run the Lab Management checks.


    This new option will reduce your scan times when debugging lab related issues.

    We are looking for some early feedback on the existing functionality and suggestions moving forward. So download the bits and give it a whirl.

    Note: The pre-req for the lab scan is the VMM Admin Console 2008 R2

Page 1 of 2 (12 items) 12