allen@msft

random bits, rants and raves

Posts
  • allen@msft

    Test controller/agent usage recommendations

    • 2 Comments

    Over the years we have had many people ask us on what is the best way to configure and use the test controller and agent for remote test execution. Here is a summary of the best practices that we advocate.

    Setup and Settings

    If you are on the latest VS2012U1+ or VS2013 you can ignore step#1&2

    1. Ensure that Visual Studio 2010 SP1 is installed on all machines
    2. Dev10 SP1 - http://www.microsoft.com/en-us/download/details.aspx?id=23691

    3. Next install the latest patch available
    4. http://support.microsoft.com/kb/2801364

    5. Open "<VSINSTALLDIR>\Common7\IDE\QTControllerService.exe.config" and add the following entry under
    6. <add key="ControllerJobSpooling" value="false"/>

      • Restart Controller
    7. Agent setup
      • Install the latest patch available
      • Open "<VSINSTALLDIR>\Common7\IDE\QTAgentService.exe.config" and add the following

      <add key="RestartTestExecutionProcessForEachRun" value="true"/>

      • Restart Agent
    8. Client Setup
      • Install the latest patch available
      • Open "<VSINSTALLDIR>\common7\ide\mstest.exe.config and add the following key

      <add key="DeleteTestDeploymentFiles" value="yes" />

    9. In your test settings
      • Keep bucket size as 1000. This will result in a test run going to smaller set of agents.

      <Execution>

          <Buckets size="1000"/>

    10. Set test run timeouts to ensure no runs go beyond acceptable limits
    11. <Execution>

      <Timeouts runTimeout="7200000" />

      7200000 = 2hrs specified in milliseconds

    12. Run your tests in parallel if your agents are on a multiproc system
    13. <Execution parallelTestCount=0>

      http://blogs.msdn.com/b/vstsqualitytools/archive/2009/12/01/executing-unit-tests-in-parallel-on-a-multi-cpu-core-machine.aspx

    Other Recommendations

    1. Test Controller keeps all results for a run in memory. Therefore it is necessary that test runs be split into logical groups. We recommend you split tests into assemblies so that each test dll has <=5000 tests and/or run the TC in 64bit mode
      1. testcontrollerconfig.exe configure <… configuration parameters… > /platform64

    2. When any component of the Test Rig (any of the agents or controller) faces an issues (network disconnect, deployment failures, test crashes), the current test run is aborted. The probability of the test run getting aborted due to failure in agents reduces when we send one test run to a smaller set of agents. Keep a run limited to minimum number of test assemblies
    3. To scale out you should be using multiple such rigs and breaking up your test runs amongst these rigs as required.
  • allen@msft

    Empty .coverage file with profiler related errors in the event logs

    • 1 Comments

    ​If you find yourself with a an empty .coverage file and see errors similar to the below in your event logs you most probably have a corrupt install

    (info) .NET Runtime version 4.0.30319.17929 - The profiler has requested that the CLR instance not load the profiler into this process. Profiler CLSID: '{b19f184a-cc62-4137-9a6f-af0f91730165}'. Process ID (decimal): 12624. Message ID: [0x2516].

    (Error) TraceLog Profiler failed in initialization due to a lack of instrumentation methods, process vstest.executionengine.x86.exe

     

    Check

    a) Environment variable VS110COMNTOOLS is set to <vsinstalldir>\common7\tools

    b) Regkey HKLM\SOFTWARE\Microsoft\VisualStudio\11.0\InstallDir is set to your <vsinstalldir>\Common7\IDE\

    c) covrun32.dll and covrun64.dll exist in "<vsinstalldir>\Team Tools\Dynamic Code Coverage"

  • allen@msft

    Test Controller 2012 Update 3+

    • 0 Comments

    New features introduced

    1. Visual Studio Test Controller 2012 Update3+ now supports backward compat with TFS servers. So you can connect your 2013 test controller to TFS 2012+
    2. The test controller now works with hosted builds having server drop locations
  • allen@msft

    Getting the status of your test agents

    • 0 Comments

    Users routinely want to check the status of the test agents against which they are scheduling their remote test execution. The existing way of doing this was to fire up VS and open the “Manage Test Controllers” dialog

    With Visual Studio 2013 you can now check the status of your agents on the controller itself using the commandline

    Syntax-
    TestControllerConfig.exe status [/testController:<testControllerUri>]

    testController              URI of the test controller. Default is localhost:6901

    Sample output

    Microsoft (R) Visual Studio Test Controller Configuration Tool
    Version 12.0…. for Microsoft Visual Studio 2013
    Copyright (c) Microsoft Corporation.  All rights reserved.

    Total number of agents         : 1

    Status of agents :
    Agent name                     : vstfs:///…
    Agent status                   : Disconnected


    Summary of all agents :
    Ready                          : 0
    Running tests                  : 0
    Offline                        : 0
    Deploying build                : 0
    Disconnected                   : 1

    The agent status correspond to the existing known states listed here

  • allen@msft

    Getting started with Fakes using Visual Studio

    • 0 Comments

    These are a good set of references to anyone looking into getting started with fakes

    http://www.peterprovost.org/blog/2012/04/15/visual-studio-11-fakes-part-1/

    http://www.peterprovost.org/blog/2012/04/25/visual-studio-11-fakes-part-2/

    http://www.peterprovost.org/blog/2012/11/29/visual-studio-2012-fakes-part-3/

  • allen@msft

    Important timeout tweaks for controller agent

    • 0 Comments

    Based on your network you may run into test run failures associated with the agent/controller communication. This primarily happens due to network latency, high data transfer between controller & agent or high cpu utilization on the agent. In such cases you can tweak some of the timeouts to improve the resiliency of the system

    Increase the timeout setting for Agent timeout:

    [Controller machine] In QTController.exe.config: <drive letter:>\Program Files (x86)\Microsoft Visual Studio <Visual Studio Version>\Common7\IDE\.

        <add key="AgentConnectionTimeoutInSeconds" value="120"/>

        <add key=" AgentConnectionTimeoutInSeconds" value="120"/>

        <add key="AgentSyncTimeoutInSeconds " value="300"/>

    Increase the value to triple.

    [Agent machine] In QTAgentService.exe.config <drive letter:>\Program Files (x86)\Microsoft Visual Studio <Visual Studio Version>\Common7\IDE

                        Set ControllerConnectionPeriodInSeconds to 90 (making this 3 times too).

    Restart the services.

    Apart from this increasing the bucket size will help reducing communication too.

  • allen@msft

    whoami

    • 0 Comments

    Am a developer with the Lab Management team and work in the areas related to the setup and administration of the product. The intention is to publish out some random bits that could help other developers.

    Since I don't want my first post to be a total dud I'm adding a slighlty modified version of jmanning's infamous get-tfs.ps1 script. This introduces you to our namespace and allows you to access the public lab services. There are only 3 new lines added in here. HTH for an appetizer.

    param(
        [string] $serverName = $(throw 'serverName is required')
    )

    begin
    {
        # load the required dll
        [void][System.Reflection.Assembly]::LoadWithPartialName("Microsoft.TeamFoundation.Client")
        [void][System.Reflection.Assembly]::LoadWithPartialName("Microsoft.TeamFoundation.Lab.Client")

        $propertiesToAdd = (
            ('VCS', 'Microsoft.TeamFoundation.VersionControl.Client', 'Microsoft.TeamFoundation.VersionControl.Client.VersionControlServer'),
            ('WIT', 'Microsoft.TeamFoundation.WorkItemTracking.Client', 'Microsoft.TeamFoundation.WorkItemTracking.Client.WorkItemStore'),
            ('CSS', 'Microsoft.TeamFoundation', 'Microsoft.TeamFoundation.Server.ICommonStructureService'),
            ('GSS', 'Microsoft.TeamFoundation', 'Microsoft.TeamFoundation.Server.IGroupSecurityService'),
            ('LS',  'Microsoft.TeamFoundation.Lab.Client', 'Microsoft.TeamFoundation.Lab.Client.LabService'),
            ('LAS', 'Microsoft.TeamFoundation.Lab.Client', 'Microsoft.TeamFoundation.Lab.Client.LabAdminService')

        )
    }

    process
    {
        # fetch the TFS instance, but add some useful properties to make life easier
        # Make sure to "promote" it to a psobject now to make later modification easier
        [psobject] $tfs = [Microsoft.TeamFoundation.Client.TeamFoundationServerFactory]::GetServer($serverName)
        foreach ($entry in $propertiesToAdd) {
            $scriptBlock = '
                [System.Reflection.Assembly]::LoadWithPartialName("{0}") > $null
                $this.GetService([{1}])
            ' -f $entry[1],$entry[2]
            $tfs | add-member scriptproperty $entry[0] $ExecutionContext.InvokeCommand.NewScriptBlock($scriptBlock)
        }
        return $tfs
    }

  • allen@msft

    Lab Management Dictionary

    • 0 Comments

    This is an attempt to shed some light on the variety of nouns introduced by Lab Management, as is common with every new product. The list below primarily contains terms that you would encounter while accesing the sdk exposed by Lab Management ie this targets developers.

    SCVMM : System Center Virtual Machine Manager aka VMM

    Host Group [HG] : A logical entity which groups a set of hosts. Host groups can be hierarchial in nature and hosts cannot be shared across disjoint host groups.

    Library Server : A host which exposes UNC shares that would be used as resources to be managed by the SCVMM server.

    Library Share [LS] : A subset of the UNC paths on library servers that are explicity added to the SCVMM server

    Integration Components : A set of drivers installed on virtual machines to provide better integration with the host housing the virtual machine.

    Virtual Network : A virtual network allows you to configure various network topologies for virtual machines and hosts. Virtual networks can be external/internal/private

    Network Location : Each physical network adapter has network location associated with it eg. xyz.com. This is something new introduced in Vista.

    TeamProjectCollectionHostGroup [TPCHG] / TeamProjectCollectionLibraryShare [TPCLS] : Logical lab entities at the TeamProjectCollection level which map onto related SCVMM Host Groups and Library Shares. You would want to have a 1-1 mapping of a library share or host group on SCVMM to a TPCLS and TPCHG. It is not recommened to have multiple TPCLS/TPCHG objects mapping to the same LS/HG objects on SCVMM.

    For a given project collection you can add multiple such TPCHGs and TPCLSs

    TeamProjectHostGroup [TPHG] / TeamProjectLibraryShare [TPLS] : Logical entities at the TeamProject level which map onto TPCHG and TPCLS objects. Think of these objects being contained within their corresponding TPC objects

    For a given project you can add multiple such TPHGs and TPLSs


    Bird's View : The above objects are essentially a hierarchy of logical objects, one contained within another at most times, as shown here: 

    MyCollection {TeamProjectCollection}

    MyTPCHG1 {TeamProjectCollectionHostGroup -> internally maps to HG1 on SCVMM}

    MyTPCLS1 {TeamProjectCollectionLibraryShare -> internally maps to LS1 on SCVMM}

    MyProject {TeamProject}

    MyTPHG1 {TeamProjectHostGroup -> internally mapping to MyTPCHG1 and therefore to HG1 on SCVMM}

    MyTPLS1 {TeamProjectLibraryShare -> internally mapping to MyTPCLS1 and therefore to LS1 on SCVMM}

     

  • allen@msft

    Using the TFS - Best Practice Analyzer (BPA) for Lab Management

    • 0 Comments

    As announced earlier the TFS Best Practice Analyzer (BPA) has been released for RC. You can install those bits from here.

    One of the new and useful features in the RC release is the ability to exclusively run the Lab Management checks.

    scantypes

    This new option will reduce your scan times when debugging lab related issues.

    We are looking for some early feedback on the existing functionality and suggestions moving forward. So download the bits and give it a whirl.

    Note: The pre-req for the lab scan is the VMM Admin Console 2008 R2

  • allen@msft

    Lab Management is now available

    • 0 Comments

    Visual Studio 2010 Lab Management is now available for download.

     

    So how do you get started?

    Install Visual Studio 2010 Team Foundation Server 10.0 and SCVMM Admin Console on a machine.

    Set up a SCVMM Server on a machine.

    Install Visual Studio 2010 Ultimate sku to get the Microsoft Test Manager client and the agents skus. Find the detailed steps @ the msdn download page

    Then download and apply the patch on all your client and server machines (available here). This has all the right fixes making it sturdy to be used in your live environments.

     

    Soma is excited and so is Brian and so should you. Try it out and send us your feedback while we keep working hard on the next release.

Page 1 of 2 (11 items) 12