Visual Studio 2005 Team Edition for Software Testers
Visual Studio 2005 Team Test Load Agent
Summary: A deeper Look at settings and how the controller in Microsoft Visual Studio Team Load Agent 2005 works.
The Microsoft Visual Studio Team Load Agent 2005 Controller (controller) is a Windows Service that can manage one or more agent machines. The configuration settings of a test run and attributes of the test cases in the run determine the actions and scheduling the controller will perform. This document will describe the algorithms the controller runs through to start and monitor test runs.
When the controller service starts it first opens the configuration file, QTController.exe.config. The settings in the configuration file determine what port the controller should listen on, the working directory for temporary files, whether to create a log file or not and a few other items described here:
If any of these settings are changed, the controller service needs to be stopped and restarted. Removing some settings will cause the controller to default to the value specified in the configuration file. Removing functional settings such as the ‘ControllerServicePort’ will keep the controller service from starting. Any event keeping the controller from starting will be logged in the event viewer. Some events may be difficult to diagnose the exact cause. For instance, if any invalid xml formatting is used an error will be logged, but the exact line or value is not always determinable. Removing the settings ‘ControllerUsersGroup’ and ‘ControllerAdminsGroup’ effectively turns off role validation. It is recommended that you use role validation as a security emasure.
Here are the default values set in the qtcontroller.exe.config file:
<add key="LogSizeLimitInMegs" value="20"/>
<add key="AgentSyncTimeoutInSeconds" value="120"/>
<add key="ControllerServicePort" value="6901"/>
<add key="ControllerUsersGroup" value="TeamTestControllerUsers"/>
<add key="ControllerAdminsGroup" value="TeamTestControllerAdmins"/>
<add key="CreateTraceListener" value="no"/>
Say you want to change the working directory to, “c:\ControllerWorkingDir”. First create the directory on the controller machine. And then add the ‘WorkingDirectory’ key to the ‘<appSettings>’ section in the config file:
<add key="WorkingDirectory" value="c:\ControllerWorkingDir"/>
After reading the settings above the controller reads the agents it will manage, and the assciated agent attributes (persisted in qtcontrollerconfig.xml). There is a user interface for modifying this file so you should not need to modify it by hand. Any agent not in this list is ignored if it tries to connect to the controller. It is denied the connection if the user the agent is running as is not in the ‘TeamTestAgentService’ group.
If files are deployed as part of the test run, the controller does not use a regular file copy buts copies the data over the connection from VS. The path of each file in the file list is relative to the client machine (the machine from which the test run was submitted). The controller calls back into the host application and asks it for a block of the file currently being copied. The files are copied into a directory under the WorkingDirectory. Using the example above:
Once all the blocks of a file has been copied, the same steps are taken for the next file. Once all the files are copied the test run is placed in a ‘ready to execute’ state. When the test run begins to execute, the same deployment process occurs between the controller and each of the agents the test is submitted - before actual test execution can begin.
When an agent completes the test run it may return files back to the controller, these are stored here:
Test Run Scheduling – Part 1
After completion of the deployment stage the test run will remain idle until it is popped off the queue for execution. Scheduling the test run is determined by a number of variables contained in the test run configuration and the list of test cases in the test run.
Initially the controller splits the test cases up into a set of tasks. This allows for the possibility of multiple test cases to execute simultaneously. Based on what was just stated, a test run can be thought of as a job, and the job manages a set of one or more tasks. Before you go further some terminology needs to be defined.
An agent can only execute one test at a time, and should a catastrophic error occur during execution the run may be aborted leaving many tests unexecuted. Bucketing helps in a couple of ways: a) it allows the test cases to execute simultaneously on different agent machines; b) if a catastrophic error occurs during execution, only the remaining tests in a bucket will go unexecuted.
One bucket at a time is sent to each agent that has been selected to execute the test run. If more than one agent is available to execute a test run, then more than one bucket can be executing at a time. As an agent completes the execution of a bucket, it is sent another if more exist.
If 1000 or more non-load tests exist in a submitted test run, buckets of size 200 will be created. You can change these values, in a test run configuration file, but note there is no user interface to make the changes. In the test run configuration file (*.testrunconfig), the following two tags exist:
You change at your own risk – do not save the file without making a back up. If the file is saved with invalid xml formatting the file may become unreadable and you will not be able to run tests with this test run configuration.
Test Run Scheduling - Part 2
When a job is scheduled to execute it first creates a set of tasks to execute the tests within a test run. Tasks are created using the following algorithm:
For example, say you have a test run with the following tests in it:
All the non-load tests total 1000 (UnitTest1-UnitTest999 + OrderedTest1). Five tasks would be created:
Task3: OrderesTest1 + 199 Unit tests
Task4: 200 unit tests
Task5-7: 200 unit tests each
If there are enough agent machines, it is possible Tasks3-Task7 will execute simultaneously. Load tests use all available agents so Task1 and Task2 are idle until all agents are available to execute.
Multiple test runs can execute at the same time. The initial test run popped from the queue will always have first rights to any agents meeting the test run configuration criteria. Subsequent test runs can use any agents still available that meet test run configuration criteria associated with the test run. Note that an agent must be marked online to be available for execution.
See section ‘Miscellaneous’, ‘Examples of Agent Selection and Scheduling’ for some example test run submissions.
Executing Test Types
Load tests require a synchronized start. As each agent selected to execute a load test completes any deployment it signals the controller it is ready to execute the test. Once all the agents have entered this state, the controller will send a start event to all agents. The agents will then begin execution. Load tests will use a thread-pool when executing Web Tests and multiple threads to execute non-Web tests. This allows multiple tests to execute simultaneously. This is all managed by the test adapter plug-in.
Non-load tests do not require a synchronized start. If more than one agent is used to execute non-load tests they will begin executing immediately after deployment, no matter the state of the other agents. These tests are executed synchronously one after another.
Monitoring Test Runs
The controller task is charged with monitoring the test list the job assigned it – making sure the tests/agents remain responsive. It does this by monitoring events returned by the agents. If an agent does not return a test result within the timeout period specified in the test run configuration, the task will attempt to abort the current test case and execute the next one on the agent. This applies to all test types. Note that load tests send aggregated result messages back typically every few seconds.
The controller job is charged with monitoring the overall execution time of the entire test run. If the test run does complete within the specified run timeout, the job will stop all the tasks and eventually the entire run.
Timeouts can be accessed from the Visual Studio ‘Test’ menu. Click ‘Test’, selected ‘Edit Run Configurations’ and select the run configuration you want to edit. A dialog will appear, in the left pane select ‘Test Timeouts’.
Firewall and some proxy devices can block the communication between the controller and agents, or the controller and Visual Studio. Be sure to open the listening ports used for the controller and agent to allow communication. See the InstallGuide.htm on the installation CD for more information on this. As noted above the listening port setting for the controller is in the qtcontroller.exe.config file. The setting for the agent is in it’s respective configuration file, qtagentservice.exe.config or qtagentserviceui.exe.config.
By default Visual Studio uses any available port to communicate with the controller. Since the port can change from run to run, you would need to open all ports on your firewall device for proper communication between the controller and Visual Studio. You can set a specific port or port range for Visual Studio to use when communicating with the controller by doing the following:
You can tell Visual Studio to use a specific port by adding this registry key:
After creating the key, add two new DWORD values:
To use a single port you can just specify a value in ‘PortRangeStart’. If you specify a value in ‘PortRangeStart’ and ‘PortRangeEnd’ Visual Studio will attempt to listen on the ports starting from ‘PortRangeStart’ to ‘PortRangeEnd’. Once it successfully opens a port it will stop looking for a port in the port range.
If there is a proxy or firewall device between Visual Studio and the controller machine you will need to open the port or port range you add.
You can verify Visual Studio is listening on the specified port or a port in the specified range by:
You should see one of the ports you specified in a listening state. For example:
TCP <machine>:6905 mymachine.microsoft.com:0 LISTENING
For more information of ‘Agent Constraints’ see Distributed Functional Testing with Visual Studio 2005 Team System
Here are some examples the controller scheduling and agent selection given various test run configurations:
Controller is managing 3 agents
bucketSize = 5
bucketThreshold = 6
Using a mix of 12 Unit and Web tests.
No agent constraints specified.
Submit the test run.
The test list is split into 3 buckets of sizes 5, 5, and 2 and the buckets are executed simultaneously.
The same name/value attributes are assigned to each agent. All 3 agents meet the selection criteria.
The test list is split into 3 buckets of sizes 4, 4, and 4 and the buckets are executed simultaneously.
Using a mix of 12 Unit and Web tests.
The same name/value attributes are assigned to two agents. Two agents meet the selection criteria.
Submit the test run.
Only the two agents are used for the test run.
Using a mix of 12 Unit and Web tests.
Submit the test run.
While the test is executing submit another test run with 10 more Unit and Web tests. This run will execute on the unused third agent.
Using a load test.
The load test is executed on the two agents meting the criteria.
The load test is executed on the two agents meting the criteria. While the test is executing submit another test run with 10 more Unit and Web tests. This run will execute on the unused third agent.
The load test is executed on the two agents meting the criteria. While the test is executing submit another load test with no agent constraints specified. This run will idle until all agents become available for execution.
Assign a different name/value attribute to the third agent.
The load test is executed on the two agents meting the criteria. While the test is executing submit another load test with agent constraints specified that match the third agent. This run will execute without waiting for the other two agents to become available.
The Microsoft Visual Studio Team Load Agent 2005 Controller offers a lot of flexibility for executing test runs. A combination of the test run configuration and the settings in the qtcontroller.exe.config file allow a number of flags to adjust. The controller will use the settings as hints for agent selection, the scheduling of test runs and the scheduling of test cases to utilize the agents efficiently. It will monitor the test runs and take appropriate action if a test or test run gets into a bad state.