Sean Lumley's Blog

Description of Tables and Columns in VS 2010 Load Test Database

2010-02-12T18:36:00Z

In this blog post I am going to update Bill Barnett's post which describes the tables and columns in the load test database. Here is Bill's original post which applies to VS 2005 and VS 2008: 2008 Database

First, in VS 2010 the default load test database name has been changed from LoadTest to LoadTest2010. Now let's review the tables.

*** Each new table or column will be prefixed with ***

Table: LoadTestRun: Contains one row for each load test that is executed.

Columns:

LoadTestRunId (int): A unique Id for this load test run in this load test results store. The value is 1 for the first load test and increments for each load test that is executed. All of the tables in the load test results store use the LoadTestRunId identify the load test run to which the data belongs.
LoadTestName (nvarchar(255)): The name of the load test
RunId (char (36)): A Guid that identifies the test run in which the load test was run.
Description (nvarchar (255)): A description of the load test from the Analysis dialog in the Load Test Results Viewer.
StartTime (datetime): The time at which the load test started.
EndTime (datetime): The time at which the load test completed or was stopped by the user. EndTime is NULL while the test is in progress.
RunDuration (int): The load test run duration (in seconds) that was specified in the active Run Settings for the load test.
WarmupTime (int): The warmup time (in seconds) that was specified in the active Run Settings for the load test.
RunSettingUsed (nvarchar (32)): The name of the active Run Settings at the time the load test was run.
IsLocalRun (bit): Indicates whether the load test was run locally or a rig (controller & agent(s)): 1 for local; 0 for rig.
ControllerName (nvarchar (255)): Name of the controller computer (or local computer name for a local run).
Outcome (nvarchar (32)): The outcome of the load test, which will be one of: "InProgress", "Completed", "Aborted", or "Error".
LoadTest (image): An encoded version of the LoadTest that was run.
Comment (ntext): A potentially long comment about the load test from the Analysis dialog in the Load Test Results Viewer.
***LoadTestSchemaRev(int): An integer indicating what version of product the run was created with. If you use VS2008 against this DB, this column will be null. If you use VS2010, it will be 3.
***CooldownTime(int): This is amount of time spent in cooldown phase. This time is in seconds.

Table: LoadTestCase: Stores data on the inner tests included in the load test (one row per inner test)

Columns:

LoadTestRunId (int): The unique Id of the load test run
ScenarioId (int): The unique Id of the Scenario containing the test (from the LoadTestScenario table)
TestCaseId (int): A unique Id for the test within this load test run
TestCaseName (nvarchar (64)): The name of the test
***TestElement (image): Identifier as to which test type the test case is(Web Test, Unit Test, etc...)
***TestType (nvarchar(64)): Readable version of what test type the test case is

Table: LoadTestMessage: Stores data for each error that occurs during the load test

Columns:

LoadTestRunId (int): The unique Id of the load test run.
AgentId (int): The unique Id of the load test agent on which the error occurred (-1 for errors not specific to an agent)
MessageId (int): A unique Id for the message on the agent during this load test run
MessageType (tinyint): An integer that encodes the type of error; the values used and their meanings are: 0: TestError, 1: Exception, 2; HttpError, 3: ValidationRuleError, 4: ExtractionRuleError, 5: Timeout
MessageText nvarchar (2048): The error message text
SubType (nvarchar (64)): The sub type of the error. The values for this vary depending on the MessageType. For example, if MessageType is Exception, SubType is the name of the Exception class.
StackTrace (ntext): A stack trace for the code location where the error occurred (used when MessageType is Exception or TestError)
MessageTimeStamp (datetime): The time at which the error occurred
TestCaseId (int): The unique Id of the inner test (from the LoadTestCase table) in which the error occurred, or NULL for errors not associated with a specific inner test such as performance counter collection errors.
RequestId (int): The unique Id of the web test request (from the WebLoadTestRequestMap table) on which the error occurred, or NULL for errors not associated with web test request.
***TestLogId(int): The unique Id of the test log (from the LoadTestTestLog table). Test log contains the entire testcase and not just the request that failed.

Table: ***LoadTestMessageType: Stores a row for each type of message that occurs. Whereas LoadTestMessage stores data for each occurence of a message.

Columns:

LoadTestRunId (int): The unique Id of the load test run.
MessageTypeId (int): A unique Id for the message
MessageType (tinyint): An integer that encodes the type of error; the values used and their meanings are: 0: TestError, 1: Exception, 2; HttpError, 3: ValidationRuleError, 4: ExtractionRuleError, 5: Timeout
SubType (nvarchar (64)): The sub type of the error. The values for this vary depending on the MessageType. For example, if MessageType is Exception, SubType is the name of the Exception class.

Table: LoadTestThresholdMessage: Stores one row of data for each threshold rule violation

Columns:

LoadTestRunId (int): The unique Id of the load test run
TestRunIntervalId (int): Id of the collection interval during which the threshold rule violation occurred
CounterInstanceId (int): Id of the performance counter instance that had the threshold rule violation
MessageId (int): Unique Id of the threshold rule violation within this load test run
MessageText (nvarchar (2048)): Text describing the threshold rule violation

Table: ***LoadTestFileAttachment: Stores one row of data for each file that is beign saved with a run. Typically these will be files that are returned from datacollectors.

Columns:

LoadTestRunId (int): The unique Id of the load test run
FileAttachmentId(int): Unique id for the file
MachineName (nvarchar (255)): Machine where file came from.
FileName (nvarchar (260)): Name of file being stored
FileSize(bigint): Size of file being stored

Table: ***LoadTestFileAttachmentChunk: Stores the file content. File is broken into chunks and stored in multiple rows..

Columns:

LoadTestRunId (int): The unique Id of the load test run
FileAttachmentId(int): Unique id for the file. Corresponds to row in LoadTestFileAttachment
StartOffset(bigint): Location in file that where this chunk starts
EndOffset(bigint): Location in file that where this chunk ends
ChunkLength(bigint): Length of bytes being stored
ChunkBytes(image): Actual bytes being stored

Table: ***LoadTestDataCollectorLog: Contains one row for each data collector that is running during a loadtest

Columns:

LoadTestRunId (int): The unique Id of the load test run
DataCollectorId(int): Unique id for the data collector.
DataCollectorDisplayName(nvarchar(255)): Name of data collector
MachineName(nvarchar(255)): Name of machine the data collector is running on
TimestampColumnName(nvarchar(128)): Name of column in data collector table which corresponds to time stamp data
DurationColumnName(nvarchar(128)): Name of column in data collector table which corresponds to duration data
CreateTableFormatString(nvarchar(max)): SQL statement used to create table to store data for this data collector

Overview of performance counter tables

The metadata for all of the performance counters collected during a load test is stored in three related tables: LoadTestPerformanceCounterCategory, LoadTestPerformanceCounter, and LoadTestPerformanceCounter.

This metadata includes all four components of performance counter identifier: computer name, category name, counter name, and instance name.

Note that these three tables only store metadata about the performance counters; the sampled performance counter values are stored in LoadTestPerformanceCounterSample.

Table: LoadTestPerformanceCounter: Stores one row for each performance counter that is collected during the load test

Columns:

LoadTestRunId (int): The unique Id of the load test run
CounterCategoryId (int): The CounterCategoryId from the LoadTestPerformanceCounterCategory row for the catgory containing this counter name
CounterId (int): The unique Id of this performance counter for this load test run
CounterName (nvarchar (255)): The name of the performance counter (for example "Available MBytes")
***HigherIsBetter (bit): A value indicating wheter an increase in this performance counter is a good thing. For example an increase in Available MBytes is typically a good thing.

Table: LoadTestPerformanceCounterCategory: Stores one row for each performance counter category and computer on which the category is collected

Columns:

LoadTestRunId (int): The unique Id of the load test run
CounterCategoryId (int): The unique Id of this performance counter category for this load test run
CategoryName (nvarchar (255)): The name of the performance counter category (for example "Memory")
MachineName (nvarchar (255)): The name of the computer on which the category was collected
StartTimeStamp100nSec (bigint): A timestamp indicating when the category was first collected during the load test

Table: LoadTestPerformanceCounterInstance: Stores one for each performce counter instance that is collected during the load test

Columns:

LoadTestRunId (int): The unique Id of the load test run
CounterId (int): The CounterId from the LoadTestPerformanceCounter table for the perf counter of which this is an instance
InstanceId (int): Unique Id of this perf counter instance in this load test run
LoadTestItemId (int): For performance counters in the "LoadTest:*" categories, this uniquely identifies the item in the load test with which this performance counter is associated. For example, for performance counters in the "LoadTest:Request" category, the LoadTestItemId is the value of the RequestId in the WebLoadTestRequestMap table.
InstanceName (nvarchar (255)): The name of the performance counter instance (for example "_Total")
CumulativeValue (real): The final calculated value for this performance counter instance over the entire load test run
OverallThresholdRuleResult (tinyint): Indicates the most severe threshold rule result for this performance counter instance during the load test; 0 for Ok, 1 for Warning, 2 for Critical

Table: LoadTestPerformanceCounterSample: Stores the performance counter samples. There is one for each performance counter instance collected for each sampling interval during the loadtest. Many of the columns in this table are fields of the .Net class System.Diagnostics.CounterSample which is used internally by the load test implementation, and their values don't make much sense outside that context.

Columns:

LoadTestRunId (int): The unique Id of the load test run
TestRunIntervalId (int): The sequence number of the collection interval in which this sample was collected
InstanceId (int): The InstanceId from the LoadTestPerformanceCounterInstance table row containing for the performance counter instance of which this is a sample
ComputedValue (real): A value computed for the perf counter based on this sample and the previous sample (or NULL if there was no sample in the previous interval)
RawValue (bigint): The RawValue field of the System.Diagnostics.CounterSample value
BaseValue (bigint): The BaseValue field of the System.Diagnostics.CounterSample value
CounterFrequency (bigint): The CounterFrequency field of the System.Diagnostics.CounterSample value
SystemFrequency (bigint): The SystemFrequency field of the System.Diagnostics.CounterSample value
SampleTimeStamp (bigint): The SampleTimeStamp field of the System.Diagnostics.CounterSample value
SampleTimeStamp100nSec (bigint): The SampleTimeStamp100nSec field of the System.Diagnostics.CounterSample value
CounterType (int): The PerformanceCounterType field of the System.Diagnostics.CounterSample value
ThresholdRuleResult (tinyint): The result of any threshold rule applied to this sample: 0 for Ok, 1 for Warning, 2 for Critical
ThresholdRuleMessageId (int): The MessageId from the LoadTestThresholdMessage row with more details on any threshold violation thrown for this sample, or NULL if there was no threshold violation

Table: LoadTestRunAgent: Stores the names of the agent machines used for a load test run

Columns:

LoadTestRunId (int): The unique Id of the load test run
AgentId (int): A unique Id of this agent during this load test run
AgentName (nvarchar (255)): The computer name of the agent

Table: LoadTestRunInterval: Stores one row for each performance counter collection interval during the load test run

Columns:

LoadTestRunId (int): The unique Id of the load test run
TestRunIntervalId (int): The unique Id of this interval in this load test run
IntervalStartTime (datetime): The start time of the collection interval
IntervalEndTime (datetime): The end time of the collection interval

Table: LoadTestScenario: Stores the names of the Scenarios defined for a load test

Columns:

LoadTestRunId (int): The unique Id of the load test run
ScenarioId (int): The unique Id of the Scenario within this load test
ScenarioName (nvarchar (64)): The name of the Scenario

Table: WebLoadTestErrorDetail: Stores detailed data about a web test request when there is an error associated with a web test request

Columns:

LoadTestRunId (int): The unique Id of the load test run
AgentId (int): The Id of the agent on which the error occurred
MessageId (int): The MessageId from the row in the LoadTestMessage table for the error with which this error detil is associated
WebTestRequestResult (image): A binary encoded version of the object that stores all of the data captured regarding the result of a web test request

Table: ***LoadTestTestLog: Stores the entire test case result assoicated with an execution of a testcase. By default just failed tests will be logged, but you can also
configure successful tests to be logged as well.

Columns:

LoadTestRunId (int): The unique Id of the load test run
AgentId (int): The Id of the agent on which testcase was executed
TestCaseId (int): The id of the test case for which log is being stored
TestLogId(int): Unique id for the test log being saved
TestLog(image): A binary encoded version of the object that stores all of the data captured regarding the result of a web test

Table: WebLoadTestRequestMap: Stores one row for each unique Uri to which a web test request is sent during the load test

Columns:

LoadTestRunId (int): The unique Id of the load test run
RequestId (int): The unique Id of this request in this load test run
TestCaseId (int): The Id of the web test that generated this request
RequestUri (nvarchar (2048)): The Uri for the request. This excludes the query string parameters except for those query string paramters with the property "
***ResponseTimeGoal(float): If you set a goal for a web test request, that value will be storded in this column

Table: WebLoadTestTransaction: Store the names of all transactions defined in the load test (this includes transactions defined in web tests, and transaction defined in unit tests using the BeginTimer()/EndTimer() methods.

Columns:

LoadTestRunId (int): The unique Id of the load test run
TransactionId (int): The unique Id of the transaction in this load test run
TestCaseId (int): The TestCaseid from the LoadTestCase table for the test that defined the transaction
TransactionName (nvarchar (64)): The name of the transaction

Table: LoadTestSqlTrace: Stores collected when performing SQL tracing during a load test; the field correspond to SQL trace fields

Columns:

LoadTestRunId (int): The unique Id of the load test run
TextData (ntext): The text of the SQL operation
Duration (bigint): The duration of the SQL operation
StartTime (datetime): The start time of the SQL operation
EndTime (datetime): The end time of the SQL operation
Reads (bigint): The number of reads that occurred while performing the SQL operation
Writes (bigint): The number of writes that occurred while performing the SQL operation
CPU (int): The CPU time required to perform the SQL operation
EventClass (int): An integer identifying the type of SQL operation traced

Table: LoadTestBrowsers: Stores the names of all browser types used during the load test

Columns:

LoadTestRunId (int): The unique Id of the load test run
BrowserId (int): The unique Id of this browser
BrowserName (nvarchar (255)): The name of the browser

Table: LoadTestNetworks: Stores the names of all network types used during the load test

Columns:

LoadTestRunId (int): The unique Id of the load test run
NetworkId (int): The unique Id of the network type
NetworkName (nvarchar (255)): The name of the network type

Tables used to store Timing Details

All of the following tables only have data stored in them when the "Timing Details Storage" option on the Load Test Run Settings is set to a value other than "None".

Table: LoadTestTestDetail: Stores the results of individual inner tests; there is one row for each test completed during the load test

Columns:

LoadTestRunId (int):The unique Id of the load test run
TestDetailId (int): A unique Id for this row in the load test run
TimeStamp (datetime): The time that the test completed
TestCaseId (int): The TestCaseId from the LoadTestCase table for the row that identifies the test
ElapsedTime (float): The elapsed time to run the test in seconds
AgentId (int): The AgentId from the LoadTestAgent table row that identifies the agent on which the test was run
BrowserId (int): The BrowserId from the LoadTestBrowser table that identifies the browser type used for a web test
NetworkId (int): The NetworkId from the LoadTestNetwork table that identifies the network type used for a web test
***Outcome (tinyint): A value indicating the outcome of the test case detail
***TestLogId (int): A unique id indicating associated test log from the LoadTestTestLog table
***UserId (int): An int indicating which virtual user executed the test
***EndTime (datetime): The time the test completed
***InMeasurementInterval(bit): 0 if test completed during warmup or cooldown. 1 if test completed during actual duration of load test. Only tests with a value of 1 will be included when computing statistics.

Table: LoadTestPageDetail: Stores timing results of individual web test pages; there is one row for each web test page completed during the load test

Columns:

LoadTestRunId (int): The unique Id of the load test run
PageDetailId (int): A unique Id for this row in the load test run
TestDetailId (int): The TestDetailId from LoadTestTestDetail row with the result for the test containing this page
TimeStamp (datetime): The time at which the page completed
PageId (int): The RequestId from the WebLoadRequestMap table (join on this to identify the Uri for the page)
ResponseTime (float): The response time for the page in seconds
ResponseTimeGoal (float): The response time goal for the page in seconds
GoalExceeded (bit): Bool indicating whether or not the response time goal was exceeded by the response time
***EndTime (datetime): The time the page completed
***Outcome (tinyint): A value indicating the outcome of the page. 0 for pass, 1 for fail
***InMeasurementInterval(bit): 0 if page completed during warmup or cooldown. 1 if page completed during actual duration of load test. Only pages with a value of 1 will be included when computing statistics.

Table: LoadTestTransactionDetail

Columns:

LoadTestRunId (int): The unique Id of the load test run
TransactionDetailId (int): A unique Id for this row in the load test run
TestDetailId (int): The TestDetailId from LoadTestTestDetail row with the result for the test containing this transaction
TimeStamp (datetime): The time at which the transaction completed
TransactionId (int): The TransactionId from the WebLoadTestTransaction table (join on this to identify the transaction name)
ElapsedTime (float): The elapsed time to complete the transaction in seconds
***EndTime (datetime): The time the transaction completed
***InMeasurementInterval(bit): 0 if transaction completed during warmup or cooldown. 1 if transaction completed during actual duration of load test. Only transactions with a value of 1 will be included when computing statistics.
***ResponseTime(float): The difference between this column and the ElapseTime column is that this value does not include thinktime assoicated with a request

Table: LoadTestTestSummaryData: Summarizes the detailed Test result data that is stored in LoadTestTestDetail; there is one row for each test in the load test run which rolls up the results for that test

Columns:

LoadTestRunId (int): The unique Id of the load test run
TestCaseId (int): The TestCaseId from the LoadTestCase table for the row that identifies the test
TestsRun (int): The number of tests completed
Average (float): The average time to run the test in seconds
Minimum (float): The minimum time to run the test in seconds
Maximum (float): The maximim time to run the test in seconds
Percentile90 (float): The 90% percentile test time (90% of the tests completed in less than this time)
Percentile95 (float): The 95% percentile test time (95% of the tests completed in less than this time)
***Percentile99 (float): The 99% percentile test time (99% of the tests completed in less than this time)
***Median(float): The median time to run the test in seconds
***StandardDeviation(float): The standard deviation of the time to run the test

Table: LoadTestTransactionSummaryData: Summarizes the detailed transaction data that is stored in LoadTestTransactionDetail; there is one row for each transaction name used during the load test run which rolls up the results for that transaction. The average column is calculated based on the elapsed time. All other stats are calculated base on the response time of transaction. The response time does not include thinktime.

Columns:

LoadTestRunId (int): The unique Id of the load test run
TransactionId (int): The TransactionId from the WebLoadTestTransaction table (join on this to identify the transaction name)
TransactionCount (int): The number of tests completed
Average (float): The average time to run the transaction in seconds
Minimum (float): The minimum time to run the transaction in seconds
Maximum (float): The maximim time to run the transaction in seconds
Percentile90 (float): The 90% percentile transaction time (90% of the transactions completed in less than this time)
Percentile95 (float): The 95% percentile transaction time (95% of the transactions completed in less than this time)
***Percentile99 (float): The 99% percentile transaction time (99% of the transaction completed in less than this time)
***Median(float): The median time to run the transaction in seconds
***StandardDeviation(float): The standard deviation of the time to run the transaction
***AverageTransactionTime (float): The average time to run the transaction in seconds. This value does not include thinktime.

Table: LoadTestPageSummaryData: Summarizes the detailed page data that is stored in LoadTestPageDetail; there is one row for each page Uri invoked during the load test run which rolls up the results for that page.

Columns:

LoadTestRunId (int): The unique Id of the load test run
PageId (int): The RequestId from the WebLoadRequestMap table (join on this to identify the Uri for the page)
PageCount (int): The number of times this page was completed
Average (float): The average time to run the page in seconds
Minimum (float): The minimum time to run the page in seconds
Maximum (float): The maximim time to run the page in seconds
Percentile90 (float): The 90% percentile page time (90% of the pages completed in less than this time)
Percentile95 (float): The 95% percentile page time (95% of the pages completed in less than this time)
***Percentile99 (float): The 99% percentile page time (99% of the pages completed in less than this time)
***Median(float): The median time to run the page in seconds
***StandardDeviation(float): The standard deviation of the time to run the page

Table: LoadTestPageSummaryByNetwork: Summarizes the detailed page data that is stored in LoadTestPageDetail by network type; there is one row for each page Uri for each network type invoked during the load test run which rolls up the results for that page on each network type.

Columns:

LoadTestRunId (int): The unique Id of the load test run
PageId (int): The RequestId from the WebLoadRequestMap table (join on this to identify the Uri for the page)
NetworkId (int): The NetworkId from the LoadTestNetwork table that identifies the network type used for a web test
PageCount (int): The number of times this page was completed (on the the network type identified by NetworkId)
Average (float): The average time to run the page in seconds (on the the network type identified by NetworkId)
Minimum (float): The minimum time to run the page in seconds (on the the network type identified by NetworkId)
Maximum (float): The maximim time to run the page in seconds (on the the network type identified by NetworkId)
Percentile90 (float): The 90% percentile page time (on the the network type identified by NetworkId)
Percentile95 (float): The 95% percentile page time (on the the network type identified by NetworkId)
Goal (float): The response time goal for the page
PagesMeetingGoal (int): The number of pages run (on the the network type identified by NetworkId) with a response time less than the goal.
***Median(float): The median time to run the page in seconds (on the the network type identified by NetworkId)
***StandardDeviation(float): The standard deviation of the time to run the page (on the the network type identified by NetworkId)
***Percentile99 (float): The 99% percentile page time (on the the network type identified by NetworkId)

Table: ***LoadTestDetailMessge: This table will show the links between a test or page detail and an error message. For example, you can use this to figure out what page details failed with a HTTP Status Code of 404.

Columns:

LoadTestRunId (int): The unique Id of the load test run
LoadTestDetailMessageId (int): The unique id for this message/test/page combination.
TestDetailId(int): The TestDetailId from LoadTestTestDetail table which corresponds to this test
PageDetailId(int): The PageDetailId from LoadTestPageDetail table which corresponds to this page
MessageTypeId (int): The unique id for the message assoicated with this test/page detail. This corresponds to a row in LoadTestMessageType.

Convert Comments entered in Web Test Recorder into Transactions

2010-01-26T20:22:00Z

This post is going to be one more sample of a Web Test Recorder Plug-in. The recorder plug-ins are a new feature in VS 2010. In this sample, we will convert comments that are entered in the recorder into transactions. I have heard this request from a number of people and thought it would be a useful plug-in. Here are 2 previous recorder plug-in posts: Correlation plugin and Plug-in which adds Validation rule to test

Follow these steps to create plug-in which converts comments to transactions.

Create a new class library project
Right Click references node and select “Add reference…”
Click the .NET tab and add the Microsoft.Visual Studio.QualityTools.WebTestFramework assembly
Copy the code below and add it to your class file

using System.ComponentModel;
using Microsoft.VisualStudio.TestTools.WebTesting;
using Microsoft.VisualStudio.TestTools.WebTesting.Rules;

namespace SampleRecorderPlugin
{
    
    public class CovertRecordedCommentsToTransactions : WebTestRecorderPlugin
    {
        public override void PostWebTestRecording(object sender, PostWebTestRecordingEventArgs e)
        {
            for (int i = 0; i < e.RecordedWebTest.Items.Count; i++)
            {
                //get the item and see if it is a comment
                WebTestItem item = e.RecordedWebTest.Items[i];
                Comment c = item as Comment;
                if (c!=null)
                {
                    //if it is a comment create a transaction
                    TransactionTimer tt = new TransactionTimer();

                    //set name of transaction to the comment text
                    tt.Name = c.CommentText;
                    
                    //now add all items up until next comment
                    int j = i+1;
                    while (j < e.RecordedWebTest.Items.Count)
                    {
                        WebTestItem nextItem = e.RecordedWebTest.Items[j];                        

                        //if the item is not a comment move it into the transaction
                        if (!(nextItem is Comment))
                        {
                            tt.Items.Add(nextItem);
                            e.RecordedWebTest.Items.RemoveAt(j);
                        }
                        else
                        {
                            break;
                        }
                    }

                    //add the transaction to the web test
                    e.RecordedWebTest.Items.Insert(j, tt);

                    //remove the comment from the web test
                    e.RecordedWebTest.Items.RemoveAt(i);

                    //indicate that the web test has been modified
                    e.RecordedWebTestModified = true;
                    
                }
            }
            
        }
    }
}

Compile the project
Copy the dll to \Program Files\Microsoft Visual Studio 10.0\Common \IDE\PrivateAssemblies\WebTestPlugins
Restart VS and go to your test project.
Add a new Web Performance Test. You will see the following dialog:
Select the CovertRecordedCommentsToTransactions plug-in.
In your web test add some comments while you are recording. Here is what test looks like in the recorder. You can see the Browse, Add To Cart, and Checkout comments
Then hit stop on the recorder
You can see that back in the Web Test Editor these comments have been converted to transactions and the requests have been moved under the transactions. I have the plug-in removing the comment once it creates the transaction but you can easily change that.

Let walk through the code. First I am looping through the web test looking for comments:

            for (int i = 0; i < e.RecordedWebTest.Items.Count; i++)
            {
                //get the item and see if it is a comment
                WebTestItem item = e.RecordedWebTest.Items[i];
                Comment c = item as Comment;
                if (c!=null)
                {

Once I find a comment I am creating a transaction.

                    //if it is a comment create a transaction
                    TransactionTimer tt = new TransactionTimer();

                    //set name of transaction to the comment text
                    tt.Name = c.CommentText;

Next it will start adding requests into the transaction. It will add all requests up until the next comment or the end of the test. Once it adds it to the transaction it removes it from the outer test.

                    //now add all items up until next comment
                    int j = i+1;
                    while (j < e.RecordedWebTest.Items.Count)
                    {
                        WebTestItem nextItem = e.RecordedWebTest.Items[j];                        

                        //if the item is not a comment move it into the transaction
                        if (!(nextItem is Comment))
                        {
                            tt.Items.Add(nextItem);
                            e.RecordedWebTest.Items.RemoveAt(j);
                        }
                        else
                        {
                            break;
                        }
                    }

Finally add the transaction to the web test and remove the comment. Then set the event args RecordedWebTestModified to true.

                    //add the transaction to the web test
                    e.RecordedWebTest.Items.Insert(j, tt);

                    //remove the comment from the web test
                    e.RecordedWebTest.Items.RemoveAt(i);

                    //indicate that the web test has been modified
                    e.RecordedWebTestModified = true;

I hope this shows another use of the recorder plug-ins.  Let me know if you have any questions on this feature.

VSTS 2010 Feature: Another Sample Web Test Recorder Plug-in

2010-01-26T18:43:00Z

In a previous post I wrote about a new feature that was added for VS 2010 called extensible recorder plug-ins. Here is that post: Recorder Plug-ins Basically recorder plug-ins give you the opportunity to modify a recorded web test prior to the test being saved back to the web test editor. So if you always have to manually do a set of steps prior to being able to run a web test, you could create a recorder plug-in which will do those steps for you. In my previous post I showed you how to create a plug-in which will correlate dynamic parameters. I added a extraction rule to one of the requests in the recorded test and then bound the extracted value to a query string parameter further down the web test.

In this post I am going to give you another example of what you can do with recorder plug-ins. In this post we are going to add validation rules to the web test. We will add a validation rule that will validate the value of the Title tag matches the value at record time. Follow these steps.

Create a new class library project
Right Click references node and select “Add reference…”
Click the .NET tab and add the Microsoft.Visual Studio.QualityTools.WebTestFramework assembly

Copy the code below and add it to your class file

using System.ComponentModel;
using Microsoft.VisualStudio.TestTools.WebTesting;
using Microsoft.VisualStudio.TestTools.WebTesting.Rules;

namespace SampleRecorderPlugin
{
    [DisplayName("WebTestRecorderPlugin that adds title validation rule")]
    [Description("Adds a validation rule that verifies the recorded title tag matches playback title tag value")]
    public class RecorderPluginThatAddsTitleValidation : WebTestRecorderPlugin
    {
        public override void PostWebTestRecording(object sender, PostWebTestRecordingEventArgs e)
        {
            WebTestResultIteration recordedIteration = e.RecordedWebTestResult;
            foreach (WebTestResultUnit resultUnit in recordedIteration.Children)
            {
                WebTestResultPage pageResult = resultUnit as WebTestResultPage;
                if (pageResult != null)
                {
                    WebTestRequestResult finalRequestResult = pageResult.RequestResult;

                    //if there were redirects then need to get the last redirected requests
                    if (pageResult.RedirectedPages.Count > 0)
                    {
                        finalRequestResult = 
                          pageResult.RedirectedPages[pageResult.RedirectedPages.Count - 1].RequestResult;
                    }

                    //now get the response for the final request
                    WebTestResponse response = finalRequestResult.Response;
                    if (response.IsHtml)
                    {
                        HtmlTagInnerTextParser innerTextParser = new HtmlTagInnerTextParser(response.BodyString);
                        HtmlTagInnerText titleInnerTextTag = null;

                        //search for the <Title> tag
                        foreach (HtmlTagInnerText innerTextTag in 
                                 innerTextParser.GetInnerTextForHtmlTags("title", true, false, false))
                        {
                            if (!string.IsNullOrEmpty(innerTextTag.Value))
                            {
                                titleInnerTextTag = innerTextTag;
                                break;
                            }
                        }

                        if (titleInnerTextTag == null)
                        {
                            continue;
                        }
                        
                        //if we found a title tag then add a validation rule
                        ValidationRuleReference validateTitleRuleReference = new ValidationRuleReference();
                        validateTitleRuleReference.Type = typeof(ValidateHtmlTagInnerText);
                        validateTitleRuleReference.ValidationLevel = ValidationLevel.Medium;
                        validateTitleRuleReference.Properties.Add(new PluginOrRuleProperty("TagName", "title"));
                        validateTitleRuleReference.Properties.Add(new PluginOrRuleProperty("ExpectedInnerText", 
                                                                                           titleInnerTextTag.Value));
                        validateTitleRuleReference.Properties.Add(new PluginOrRuleProperty("Index", "0"));
                        validateTitleRuleReference.Properties.Add(new PluginOrRuleProperty("RemoveInnerTags", "False"));
                        validateTitleRuleReference.Properties.Add(new PluginOrRuleProperty("HasClosingTags", "True"));
                        validateTitleRuleReference.Properties.Add(new PluginOrRuleProperty("IgnoreCase", "True"));
                        validateTitleRuleReference.Properties.Add(new PluginOrRuleProperty("CollapseWhiteSpace", 
                                                                                           "False"));

                        // Get the corresponding request in the Declarative Web test
                        WebTestRequest requestInWebTest = 
                                e.RecordedWebTest.GetItem(pageResult.DeclarativeWebTestItemId) as WebTestRequest;
                        if (requestInWebTest != null)
                        {
                            requestInWebTest.ValidationRuleReferences.Add(validateTitleRuleReference);
                            e.RecordedWebTestModified = true;
                        }
                    }
                }
            }
        }
    }
}

Copy the dll to \Program Files\Microsoft Visual Studio 10.0\Common7\IDE\PrivateAssemblies\WebTestPlugins
Restart VS and go to your test project.
Add a new Web Performance Test. You will see the following dialog:
Select the plug-in we just created and record your web test
When test completes, expand the requests nodes and look for the validation rules. You should see the following:
I had a validation rule added for each of my requests.

Let’s walk through parts of the code. Here is the first part of the method. In this code fragment we are looping through all the recorded items and looking for pages. You can also add comments during playback which is why the code is checking for page results.

            WebTestResultIteration recordedIteration = e.RecordedWebTestResult;
            foreach (WebTestResultUnit resultUnit in recordedIteration.Children)
            {
                WebTestResultPage pageResult = resultUnit as WebTestResultPage;
                if (pageResult != null)
                {

Once we have the page result, we are checking to see if the page was part of a redirect. If so, we are grabbing the last request in the redirect chain:

                    WebTestRequestResult finalRequestResult = pageResult.RequestResult;

                    //if there were redirects then need to get the last redirected requests
                    if (pageResult.RedirectedPages.Count > 0)
                    {
                        finalRequestResult = pageResult.RedirectedPages[pageResult.RedirectedPages.Count - 1].RequestResult;
                    }

Next we are going to grab the response and run it through the tag parser. Once we have the tag collection it is going to iterate over that collection looking for the Title tag:

                   //now get the response for the final request
                    WebTestResponse response = finalRequestResult.Response;
                    if (response.IsHtml)
                    {
                        HtmlTagInnerTextParser innerTextParser = new HtmlTagInnerTextParser(response.BodyString);
                        HtmlTagInnerText titleInnerTextTag = null;

                        //search for the <Title> tag
                        foreach (HtmlTagInnerText innerTextTag in 
                                 innerTextParser.GetInnerTextForHtmlTags("title", true, false, false))
                        {
                            if (!string.IsNullOrEmpty(innerTextTag.Value))
                            {
                                titleInnerTextTag = innerTextTag;
                                break;
                            }
                        }

If we find a title tag it will break out of the loop and add a validation rule. First we create the reference:

                        //if we found a title tag then add a validation rule
                        ValidationRuleReference validateTitleRuleReference = new ValidationRuleReference();
                        validateTitleRuleReference.Type = typeof(ValidateHtmlTagInnerText);
                        validateTitleRuleReference.ValidationLevel = ValidationLevel.Medium;
                        validateTitleRuleReference.Properties.Add(new PluginOrRuleProperty("TagName", "title"));
                        validateTitleRuleReference.Properties.Add(new PluginOrRuleProperty("ExpectedInnerText", 
                                                                                           titleInnerTextTag.Value));
                        validateTitleRuleReference.Properties.Add(new PluginOrRuleProperty("Index", "0"));
                        validateTitleRuleReference.Properties.Add(new PluginOrRuleProperty("RemoveInnerTags", "False"));
                        validateTitleRuleReference.Properties.Add(new PluginOrRuleProperty("HasClosingTags", "True"));
                        validateTitleRuleReference.Properties.Add(new PluginOrRuleProperty("IgnoreCase", "True"));
                        validateTitleRuleReference.Properties.Add(new PluginOrRuleProperty("CollapseWhiteSpace", "False"));

Then we find the request in the web test that corresponds to this result page and add the reference to the validation rule collection:

                       // Get the corresponding request in the Declarative Web test
                        WebTestRequest requestInWebTest = 
                                e.RecordedWebTest.GetItem(pageResult.DeclarativeWebTestItemId) as WebTestRequest;
                        if (requestInWebTest != null)
                        {
                            requestInWebTest.ValidationRuleReferences.Add(validateTitleRuleReference);
                            e.RecordedWebTestModified = true;
                        }

Hopefully this shows you another use for recorder plug-ins. Try it out and let us know what you think.

VSTS 2010 Feature: ASP.NET profiler integration with Load Tests

2010-01-12T18:55:00Z

One of the new features in VSTS 2010 is Data Collectors. Data collectors can run as part of your test run and then associate the data collected with the test. This is also an extensibility point. You can create your own data collectors and store any kind of data with your test runs. The following are available out of the box:

ASP.NET Client Proxy for IntelliTrace and Test Impact
ASP.NET Profiler
Code Coverage
Event Log
IntelliTrace
Network Emulation
System Information
Test Impact
Video Recorder

This link can give you more high level information about data collectors: Data Collectors

I am going to focus on the ASP.NET Profiler collector and show you how this can integrate with load tests. First off this collector profiles the IIS process, so it will not work against the development web server. For this demo I setup the IbuySpy website. To profile the web site, you need to install an agent on the machine that the IIS is running on. This agent will not be generating load, but will be a collection only agent. The link above goes into detail about how to create and configure these types of agents. So let’s walk through process of running a load test with the ASP.NET profiler collector running.

Create a new test project
Create a Web Performance Test that will test the web site you want to profile. I recorded a test that adds an item to the shopping cart, logs into the site and then places the order.
Now Create a Load Test and add the web test.
Before we run the test, we need to modify the run settings and tell it to run the ASP.NET data collector. So go to the top level test menu, select Edit Test Settings and then select the active test setting.
I am running my test locally and have IIS running locally, so I do not need to configure roles. But if you are running IIS on a different machine, follow the link above to create appropriate role. Then Click on the “Data and Diagnostics” tab.
Choose the correct role for your IIS machine and then click the enabled check box for the ASP.NET profiler.
Now the test is ready to run. Just start your load test like you normally do. Click the play button on the Load Test Editor toolbar. while the test is running, you will not see any differences. When the test completes, go to the post run view.
In the post run view, you will see a new button on the load test execution UI toolbar.
Click the Profiler Report button which is the second from the right. This will generate a profiler report. Here is the initial page:
Check out this help link to get more information on all the data that is available in the profiler report: Profiler Report Information
I do want to point out one new feature of the profiler that I particularly like and that is the database interaction page. Change the current view from Summary to “Tier Interactions”. Then expand the node for one of your pages. I am expanding the login page. When I expand it I see the following:
This is showing me my database connections and when I click on a connection, it is showing me the calls that were made. In the command text column, you see the stored procedures that were called on the login page. Then you see # of times each was called and elapsed time. This is an awesome view into the interactions between your application and the sql calls being made.

In future posts I will show you how to create your own data collectors. For now please try out this data collector and others and let us know what you think.

VSTS 2010 Feature: Branching and Looping in Declarative Web Tests

2010-01-12T13:58:00Z

We continue to add features to the declarative web test model that will try to allow you to stay in the web test editor and only go to code for custom rules, plugins, etc. Prior to VSTS 2010, if you wanted to do any real branching or looping in web tests, you would have to go to code. In VSTS 2010 we have added branching and looping into the declarative web test model. Check out this great blog post by Chris Patterson for a complete description of the feature: http://blogs.msdn.com/chrispat/archive/2010/01/10/branching-and-looping-in-a-declarative-web-test.aspx

VSTS 2010 Feature: Find/Replace in Web Test Editor

2010-01-11T13:57:00Z

This blog post will describe the find/replace functionality added to the web test editor. This functionality was added in the Beta 2 release. A piece of feedback that we have heard a number of times is “It would be nice if I could create a context parameter and then easily replace all instances of a string in the web test with the parameter”. For example, suppose you want to parameterize part of the URL’s path. Each URL in your web test looks like http://server1/site1/page123.aspx. Now you want to parameterize this so that the server and path are broken out. You can use the Parameterize server feature to handle server1. But to parameterize site1, you would have to do the following

1) Create a context parameter called Path with value site2.

2) Then modify each request in the web test and replace each instance. So if you have 10 requests, you have to repeat this 10 times.

3) Also you may need to modify parts of query string parameters that have site1 in them.

This can be a time consuming and error prone process. Now in VS 2010, we have added typical find/replace behavior into the web test editor. In VS2010, you would do the following:

1) Create a context parameter called Path with value site2

2) Right click on root node in the web test and select “Find and Replace in Request…”. This is the only way to launch the find/replace that will work with the web test editor. If you hit ctrl-f, you will see the normal find dialog in VS, but this one will NOT work with web test editor.

3) Enter site1 in find text box and enter {{Path}} in replace text box. Click Replace All.

Now your test should be all fixed up.

In beta 2, this feature does not search string body nodes. We have fixed this post beta2. This will make it easier to parameterize parts of string bodies. With this change, you will not have to open each string body node, find part of body to replace and correctly type the new text. Again, this does not exist in Beta2 drop, but has been fixed post beta 2.

This feature should make editing your web test easier. Please try it out and let us know what you think.

VSTS 2010 Feature: Data Source Enhancements

2010-01-04T14:55:00Z

This blog post will describe the enhancements that have been made to the data source functionality in web tests. We have heard the following feedback a number of times:

I would like to be able to reload the data source while a load test is running
I want to be able to use a column in my data source that is not bound to any parameter in my web test
I want more control over how the data source is advanced.

We have provided enhancements to each of these areas.

Reloading a data source – Prior to this change, all data for the data source was read once at the beginning of the test. Now this will allow you to programmatically reload a data source while a test is running. So if you have a set of web tests that are creating data that other tests need, you will now be able to pull that data in while the test is running. The way to accomplish this is by using a WebTestPlugin. The API call is the following: void WebTest.ReloadDataTable(string dataSourceName, string dataTableName) Here is a very simple plug-in that would call this. I have a csv data source called DataSource1 and the table to Products#csv.

using Microsoft.VisualStudio.TestTools.WebTesting;

namespace TestProject4
{
    public class DatasourcePlugin : WebTestPlugin
    {
        public override void PreWebTest(object sender, PreWebTestEventArgs e)
        {
            e.WebTest.ReloadDataTable("DataSource1", "Products#csv");
        }
    }
}

Using a column in data source not bound to a parameter – This is useful if you are binding some parameter to a column but then want to use another column in a plug-in or custom validation rule. Prior to this change, you would have to bind all columns that you were going to use to some parameter in the web test. Now you just do the following.

Expand the data sources node in the web test
Click on the table that you want all the data from
Bring up the property browser
Change the Select Columns property from “Only Select Bound Columns “ to ”Select All Columns”. This will then add all of the columns into the context and make them available for use in your custom plug-ins and rules.

Controlling how data source is advanced – In VSTS 2008 and VSTS 2005 we had 3 access methods for data sources: Sequential, Random and Unique. Check out this blog post for a better description of these: Access Methods We have now added “Do Not Move Cursor Automatically” Basically, you now have full control on when the cursor is advanced. If you do nothing, the cursor stays on the first row. To move the cursor, you will use the following API call: void WebTest.MoveDatatableCursor(string dataSourceName, string dataTableName). We have also added another overload of this method which is void WebTest.MoveDatatableCursor(string dataSourceName, string dataTableName, int newRowIndex). Here is a very simple plug-in which will move the cursor to row 3 in the data source.

using Microsoft.VisualStudio.TestTools.WebTesting;

namespace TestProject4
{
    public class DatasourcePlugin : WebTestPlugin
    {
        public override void PreWebTest(object sender, PreWebTestEventArgs e)
        {
            e.WebTest.MoveDataTableCursor("DataSource1", "Products#csv", 3);
        }
    }
}

Hopefully these enhancements will make it easier to work with data sources. These are now available in beta 2. Please try them out and let us know what you think.

Debugging Custom Validation/Extraction rules, Web Test and Load Test Plug-ins

2009-11-17T02:24:00Z

Most of my recent posts have been about new features in VS 2010. This post is going to show you how to debug your custom rules. This applies to VS 2005, VS 2008 and VS 2010.

First let’s create a Test Project
Then Right click on the project node and select Add –> Web Test
Record a simple web test
Now add a new class to the project. We can re-use the sample extraction rule from this earlier post: Sample Extraction Rule Copy and paste the code from this sample into the new class file.
Compile the project
Right click on a request that has a post parameter and select the new rule:
Enter a context parameter name, name of the form field to extract and the index of the form field to extract. I am going to enter _EVENTVALIDATION. Then click OK.
The test is ready to run. If you click the play button in the web test editor the test will run and the you will see the results in the Web Test viewer. But what if your custom rule/plugin isn’t working as expected. When I playback my test, my rule failed to find the _EVENTVALIDATION field.

Debugging the rule

First we need to place a break point in the custom rule. For an extraction rule, place a break point in the extract method.
Now we need to start the test. This time we don’t want to just click play. If you hit the drop down on the web test editor play button, you will see that there are 4 options. This time choose Debug Test.
When the test runs, it will hit your break point.
Now we can step through and see why the rule failed.
In my case I can see that when I typed in the form field, I missed a _. Now if I correct my mistake and rerun, I can see the value extracted into the context.

This shows you how to debug web tests rules/plugins. But the same applies to load test plugins. The load test editor play button is also a split button. So just place your break point in the load test plug-in and choose debug test instead of play test.

I hope this helps in debugging your custom rules.

VSTS 2010 Feature: API for Processing Web Test Results

2009-11-11T15:49:00Z

One request which we have received a number of times is something like, “The Web test playback UI is great for when you are in VS, but I need to share these results with others. Can I generate a report from the web test result?” In VS 2005 and 2008, the web test result was stored in a trx file which was an XML file. We did not have a public API for processing the web test results. In VS 2010, we have made some changes in this area.

First, the web test results are no longer completely stored in the trx file. The trx file has reference to a .webtestresult file which has the actual result. Look under you TestResults directory which is located in your TestProject directory by default. In this directory, you will see a number of trx files. If you then search into the folder that corresponds with the trx and then navigate into the IN directory, you will find the .webtestresult file. This file will have the name of the web test that was being executed. So if you ran WebTest1.webtest, you will find a result file names WebTest1.webtestresult.

We have made the API for processing .webtestresult files public. So now you can process this file and create your own custom reports for web test results. I am going to walk you through a demo for how to do this. We will create a simple console app that will read in the webtestresult file and then create an XML document. This will show you how to process the result file so you can generate your own custom reports.

WebTestResult Model

There are a few classes that you will have to become familiar with. These are all in the Microsoft.Visual Studio.TestTools.WebTesting namespace.

WebTestResultDetailsSerializer – This class will be used to read the .webtestresult file.
WebTestResultDetails – This class is what the serializer will return. This contains the full result. It contains a collection of iterations. Each iteration contains a collection of WebTestResultUnit objects.
WebTestResultUnit - This is the base class for page, transaction, comments, loops, and conditional results.
WebTestResultPage – This contains information about a page including the request, response, dependent Urls, redirected Urls, timing info etc.
WebTestResultTransaction – This contains information about a transaction such as name and response time. It also contains a collection of WebTestResultUnit objects which are contained within the transaction.
WebTestResultCondition – This contains information about a web test condition such as condition being tested and whether or not the condition was met. It also contains a collection of WebTestResultUnit objects which are contained within the condition.
WebTestResultLoop – This contains information about a web test loop such as the loop condition. It contains a collection of WebTestResultLoopIteration objects. It has one object for each iteration of the loop.
WebTestResultLoopIteration – This contains information about a particular loop iteration. It also contains a collection of WebTestResultUnit objects which are contained within the loop iteration.
WebTestResultComment – This contains information about a web test comment.

Sample Console Application

Now that we have gone over the basic classes we will be using, let’s create a sample app which will create a custom report.

First launch VS and create a new Console Application project.
Right click on the project in solution explorer and select properties.
Make sure the target framework is set to .NET Framework 4 and not .NET Framework 4 client profile
Right reference node and select Add Reference…
In the Add Reference dialog select the Microsoft.Visual Studio.QualityTools.WebTestFramework.dll
Right click the project node and add a new class called ReportGenerator. Here is the code for that class. Let’s go over what is happening.
- GenerateReport – This method takes the webtestresult file. It creates and instance of the serializer which will then return the WebTestResultDetails. My sample is going to return an xml document, so I am also creating an Xml document here which I will be building up. Then you can see that it starts iterating over the Iterations. For each iteration it then iterates over the children in the iteration which are all WebTestResultUnit objects. It passes the WebTestResultUnit to ProcessResultUnit
- ProcessResultUnit – This method needs to figure out what kind of result unit it is and then do the appropriate thing. For example, for pages, it will Create a page node with the url and response times as attributes. Then it will iterates through the redirected requests and the dependent requests for the page. For a transaction, it will create a transaction node with the name and response time of the transaction. It will then iterate over the children of the transaction which are all WebTestResultUnit objects and call ProcessResultUnit for each of the children

using System.Xml;
using Microsoft.Visual Studio.TestTools.WebTesting;

namespace ConsoleApplication1
{
    internal class ReportGenerator
    {
        public ReportGenerator()
        {

}

        public XmlDocument GenerateReport(string file)
        {
            //create the serialzer
            WebTestResultDetailsSerializer serializer = new WebTestResultDetailsSerializer();

//deserialize the webtest result
WebTestResultDetails details = serializer.Deserialize(file);

            //create root element for the document
            XmlDocument doc = new XmlDocument();
            XmlElement elem = doc.CreateElement("WebTestResultDetails");
            doc.AppendChild(elem);

            //now we need to loop through each iteration
            foreach (WebTestResultIteration iteration in details.WebTestIterations)
            {
                XmlElement iterationElement = doc.CreateElement("Iteration");
                elem.AppendChild(iterationElement);
                //loop through and process each child. A Child could be a
                //transaction, page, comment, innertest, etc.
                foreach (WebTestResultUnit unit in iteration.Children)
                {
                    ProcessResultUnit(unit, doc, iterationElement);
                }
            }

return doc;
}

        private void ProcessResultUnit(WebTestResultUnit unit, XmlDocument doc, XmlElement elem)
        {
            if (unit is WebTestResultPage)
            {
                XmlElement pageElement = doc.CreateElement("Page");
                elem.AppendChild(pageElement);

                WebTestResultPage page = unit as WebTestResultPage;
                pageElement.SetAttribute("Url", page.RequestResult.Request.UrlWithQueryString);
                pageElement.SetAttribute("ResponseTime", page.RequestResult.Response.Statistics.MillisecondsToLastByte.ToString());
                //process redirects
                if (page.RedirectedPages.Count > 0)
                {
                    //create the Redirects element
                    XmlElement redirectsElement = doc.CreateElement("Redirects");
                    pageElement.AppendChild(redirectsElement);
                    foreach (WebTestResultPage redirect in page.RedirectedPages)
                    {
                        ProcessResultUnit(redirect, doc, redirectsElement);
                    }
                }

                //process each dependent
                if (page.RequestResult.DependantResults.Count > 0)
                {
                    //create the DependentRequests element
                    XmlElement dependentsElement = doc.CreateElement("DependentRequests");
                    pageElement.AppendChild(dependentsElement);
                    foreach (WebTestRequestResult request in page.RequestResult.DependantResults)
                    {
                        XmlElement dependentElement = doc.CreateElement("DependentRequest");
                        dependentsElement.AppendChild(dependentElement);
                        dependentElement.SetAttribute("Url", request.Request.UrlWithQueryString);
                    }
                }
            }
            else if (unit is WebTestResultTransaction)
            {
                WebTestResultTransaction transaction = unit as WebTestResultTransaction;

                //create the transaction element
                XmlElement transactionElement = doc.CreateElement("Transaction");
                elem.AppendChild(transactionElement);
                transactionElement.SetAttribute("Name", transaction.Name);
                transactionElement.SetAttribute("IsIncludedTest", transaction.IsIncludedTest.ToString());
                transactionElement.SetAttribute("ResponseTime", transaction.ResponseTime.ToString());

                //now we need to process each child of the transacction which can any resultunit type.
                //so iterate through children and call ProcessResultUnit
                if (transaction.Children.Count > 0)
                {
                    XmlElement children = doc.CreateElement("Children");
                    transactionElement.AppendChild(children);
                    foreach (WebTestResultUnit child in transaction.Children)
                    {
                        ProcessResultUnit(child,doc,children);
                    }
                }
            }
            else if (unit is WebTestResultCondition)
            {
                WebTestResultCondition condition = unit as WebTestResultCondition;

                //create the Condition element
                XmlElement conditionElement = doc.CreateElement("Condition");
                elem.AppendChild(conditionElement);
                conditionElement.SetAttribute("Name", condition.ConditionStringRepresentation);
                //now we need to process each child of the condition which can any resultunit type.
                //so iterate through children and call ProcessResultUnit
                if (condition.Children.Count > 0)
                {
                    XmlElement children = doc.CreateElement("Children");
                    conditionElement.AppendChild(children);
                    foreach (WebTestResultUnit child in condition.Children)
                    {
                        ProcessResultUnit(child, doc, children);
                    }
                }
            }
            else if (unit is WebTestResultLoop)
            {
                WebTestResultLoop loop = unit as WebTestResultLoop;

                //create the Loop element
                XmlElement loopElement = doc.CreateElement("Loop");
                elem.AppendChild(loopElement);
                loopElement.SetAttribute("Name", loop.LoopStringRepresentation);

                //now we need to process each child of the loop which can any resultunit type.
                //so iterate through children and call ProcessResultUnit
                if (loop.Children.Count > 0)
                {
                    XmlElement children = doc.CreateElement("LoopIterations");
                    loopElement.AppendChild(children);
                    foreach (WebTestResultUnit child in loop.Children)
                    {
                        ProcessResultUnit(child, doc, children);
                    }
                }
            }
            else if (unit is WebTestResultLoopIteration)
            {
                WebTestResultLoopIteration loopIteration = unit as WebTestResultLoopIteration;

                //create the LoopIteration element
                XmlElement loopIterationElement = doc.CreateElement("LoopIteration");
                elem.AppendChild(loopIterationElement);
                loopIterationElement.SetAttribute("Number", loopIteration.IterationNumber.ToString());
                loopIterationElement.SetAttribute("IsConditionalRuleMet", loopIteration.IsConditionalRuleMet().ToString());

                //now we need to process each child of the loop which can any resultunit type.
                //so iterate through children and call ProcessResultUnit
                if (loopIteration.Children.Count > 0)
                {
                    XmlElement children = doc.CreateElement("Children");
                    loopIterationElement.AppendChild(children);
                    foreach (WebTestResultUnit child in loopIteration.Children)
                    {
                        ProcessResultUnit(child,doc,children);
                    }
                }
            }
            else if (unit is WebTestResultComment)
            {
                WebTestResultComment comment = unit as WebTestResultComment;

                //create the Comment element
                XmlElement commentElement = doc.CreateElement("Comment");
                elem.AppendChild(commentElement);
                commentElement.SetAttribute("Text", comment.Comment);
            }
        }
    }
}

Now go back to Program.cs file. This class just needs to know where the results file is and where to store the output file. So we will check to make sure we get this information and then pass it along the report generator. This class looks like this:

using System;
using System.Xml;
using System.IO;

namespace ConsoleApplication1
{
    class Program
    {
        static void Main(string[] args)
        {
            if (args.Length != 2)
            {
                Console.WriteLine("Usage: ReportGenerator <WebTestResultFile> <OutputFile>");
                return;
            }

            if (!File.Exists(args[0]))
            {
                Console.WriteLine("WebTestResult file does not exist");
                return;
            }

            if (File.Exists(args[1]))
            {
                Console.WriteLine("Output file already exists. Please specify a new name.");
                return;
            }

ReportGenerator generator = new ReportGenerator();
XmlDocument report = generator.GenerateReport(args[0]);

            XmlTextWriter writer = new XmlTextWriter(args[1], null);
            writer.Formatting = Formatting.Indented;
            report.Save(writer);
        }
    }
}

Now compile your code. to run it, open a command prompt and go to directory that the exe was compiled into. You would run it with something like: ReportGenerator c:\Webtest1.webtestresult c:\NewReport.xml

Here is a sample output created from one of my webtests:

<?xml version="1.0"?>
<WebTestResultDetails>
<Iteration>
    <Transaction Name="Transaction1" IsIncludedTest="False" ResponseTime="88.384">
      <Children>
        <Transaction Name="WebTest8.Storecsvs" IsIncludedTest="True" ResponseTime="56.825">
          <Children>
            <Page Url="http://sampleserver/Storecsvs" ResponseTime="28829">
              <Redirects>
                <Page Url="http://sampleserver/Storecsvs/" ResponseTime="27996">
                  <DependentRequests>
                    <DependentRequest Url="http://sampleserver/Storecsvs/IBuySpy.css" />
                    <DependentRequest Url="http://sampleserver/Storecsvs/images/grid_background.gif" />
                  </DependentRequests>
                </Page>
              </Redirects>
            </Page>
          </Children>
        </Transaction>
        <Page Url="http://sampleserver/Storecsvs/productslist.aspx?CategoryID=15&selection=1" ResponseTime="27737">
          <DependentRequests>
            <DependentRequest Url="http://sampleserver/Storecsvs/images/sitebkgrdnogray.gif" />
            <DependentRequest Url="http://sampleserver/Storecsvs/IBuySpy.css" />
          </DependentRequests>
        </Page>
        <Page Url="http://sampleserver/Storecsvs/ProductDetails.aspx?productID=394" ResponseTime="316">
          <DependentRequests>
            <DependentRequest Url="http://sampleserver/Storecsvs/IBuySpy.css" />
            <DependentRequest Url="http://sampleserver/Storecsvs/images/sitebkgrd.gif" />
          </DependentRequests>
        </Page>
      </Children>
    </Transaction>
    <Comment Text="Sample Comment" />
    <Page Url="http://sampleserver/Storecsvs/AddToCart.aspx?ProductID=394" ResponseTime="251">
      <Redirects>
        <Page Url="http://sampleserver/Storecsvs/ShoppingCart.aspx" ResponseTime="0">
          <DependentRequests>
            <DependentRequest Url="http://sampleserver/Storecsvs/images/sitebkgrd.gif" />
            <DependentRequest Url="http://sampleserver/Storecsvs/IBuySpy.css" />
          </DependentRequests>
        </Page>
      </Redirects>
    </Page>
    <Loop Name="Loop ( Repeat 1 times )">
      <LoopIterations>
        <LoopIteration Number="1" IsConditionalRuleMet="True">
          <Children>
            <Page Url="http://sampleserver/Storecsvs/ShoppingCart.aspx" ResponseTime="27729">
              <DependentRequests>
                <DependentRequest Url="http://sampleserver/Storecsvs/IBuySpy.css" />
                <DependentRequest Url="http://sampleserver/Storecsvs/images/sitebkgrd.gif" />
              </DependentRequests>
            </Page>
          </Children>
        </LoopIteration>
        <LoopIteration Number="2" IsConditionalRuleMet="False" />
      </LoopIterations>
    </Loop>
    <Page Url="http://sampleserver/Storecsvs/ShoppingCart.aspx" ResponseTime="542">
      <DependentRequests>
        <DependentRequest Url="http://sampleserver/Storecsvs/images/sitebkgrd.gif" />
        <DependentRequest Url="http://sampleserver/Storecsvs/IBuySpy.css" />
      </DependentRequests>
    </Page>
</Iteration>
</WebTestResultDetails>

I hope this helps getting you started in creating your own custom web test result reports. There is plenty of other information available to you. My sample just gives you an idea of how to parse through the objects. More or less anything that you see in the web test playback UI is available in the result objects.

VSTS 2010 Feature: Load Testing Run Comparison Report in Excel

2009-11-07T00:43:51Z

This blog post is going to be about one of my favorite new features added for Beta2. In Beta1 we added excel integration for reporting on Load Test Reports. In beta1, the report that was added was a trend report which would give you a feel for how certain counters changed from run to run. Check out this blog post for more info on that report: Excel Trend Report

The new report type which has been added gives a more detailed comparison of 2 runs. Let’s walk through creating the new report and looking at the output.

1) Launch Excel and click on the load test ribbon. The click the Load Test Report Button.

2) In the first page of the wizard enter the database server and select the database. Then click next.

3) On second page of the wizard choose “Create a Report” and click Next.

4) On third page, select Run Comparison as the report type. Then click next.

5) On fourth page, Enter a name for the report and choose a load test that you want to use for report generation. Click next.

6) The fifth page is where you select the runs to compare. Select 2 runs and click next.

7) The sixth page is where you select the counters that you want included on the report. You will notice that there are some counters which have been pre-selected. These are just some default counters. Select whichever counters you want.

8) Then click finish and the report will be generated.

Now let’s take a look at what was generated for you.

1) Table of Contents – This page gives you a list and links to each page in the report.

2) Runs – This page gives a little info about the 2 runs which were selected.

3) The next three sheets give a comparison of each test, page and transaction in the load test.

4) Test Comparison – The page and transaction sheets will be very similar to this. The 2 charts at the top will show the top 5 tests which had the largest performance improvement and top 5 tests which showed the worst regressions. The table underneath the chart will show all tests in the load test. It lists out the scenario and test name. Then gives the Baseline value and Comparison value followed by a % change from baseline. There is also a color scale added to help you quickly identify problems.

5) Page Comparison – This is very similar to the test sheet, but gives a breakdown of each page. The other difference is that it includes the response goal value and gives you a % change from goal. So if you set page goals, they would appear here.

6) Transaction Comparison – Again, this is similar to test and page sheets.

7) Machine Comparison – This page will compare counters between the two runs. As you can see from the screen shot below, it lists the Machines used in the Baseline and Comparison runs as well as the roles for the machine. The role is important because in order for two machines to be compared the roles need to match. For example, if MachineA was the IIS server in Baseline and was SQL Server in Comparison run, then the report will not compare that machine between the 2 runs. You will still have the counters for that machine listed, but it will not line them up for a comparison. Now if MachineA was the IIS server in Baseline run and MachineB was IIS role for comparison run, those 2 machines will be compared. This is useful if you will be doing many runs, but are not guaranteed the same set of machines for every run you do. As you can see from my report, TeamTestWeb1 played the role of IIS and SQL Server for both of my runs which is why they were compared.

8) Error Comparison – This page give you a breakdown of errors in each run and % change between the runs.

As mentioned in the previous blog on excel reporting, the report definition that you generate is saved in the load test database. So you can come back later and generate the report again, or use that as a template to generate other run comparison reports.

I hope you find this report useful and let us know what you think.