Sean Lumley's Blog

Debugging Custom Validation/Extraction rules, Web Test and Load Test Plug-ins

Most of my recent posts have been about new features in VS 2010.  This post is going to show you how to debug your custom rules.  This applies to VS 2005, VS 2008 and VS 2010. 

1. First let’s create a Test Project
2. Then Right click on the project node and select Add –>  Web Test
3. Record a simple web test
4. Now add a new class to the project.  We can re-use the sample extraction rule from this earlier post: Sample Extraction Rule  Copy and paste the code from this sample into the new class file.
5. Compile the project
6. Right click on a request that has a post parameter and select the new rule:
7. 
8. Enter a context parameter name, name of the form field to extract and the index of the form field to extract.  I am going to enter _EVENTVALIDATION. Then click OK.
9. The test is ready to run.  If you click the play button in the web test editor the test will run and the you will see the results in the Web Test viewer.  But what if your custom rule/plugin isn’t working as expected.  When I playback my test, my rule failed to find the _EVENTVALIDATION field.  

Debugging the rule

1. First we need to place a break point in the custom rule.  For an extraction rule, place a break point in the extract method. 
2. Now we need to start the test.  This time we don’t want to just click play.  If you hit the drop down on the web test editor play button, you will see that there are 4 options.  This time choose Debug Test.                                                  
3. When the test runs, it will hit your break point.                                                                                  
4. Now we can step through and see why the rule failed.                                                                 
5. In my case I can see that when I typed in the form field, I missed a _.  Now if I correct my mistake and rerun, I can see the value extracted into the context.                                                                                                                                 

This shows you how to debug web tests rules/plugins.  But the same applies to load test plugins.  The load test editor play button is also a split button.  So just place your break point in the load test plug-in and choose debug test instead of play test. 

I hope this helps in debugging your custom rules.

VSTS 2010 Feature: API for Processing Web Test Results

One request which we have received a number of times is something like, “The Web test playback UI is great for when you are in VS, but I need to share these results with others.  Can I generate a report from the web test result?” In VS 2005 and 2008, the web test result was stored in a trx file which was an XML file.  We did not have a public API for processing the web test results.  In VS 2010, we have made some changes in this area. 

First, the web test results are no longer completely stored in the trx file.  The trx file has reference to a .webtestresult file which has the actual result. Look under you TestResults directory which is located in your TestProject directory by default.  In this directory, you will see a number of trx files.  If you then search into the folder that corresponds with the trx and then navigate into the IN directory, you will find the .webtestresult file.  This file will have the name of the web test that was being executed.  So if you ran WebTest1.webtest, you will find a result file names WebTest1.webtestresult. 

We have made the API for processing .webtestresult files public.  So now you can process this file and create your own custom reports for web test results.  I am going to walk you through a demo for how to do this.  We will create a simple console app that will read in the webtestresult file and then create an XML document.  This will show you how to process the result file so you can generate your own custom reports.

WebTestResult Model

There are a few classes that you will have to become familiar with.  These are all in the Microsoft.Visual Studio.TestTools.WebTesting namespace.

1. WebTestResultDetailsSerializer – This class will be used to read the .webtestresult file.
2. WebTestResultDetails – This class is what the serializer will return.  This contains the full result.  It contains a collection of iterations.  Each iteration contains a collection of WebTestResultUnit objects.
3. WebTestResultUnit -  This is the base class for page, transaction, comments, loops, and conditional results.
4. WebTestResultPage – This contains information about a page including the request, response, dependent Urls, redirected Urls, timing info etc.
5. WebTestResultTransaction – This contains information about a transaction such as name and response time.  It also contains a collection of WebTestResultUnit objects which are contained within the transaction.
6. WebTestResultCondition – This contains information about a web test condition such as condition being tested and whether or not the condition was met.  It also contains a collection of WebTestResultUnit objects which are contained within the condition.
7. WebTestResultLoop – This contains information about a web test loop such as the loop condition.  It contains a collection of WebTestResultLoopIteration objects.  It has one object for each iteration of the loop.
8. WebTestResultLoopIteration – This contains information about a particular loop iteration.  It also contains a collection of WebTestResultUnit objects which are contained within the loop iteration.
9. WebTestResultComment – This contains information about a web test comment.

 

Sample Console Application

Now that we have gone over the basic classes we will be using, let’s create a sample app which will create a custom report.

1. First launch VS and create a new Console Application project.
2. Right click on the project in solution explorer and select properties.
3. Make sure the target framework is set to .NET Framework 4 and not .NET Framework 4 client profile
4. Right reference node and select Add Reference…
5. In the Add Reference dialog select the Microsoft.Visual Studio.QualityTools.WebTestFramework.dll
6. Right click the project node and add a new class called ReportGenerator.  Here is the code for that class.  Let’s go over what is happening.
   • GenerateReport – This method takes the webtestresult file.  It creates and instance of the serializer which will then return the WebTestResultDetails.  My sample is going to return an xml document, so I am also creating an Xml document here which I will be building up.  Then you can see that it starts iterating over the Iterations.  For each iteration it then iterates over the children in the iteration which are all WebTestResultUnit objects.  It passes the WebTestResultUnit to ProcessResultUnit
   • ProcessResultUnit – This method needs to figure out what kind of result unit it is and then do the appropriate thing.  For example, for pages, it will Create a page node with the url and response times as attributes.  Then it will iterates through the redirected requests and the dependent requests for the page.  For a transaction, it will create a transaction node with the name and response time of the transaction.  It will then iterate over the children of the transaction which are all WebTestResultUnit objects and call ProcessResultUnit for each of the children

using System.Xml;
using Microsoft.Visual Studio.TestTools.WebTesting;

namespace ConsoleApplication1
{
    internal class ReportGenerator
    {
        public ReportGenerator()
        {

        }

        public XmlDocument GenerateReport(string file)
        {
            //create the serialzer
            WebTestResultDetailsSerializer serializer = new WebTestResultDetailsSerializer();

            //deserialize the webtest result
            WebTestResultDetails details = serializer.Deserialize(file);

            //create root element for the document
            XmlDocument doc = new XmlDocument();
            XmlElement elem = doc.CreateElement("WebTestResultDetails");
            doc.AppendChild(elem);

            //now we need to loop through each iteration
            foreach (WebTestResultIteration iteration in details.WebTestIterations)
            {
                XmlElement iterationElement = doc.CreateElement("Iteration");
                elem.AppendChild(iterationElement);
                //loop through and process each child.  A Child could be a
                //transaction, page, comment, innertest, etc.
                foreach (WebTestResultUnit unit in iteration.Children)
                {
                    ProcessResultUnit(unit, doc, iterationElement);
                }
            }

            return doc;
        }

        private void ProcessResultUnit(WebTestResultUnit unit, XmlDocument doc, XmlElement elem)
        {
            if (unit is WebTestResultPage)
            {
                XmlElement pageElement = doc.CreateElement("Page");
                elem.AppendChild(pageElement);

                WebTestResultPage page = unit as WebTestResultPage;
                pageElement.SetAttribute("Url", page.RequestResult.Request.UrlWithQueryString);               
                pageElement.SetAttribute("ResponseTime", page.RequestResult.Response.Statistics.MillisecondsToLastByte.ToString());
                //process redirects
                if (page.RedirectedPages.Count > 0)
                {
                    //create the Redirects element
                    XmlElement redirectsElement = doc.CreateElement("Redirects");
                    pageElement.AppendChild(redirectsElement);
                    foreach (WebTestResultPage redirect in page.RedirectedPages)
                    {
                        ProcessResultUnit(redirect, doc, redirectsElement);
                    }
                }

                //process each dependent
                if (page.RequestResult.DependantResults.Count > 0)
                {
                    //create the DependentRequests element
                    XmlElement dependentsElement = doc.CreateElement("DependentRequests");
                    pageElement.AppendChild(dependentsElement);
                    foreach (WebTestRequestResult request in page.RequestResult.DependantResults)
                    {
                        XmlElement dependentElement = doc.CreateElement("DependentRequest");
                        dependentsElement.AppendChild(dependentElement);
                        dependentElement.SetAttribute("Url", request.Request.UrlWithQueryString);                       
                    }
                }
            }
            else if (unit is WebTestResultTransaction)
            {
                WebTestResultTransaction transaction = unit as WebTestResultTransaction;

                //create the transaction element
                XmlElement transactionElement = doc.CreateElement("Transaction");
                elem.AppendChild(transactionElement);
                transactionElement.SetAttribute("Name", transaction.Name);
                transactionElement.SetAttribute("IsIncludedTest", transaction.IsIncludedTest.ToString());
                transactionElement.SetAttribute("ResponseTime", transaction.ResponseTime.ToString());

                //now we need to process each child of the transacction which can any resultunit type. 
                //so iterate through children and call ProcessResultUnit
                if (transaction.Children.Count > 0)
                {
                    XmlElement children = doc.CreateElement("Children");
                    transactionElement.AppendChild(children);
                    foreach (WebTestResultUnit child in transaction.Children)
                    {
                        ProcessResultUnit(child,doc,children);
                    }
                }
            }
            else if (unit is WebTestResultCondition)
            {
                WebTestResultCondition condition = unit as WebTestResultCondition;

                //create the Condition element
                XmlElement conditionElement = doc.CreateElement("Condition");
                elem.AppendChild(conditionElement);
                conditionElement.SetAttribute("Name", condition.ConditionStringRepresentation);
                //now we need to process each child of the condition which can any resultunit type. 
                //so iterate through children and call ProcessResultUnit
                if (condition.Children.Count > 0)
                {
                    XmlElement children = doc.CreateElement("Children");
                    conditionElement.AppendChild(children);
                    foreach (WebTestResultUnit child in condition.Children)
                    {
                        ProcessResultUnit(child, doc, children);
                    }
                }
            }
            else if (unit is WebTestResultLoop)
            {
                WebTestResultLoop loop = unit as WebTestResultLoop;

                //create the Loop element
                XmlElement loopElement = doc.CreateElement("Loop");
                elem.AppendChild(loopElement);
                loopElement.SetAttribute("Name", loop.LoopStringRepresentation);               

                //now we need to process each child of the loop which can any resultunit type. 
                //so iterate through children and call ProcessResultUnit
                if (loop.Children.Count > 0)
                {
                    XmlElement children = doc.CreateElement("LoopIterations");
                    loopElement.AppendChild(children);
                    foreach (WebTestResultUnit child in loop.Children)
                    {
                        ProcessResultUnit(child, doc, children);
                    }
                }
            }
            else if (unit is WebTestResultLoopIteration)
            {
                WebTestResultLoopIteration loopIteration = unit as WebTestResultLoopIteration;

                //create the LoopIteration element
                XmlElement loopIterationElement = doc.CreateElement("LoopIteration");
                elem.AppendChild(loopIterationElement);
                loopIterationElement.SetAttribute("Number", loopIteration.IterationNumber.ToString());
                loopIterationElement.SetAttribute("IsConditionalRuleMet", loopIteration.IsConditionalRuleMet().ToString());               

                //now we need to process each child of the loop which can any resultunit type. 
                //so iterate through children and call ProcessResultUnit
                if (loopIteration.Children.Count > 0)
                {
                    XmlElement children = doc.CreateElement("Children");
                    loopIterationElement.AppendChild(children);
                    foreach (WebTestResultUnit child in loopIteration.Children)
                    {
                        ProcessResultUnit(child,doc,children);
                    }
                }
            }
            else if (unit is WebTestResultComment)
            {
                WebTestResultComment comment = unit as WebTestResultComment;

                //create the Comment element
                XmlElement commentElement = doc.CreateElement("Comment");
                elem.AppendChild(commentElement);
                commentElement.SetAttribute("Text", comment.Comment);                                          
            }
        }
    }
}

1. Now go back to Program.cs file.  This class just needs to know where the results file is and where to store the output file.  So we will check to make sure we get this information and then pass it along the report generator.  This class looks like this:

using System;
using System.Xml;
using System.IO;

namespace ConsoleApplication1
{
    class Program
    {
        static void Main(string[] args)
        {
            if (args.Length != 2)
            {
                Console.WriteLine("Usage:  ReportGenerator <WebTestResultFile> <OutputFile>");
                return;
            }

            if (!File.Exists(args[0]))
            {
                Console.WriteLine("WebTestResult file does not exist");
                return;
            }

            if (File.Exists(args[1]))
            {
                Console.WriteLine("Output file already exists.  Please specify a new name.");
                return;
            }

            ReportGenerator generator = new ReportGenerator();
            XmlDocument report = generator.GenerateReport(args[0]);

            XmlTextWriter writer = new XmlTextWriter(args[1], null);
            writer.Formatting = Formatting.Indented;
            report.Save(writer);
        }
    }
}

 

Now compile your code.  to run it, open a command prompt and go to directory that the exe was compiled into.  You would run it with something like:  ReportGenerator c:\Webtest1.webtestresult c:\NewReport.xml

 

Here is a sample output created from one of my webtests:

 

<?xml version="1.0"?>
<WebTestResultDetails>
  <Iteration>
    <Transaction Name="Transaction1" IsIncludedTest="False" ResponseTime="88.384">
      <Children>
        <Transaction Name="WebTest8.Storecsvs" IsIncludedTest="True" ResponseTime="56.825">
          <Children>
            <Page Url="http://sampleserver/Storecsvs" ResponseTime="28829">
              <Redirects>
                <Page Url="http://sampleserver/Storecsvs/" ResponseTime="27996">
                  <DependentRequests>
                    <DependentRequest Url="http://sampleserver/Storecsvs/IBuySpy.css" />
                    <DependentRequest Url="http://sampleserver/Storecsvs/images/grid_background.gif" />                   
                  </DependentRequests>
                </Page>
              </Redirects>
            </Page>
          </Children>
        </Transaction>
        <Page Url="http://sampleserver/Storecsvs/productslist.aspx?CategoryID=15&amp;selection=1" ResponseTime="27737">
          <DependentRequests>
            <DependentRequest Url="http://sampleserver/Storecsvs/images/sitebkgrdnogray.gif" />
            <DependentRequest Url="http://sampleserver/Storecsvs/IBuySpy.css" />           
          </DependentRequests>
        </Page>
        <Page Url="http://sampleserver/Storecsvs/ProductDetails.aspx?productID=394" ResponseTime="316">
          <DependentRequests>
            <DependentRequest Url="http://sampleserver/Storecsvs/IBuySpy.css" />
            <DependentRequest Url="http://sampleserver/Storecsvs/images/sitebkgrd.gif" />           
          </DependentRequests>
        </Page>
      </Children>
    </Transaction>
    <Comment Text="Sample Comment" />
    <Page Url="http://sampleserver/Storecsvs/AddToCart.aspx?ProductID=394" ResponseTime="251">
      <Redirects>
        <Page Url="http://sampleserver/Storecsvs/ShoppingCart.aspx" ResponseTime="0">
          <DependentRequests>
            <DependentRequest Url="http://sampleserver/Storecsvs/images/sitebkgrd.gif" />
            <DependentRequest Url="http://sampleserver/Storecsvs/IBuySpy.css" />           
          </DependentRequests>
        </Page>
      </Redirects>
    </Page>
    <Loop Name="Loop ( Repeat 1 times )">
      <LoopIterations>
        <LoopIteration Number="1" IsConditionalRuleMet="True">
          <Children>
            <Page Url="http://sampleserver/Storecsvs/ShoppingCart.aspx" ResponseTime="27729">
              <DependentRequests>
                <DependentRequest Url="http://sampleserver/Storecsvs/IBuySpy.css" />
                <DependentRequest Url="http://sampleserver/Storecsvs/images/sitebkgrd.gif" />               
              </DependentRequests>
            </Page>
          </Children>
        </LoopIteration>
        <LoopIteration Number="2" IsConditionalRuleMet="False" />
      </LoopIterations>
    </Loop>
    <Page Url="http://sampleserver/Storecsvs/ShoppingCart.aspx" ResponseTime="542">
      <DependentRequests>
        <DependentRequest Url="http://sampleserver/Storecsvs/images/sitebkgrd.gif" />
        <DependentRequest Url="http://sampleserver/Storecsvs/IBuySpy.css" />       
      </DependentRequests>
    </Page>
  </Iteration>
</WebTestResultDetails>

 

I hope this helps getting you started in creating your own custom web test result reports.  There is plenty of other information available to you.  My sample just gives you an idea of how to parse through the objects.  More or less anything that you see in the web test playback UI is available in the result objects.

VSTS 2010 Feature: Load Testing Run Comparison Report in Excel

This blog post is going to be about one of my favorite new features added for Beta2. In Beta1 we added excel integration for reporting on Load Test Reports. In beta1, the report that was added was a trend report which would give you a feel for how certain counters changed from run to run. Check out this blog post for more info on that report: Excel Trend Report

The new report type which has been added gives a more detailed comparison of 2 runs. Let’s walk through creating the new report and looking at the output.

1) Launch Excel and click on the load test ribbon. The click the Load Test Report Button.

2) In the first page of the wizard enter the database server and select the database. Then click next.

3) On second page of the wizard choose “Create a Report” and click Next.

4) On third page, select Run Comparison as the report type. Then click next.

5) On fourth page, Enter a name for the report and choose a load test that you want to use for report generation. Click next.

6) The fifth page is where you select the runs to compare. Select 2 runs and click next.

7) The sixth page is where you select the counters that you want included on the report. You will notice that there are some counters which have been pre-selected. These are just some default counters. Select whichever counters you want.

8) Then click finish and the report will be generated.

Now let’s take a look at what was generated for you.

1) Table of Contents – This page gives you a list and links to each page in the report.

2) Runs – This page gives a little info about the 2 runs which were selected.

3) The next three sheets give a comparison of each test, page and transaction in the load test.

4) Test Comparison – The page and transaction sheets will be very similar to this. The 2 charts at the top will show the top 5 tests which had the largest performance improvement and top 5 tests which showed the worst regressions. The table underneath the chart will show all tests in the load test. It lists out the scenario and test name. Then gives the Baseline value and Comparison value followed by a % change from baseline. There is also a color scale added to help you quickly identify problems.

5) Page Comparison – This is very similar to the test sheet, but gives a breakdown of each page. The other difference is that it includes the response goal value and gives you a % change from goal. So if you set page goals, they would appear here.

6) Transaction Comparison – Again, this is similar to test and page sheets.

7) Machine Comparison – This page will compare counters between the two runs. As you can see from the screen shot below, it lists the Machines used in the Baseline and Comparison runs as well as the roles for the machine. The role is important because in order for two machines to be compared the roles need to match. For example, if MachineA was the IIS server in Baseline and was SQL Server in Comparison run, then the report will not compare that machine between the 2 runs. You will still have the counters for that machine listed, but it will not line them up for a comparison. Now if MachineA was the IIS server in Baseline run and MachineB was IIS role for comparison run, those 2 machines will be compared. This is useful if you will be doing many runs, but are not guaranteed the same set of machines for every run you do. As you can see from my report, TeamTestWeb1 played the role of IIS and SQL Server for both of my runs which is why they were compared.

8) Error Comparison – This page give you a breakdown of errors in each run and % change between the runs.

As mentioned in the previous blog on excel reporting, the report definition that you generate is saved in the load test database. So you can come back later and generate the report again, or use that as a template to generate other run comparison reports.

I hope you find this report useful and let us know what you think.

VSTS 2010 Feature: Making it easy to debug web tests using Fiddler

VSTS 2010 Beta2 is out and there are a number of blog posts I want to write on the features now available to you.  Check out Ed Glas’s blog post on the full set: Beta2 Features

The first small feature that I am going to talk about is better integration with Fiddler.  If you are not familiar with Fiddler, it is Web Debugging Proxy and is very useful when debugging web tests.  You can find Fiddler here: Fiddler  There are times when debugging a web test that it is helpful to get another view of the data that is being sent to the servers you are testing.  It has been possible to use fiddler for these purposes since VS 2005, but we have made it much easier.  In VS 2005 or VS 2008 or even VS 2010 Beta1, you had to write a small bit of code to correctly hook up to fiddler.  The code you would have to add looked like the following and needed to be added in the constructor of a coded test or a in a web test plug-in:

this.Proxy = "http://localhost:8888";
WebProxy webProxy = (WebProxy)this.WebProxy;
webProxy.BypassProxyOnLocal = false;

 

With the beta2 release we have made this even easier.  Now to get Fiddler to capture the traffic from your web tests, you just need to do the following:

1. Launch Fiddler
2. Play your web test

That is it.  No code to write or any extra configuration.  We have only made this work when you are running a web test by itself.  If the web test is running as part of a load test, the traffic will not be routed through fiddler.  You could still use the old method to get this to work if needed.

I will be writing a few more blog posts in the coming weeks on the new features, so please try them out and let us know what you think.

VSTS 2010 Feature: Load test virtual user activity visualization

This is another post about new VS 2010 web/load testing features. You can get the beta here: beta download

In this post I am going to talk about a new load test feature called “Load test virtual user activity visualization”. This feature is a new way to look at load test data. In this feature we will provide a look at what users are doing during a load test. It will help answer the question, “I see a spike in CPU or a drop in requests/sec, what tests or pages are running during this spike or drop?  Also are they passing or failing?”. Prior to VSTS 2010, we could answer this question, but not easily. Now you will be able to just right click on a graph in the load test analyzer and select Go to User Detail.

Configuring Load Test to Collect Full Details

In order to show the detail view, you need to configure your load test to collect full details. In this mode we collect detailed information about every test, page and transaction. This option was available in VS 2005 and VS 2008, but once the data was collected, the only way to access this info was by SQLqueries. In VS 2005 and 2008 this was not the default behavior. In VS2010, the default behavior is to collect full details. If you are upgrading a VS 2008 or 2005 load test, do the following to enable full detail collection.

1) Open the load test in the load test editor

2) Click on the Run Settings node

3) Bring up the property browser

4) Click “Timing Details Storage” and select “All Individual Details”

Accessing User Detail View

The user detail view is only available once the test is complete. After the test finishes, you will see the following dialog:

Choose Yes to enable the post run options. There are 2 ways to get the user detail view. First you can click the User Detail button on the toolbar.

Or you can right click on a graph and select “Go to user detail”.  If you use this option, the user detail view will auto zoom to the part of the test that you right clicked on.  So if you click on about the 30 second mark, then the detail view will have the 30 second mark plus/minus another number of seconds.

User Detail View

Main Screen

I executed a load test with 2 web tests that access the IBuySpy website.  One test just browses the site and the other will browse and place an order.  The tests also have transactions.  The load test has 25 users executing tests for ten minutes.  When you access the user detail view you will see the following:

Let go over what you are looking at. Each horizontal row represents a virtual user in the test. Each line in a row represents one unit of whatever you are currently looking at: Test, Page, or transaction. By default the initial view will display test. So each line in the row represents a test. The length of the line represents the duration of the test and the color represents which test was executed. Look at the legend on the left to see what each color represents.

Legend

Now let’s go over what is in the legend. Here is what it looks like:

At the top of the legend is a combo box that lets you choose what type of result you want to look at: Test, Page or transaction. Change the combo to have the view refreshed into a new result type. The first 2 options below the combo are “Show only results with errors” and “Show only results with logs”. If you check Show only results with errors, the graph will remove successful results. This can be useful, because you might see that one test is failing often during time that tests/ sec drop or CPU spikes.  This is what this looks like when selected:

As you can see, when this happens you are left with just checkout tests.  So this indicates that the browse test seems to be working, but there appears to be a problem with the checkout process.

The next 2 options are “Highlight Errors” and “Highlight results with logs”. Many times the tests that are causing errors will cause the spike in cpu. Or maybe you will notice that a certain test fails while another test is running or immediately after another test has executed.

When you click highlight errors, each test that failed will be turned red. This will help you get an idea of what tests are failing and it might show a pattern that indicates why a certain condition such as a spike in CPU is occurring. Here is what this looks like:

This now shows us which tests are failing, but it also shows us that there are Checkout tests which are passing.  So now we know that the checkout process sometimes works, but when we run it under load, it will also fail.

The other option is highlight tests with logs. If you select this, each test that has as test log will be turned green. This will show you tests that you can get additional information on. Check out this blog for how to enable test log collection: Test Log Collection.

Next you will notice that each test/transaction/page has a check box next to it. If you uncheck the check box, that test/transaction/page will be removed from the view. For example,

I unchecked the Browse test, so now I see just Checkout tests.

Tooltips

Another nice part of this view is that you can get very detailed information in a tooltip by hovering a line. For example, when I hover the top line in the view, I see the following:

As you can see, you get information about the user, scenario, test, outcome, network , start time, duration and  machine executed on.  Also if there is a test log associated with the result, you will see a hyper link with test log. If you click the test log hyper link, the log for that test will appear. For a web test, the web test result will be opened in the web test playback UI. If it was a unit test, you will have the unit test result viewer opened. For example if I click on the above hyper link, I see:

 

So now I can get the full web test result and can see the failure.  So my checkout test hits a primary key violation during the checkout request.  So this is really nice when you click highlight errors, hover a result with an error and then pull up the failed test.

Here is what the tooltip looks like if I hover a page.  You can see from here that you get the URL as well as what browser was being emulated.   Also you can drill into the test log for the page if a log was collected

Scrolling and Zooming

You may have noticed the scroll/zoom bar at the bottom.  This works exactly like the scroll/zoom bar on the regular graphs.  You can scroll through different test intervals and zoom down to one sample interval or expand it to show the entire test.  For example, here is my entire test run.

This gives you the 10,000 foot view of what is happening in your test.  For example, you might see a particular iteration of a test that takes much longer than other iterations of same test.  Again you can highlight errors and see the following:

This shows me that my checkout tests passed initially and then they all failed.

You can also highlight a part of the view and zoom to that area.  So if I highlight the following:

Then my graph will automatically zoom to this area.

You will also see a vertical scroll bar if there are too many users to display on the window.

Try it out and let us know how you like it

I will follow this post up with other examples of how to use this view and what patterns you may see, such as deadlock type problems.  But hopefully you see the value in now being able to answer the question of what is happening at a certain point in the load test.  You can quickly jump to the user detail and see the tests/pages/transactions, then even pull up a test log for the part of the test you are looking at.  Please provide feedback once you have a chance to try this out.

VSTS 2010 Feature: Extensible recorder plugins for modifying recorded web tests (including adding custom dynamic parameter correlation)

This is another post about new VS 2010 web/load testing features.  You can get the beta here:  beta download.

In this post I am going to talk about a new feature that can help with web test recording.  The feature is extensible recorder plug-ins for modifying recorded web tests.  Basically we are giving you the opportunity to modify the recorded web test after you click stop on the web test recorder bar but prior to the web test being fully saved back to the web test editor.  So what problems does this help with?  The main one is performing your own custom correlation.  In VS 2008 we added a process which runs post recording that attempts to find dynamic fields.  You can read this blog post for more information: http://blogs.msdn.com/slumley/pages/web-test-correlation-helper-feature-in-orcas.aspx 

This process still exists, but this process does not always find all dynamic fields for an application.  So if we did not find the dynamic fields in your application you had to manually perform the correlation process.  Here is a blog post that goes into detail about the manual process:   http://blogs.msdn.com/slumley/pages/how-to-debug-a-web-test.aspx  Also there are cases that our correlation process does not find the dynamic values, such as dynamic values in the URL.

At a high level, you have to:

1)      Determine what parameters are dynamic

2)      Then for each parameter find the first occurrence of this in a response body.

3)      Add an extraction rule to pull the value out of the response and add it to the context

4)      Then modify each query string or form post parameter that needs this value by changing the value to pull the value out of the context.

This new feature allows you to write your own plug-in which can perform correlation or modify the web test in many ways prior to it being saved back to the web test editor.  So once you figure out that certain dynamic variable have to be correlated for each of your recordings, you can automate the process.  To demonstrate how this works, I am going to write a recorder plug-in which will perform the correlation that I manually walked through in my previous post.  Please quickly read that: http://blogs.msdn.com/slumley/pages/vs-2010-feature-web-test-playback-enhancements.aspx

Overview

Create the plug-in

Recorder plug-ins follow the same pattern as WebTestPlugins or WebTestRequestPlugins.  To create a plug-in, you will create a class that extends WebTestRecorderPlugin and then override the PostWebTestRecording method:

 

public class Class1 : WebTestRecorderPlugin

    {

        public override void PostWebTestRecording(object sender, PostWebTestRecordingEventArgs e)

        {

            base.PostWebTestRecording(sender, e);

        }

    }

Modify the web test

The event args will give you 2 main objects to work with: the recorded result and the recorded web test.  This will allow you to iterate through the result looking for certain values and then jump to the same request in the web test to make modifications.  You can also just modify the web test if you wanted to add a context parameter or maybe parameterize parts of the URL.  If you do modify the web test, you also need to set the  ReocrdedWebTestModified property to true. e.RecordedWebTestModified = true;

 

Deploy the plug-in

After compiling the plug-in, you will need to place the dll in 1 of 2 spots:

1)  Program Files\Microsoft Visual Studio 10.0\Common7\IDE\PrivateAssemblies\WebTestRecorderPlugins

2)  %USERPROFILE%\My Documents\Visual Studio 10\WebTestRecorderPlugins

Executing the plug-in

After you deploy the plug-in, you will need to restart VS for the plug-in to be picked up.  Now when you create a web test, you will see a new dialog.  The dialog will display all of the available plug-ins that can be executed.  Select your plug-in and hit ok.  Once you are done recording your web test, the plug-in will be executed.

Creating the Sample Plug-in

First a quick review of the correlation that we are going to automate.  Here is the screen shot from correlation tool after I recorded my web test against a reporting services site.

 

 

We are going to correlate the ReportSession parameter.

1)      Create a class library project

2)      Right click references and select  Add Reference

3)      Choose Microsoft.VisualStudio.QualityTools.WebTestFramework

4)       Here is the code for my plug-in:

using System.ComponentModel;

 

using Microsoft.VisualStudio.TestTools.WebTesting;

using Microsoft.VisualStudio.TestTools.WebTesting.Rules;

 

namespace RecorderPlugins

{

    [DisplayName("Correlate ReportSession")]

    [Description("Adds extraction rule for Report Session and binds this to querystring parameters that use ReportSession")]

    public class CorrelateSessionId : WebTestRecorderPlugin

    {

        public override void PostWebTestRecording(object sender, PostWebTestRecordingEventArgs e)

        {

            //first find the session id

            bool foundId = false;

            foreach (WebTestResultUnit unit in e.RecordedWebTestResult.Children)

            {

                WebTestResultPage page = unit as WebTestResultPage;

                if (page != null)

                {

                    if (!foundId)

                    {

                        int indexOfReportSession = page.RequestResult.Response.BodyString.IndexOf("ReportSession");

                        if (indexOfReportSession > -1)

                        {

                            //add an extraction rule to this request

                            // Get the corresponding request in the Declarative Web test

                            ExtractionRuleReference ruleReference = new ExtractionRuleReference();

 

                            ruleReference.Type = typeof(ExtractText);

                            ruleReference.ContextParameterName = "SessionId";

                            ruleReference.Properties.Add(new PluginOrRuleProperty("EndsWith", "&ControlID="));

                            ruleReference.Properties.Add(new PluginOrRuleProperty("HtmlDecode", "True"));

                            ruleReference.Properties.Add(new PluginOrRuleProperty("IgnoreCase", "True"));

                            ruleReference.Properties.Add(new PluginOrRuleProperty("Index", "0"));

                            ruleReference.Properties.Add(new PluginOrRuleProperty("Required", "True"));

                            ruleReference.Properties.Add(new PluginOrRuleProperty("StartsWith", "ReportSession="));

                            ruleReference.Properties.Add(new PluginOrRuleProperty("UseRegularExpression", "False"));

 

                            WebTestRequest requestInWebTest = e.RecordedWebTest.GetItem(page.DeclarativeWebTestItemId) as WebTestRequest;

                            if (requestInWebTest != null)

                            {

                                requestInWebTest.ExtractionRuleReferences.Add(ruleReference);

                                e.RecordedWebTestModified = true;

                            }

                            foundId = true;

 

                        }

                    }

                    else

                    {

                        //now update query string parameters

                        WebTestRequest requestInWebTest = e.RecordedWebTest.GetItem(page.DeclarativeWebTestItemId) as WebTestRequest;

                        if (requestInWebTest != null)

                        {

                            foreach (QueryStringParameter param in requestInWebTest.QueryStringParameters)

                            {

                                if (param.Name.Equals("ReportSession"))

                                {

                                    param.Value = "{{SessionId}}";

                                }

                            }

                        }

                    }

                }

            }

        }

    }

}

 

5)      Let’s review parts of the class.

a.       Iterate through the result to find first page with ReportSession.  This code fragment iterates through each of the recorded objects and searches the response body for ReportSession.

           foreach (WebTestResultUnit unit in e.RecordedWebTestResult.Children)

            {

                WebTestResultPage page = unit as WebTestResultPage;

                if (page != null)

                {

                    if (!foundId)

                    {

                        int indexOfReportSession = page.RequestResult.Response.BodyString.IndexOf("ReportSession");

                        if (indexOfReportSession > -1)

                        {

                        …

b.      Now that we found the response, we need to add an extraction rule.  This code creates the extraction rule and then finds the correct request in the web test to add the extraction rule to.  Each result object has a property called DeclaraticveWebTestItemId which is what we will use to get correct request from the web test.

           ExtractionRuleReference ruleReference = new ExtractionRuleReference(); 

           ruleReference.Type = typeof(ExtractText);

           ruleReference.ContextParameterName = "SessionId";

           ruleReference.Properties.Add(new PluginOrRuleProperty("EndsWith", "&ControlID="));

           ruleReference.Properties.Add(new PluginOrRuleProperty("HtmlDecode", "True"));

           ruleReference.Properties.Add(new PluginOrRuleProperty("IgnoreCase", "True"));

           ruleReference.Properties.Add(new PluginOrRuleProperty("Index", "0"));

           ruleReference.Properties.Add(new PluginOrRuleProperty("Required", "True"));

           ruleReference.Properties.Add(new PluginOrRuleProperty("StartsWith", "ReportSession="));

           ruleReference.Properties.Add(new PluginOrRuleProperty("UseRegularExpression", "False"));

 

           WebTestRequest requestInWebTest = e.RecordedWebTest.GetItem(page.DeclarativeWebTestItemId) as WebTestRequest;

           if (requestInWebTest != null)

           {

               requestInWebTest.ExtractionRuleReferences.Add(ruleReference);

               e.RecordedWebTestModified = true;

           }

c.       Now we need to find all query string parameters that have ReportSession as name and change the value to {{SessionId}}

           WebTestRequest requestInWebTest = e.RecordedWebTest.GetItem(page.DeclarativeWebTestItemId) as WebTestRequest;

           if (requestInWebTest != null)

           {

               foreach (QueryStringParameter param in requestInWebTest.QueryStringParameters)

               {

                    if (param.Name.Equals("ReportSession"))

                    {

                        param.Value = "{{SessionId}}";

                    }

                }

            }

 

6)      Now that we have our plug-in, I need to compile and deploy it to one of the locations listed above.

7)      Restart VS

8)      Open a test project and create a new web test.  I now see the following dialog with my plug-in available:

 

 

 

9)      Select the plug-in

10)   Record the same web test against my reporting services site and click stop to end the web test.

11)   Now when the correlation process runs, you will see that it does not find the ReportSession parameter.  This is because we have already correlated it.

 

 

 

12)   Now look at the first request in the web test and you will see the extraction rule.

 

 

 

13)   Now look at the other requests to see where we are referencing the extraction rule.

 

 

 

This is a slightly more advanced feature, but it provides a huge time savings for automating changes to your recorded web test.  If you have multiple people creating web tests, you can use this plug-in to make sure the same parameters or rules are added to each web test.  And of course you can automate correlation of parameters or URLs which the built in correlation tool does not find.

 

VSTS 2010 Feature: Enhancements for Web Test Playback UI

This is my second post about new VS 2010 web/load testing features.  You can get the beta here:  beta download

In this post I am going to talk about a number of usability enhancements that we made to the Web Test Playback UI and how you can use them.  The enhancements are

1)      Find – You can now do a search in playback UI which will search request headers/bodies, response headers/bodies and the web test context.

2)      Add Extraction Rule – In the playback UI, you can now highlight some text in the response and select “Add Extraction Rule”.  This will automatically add a rule the web test.

3)      Goto webtest – This option will jump you from the request you are on in the playback UI to the same request in the web test.  Also if you are on a query string or form post parameter, it will jump you to that parameter in the web test editor.

4)      Show Recorded Result – This option will load a playback session with what was recorded.  During recording, we now create a web test result of what was recorded.  This will help when you want to try and figure out what is different between your current playback session and the recorded session.

 

To demonstrate each of these features, I am going to walk through a demo that will show you how to correlate a dynamic value.  I will add an extraction rule to a request and then bind the value from the extraction rule to another request.  The demo I am going to walk through is the same one that I use in the following blog post:  Correlation Blog Post.  This will show you how much easier it is to do this process now.  I also want to point out that values we are going to correlate in this demo would actually be caught by the correlation process that runs when you stop recording.

Recording a Web Test

The web test I am going to record is a few requests against a SQL Server Reporting Services site.  At the end of my recording session, I see the following results from our correlation dialog.

 

 

So you can see that the parameters that were flagged as needing to be correlated.  Instead of having automatic correlation happen, I am going to choose no and manually correlate the parameters.

Playback the web test

Now I am going to playback the web test and then correlate the ReportSession parameter.  Here is the test after playing it back:

 

 

Now we will start the correlation process for ReportSession.  We will do the following:

1)      Find the first occurrence of ReportSession in a response body. 

2)      Add an extraction rule for that location. 

3)      Find other locations that ReportSession is used and bind them to the extraction rule.

 

Find Report Session

First we need to bring up the Find dialog.  Right click on a request and select Quick Find or hit ctrl-F.  This will launch the find dialog.  Here is what you will see:

 

 

Enter ReportSession in the Find What text box and select response bodies in the Look in combo box.  Then I hit find.  Here is what I see. 

 

 

As you can see it has found report session in the response body of the first request.  Now we need to add an extraction rule.

Add Extraction Rule

We want to add an extraction rule for the value of report session.  In this case the value is: p4agdi3gh5qkhh55ebtyox45.  So I am going to highlight value, right click and select  “Add Extraction Rule”.

 

 

When I do this, I am brought back to my web test and can see that an extraction rule was automatically added for this value.

 

 

I am going to change the Context Parameter name associated with this rule to ReportSession.  Now we need to find query string parameters that have ReportSession.

Go to Web Test

I am going to go back to the playback UI and use find again.  This time I am searching for ReportSession in a query string parameter.  I choose ReportSession as Find What text and search in the Request Headers.  You can see it found it in the 3^rd request. 

 

 

Now I am going to uncheck the Show raw data button so I can see the friendly view of the request parameters.  When I do this I can see ReportSession in the QueryStringParameter section.  I am going to choose this parameter, right click and select Go to Web Test.

 

 

When I do this, you can see that it took me right to this parameter in the web test editor.

 

 

Now I can change the value for this parameter to {{ReportSession}} so that it will pick up the value from the extraction rule.  I am going to rerun the correlation process with the updated web test, and this time it will not find the ReportSession parameter as one that needs to be correlated.

 

 

Go To Recorded Result

There is one other playback feature that I want to point out and it is go to recorded result.  As I mentioned above a result file will now be created while you are recording.  So while I am working on my web test and want to see what particular values were at record time, I can just pull up the recorded result.  To bring up the recorded result, go back to playback.  There is a new button on the toolbar from launching the recorded result:

 

 

When I click it, a new playback session will be opened with the recorded result.  The title of the tool window will have recorded in it.

 

 

 

I hope these new features will make it easier to work with the playback UI.  Please provide feedback when you have a chance to try it out. 

VSTS 2010 Feature: Load Test Excel Report Integration

Now that VSTS 2010 beta 1 is available I will be writing a series of blog posts highlighting new Web and Load testing features.  You can get the beta here:  beta download

The first feature that I want to review is creating load test run to run comparison reports with excel.  With this release we have built integration into excel that will allow you to choose a set of runs and a set of counters and then have the reports automatically generated.

Accessing the new report wizard

First, you need to be running excel on the machine that Visual Studio is installed on.  You will only be able to create the reports from that machine.  Once created, you will be able to share the excel workbook with other users, but they will not be able to modify the report. 

There are 2 ways to access the new report wizard:

1)      Launch Excel – This is easier way.  Just launch excel.  You will see a new ribbon called Load Test.  Click on the load test ribbon.

 

 

 

2)      From the load test analyzer – After running a load test and going into post run analysis UI, you will have a new button which will launch excel and then automatically bring up the wizard.

Generating Reports

Let’s walk through creating a report.

1)      Launch Excel

2)      Click on the Load Test ribbon

3)      Click the New Report button.  You will also be able to modify existing reports and we will go over that later.

4)      When click the button you will see the following:

 

 

5)      On the first page of the wizard, you need to enter the database that the load test results are being stored in.  For example, if my database was on a machine called Perf1 and I was using sql express, I would enter Perf1\sqlexpress.  Then click next.

6)      On the next page you have 3 options. 

a.       Create a brand new report.

b.      Modify an existing report.  This option will allow you to choose a report that you have previously created and regenerate it.  You will also be able to modify settings such as runs to display or counters to display.  The reason that we can do this is that the report definitions are stored in the load test results store.  When we get to the end of the wizard and click finish, the report definition including runs and counters will be stored.  That way if you lose your workbook and need to recreate the report, you can just pull it up and recreate it.

c.       Use an existing report as a template – This is useful if you have created a report that has all of the counters that you like to see.  You will see on the next screen that you need to choose a load test for the report.  Then you will be presented will all of the runs for that load test.  So if you have created a report for load test 1 and now want to use it for load test 2, you would choose this option.

Choose Create a Report and click Next.

 

 

7)      On this screen you need to enter a name for the report and choose a load test.

 

 

8)      The next screen will display all of the runs associated with the load test.  If you have added analysis notes to the run, they will be displayed.  Here is a blog post adding these notes.  They will become very useful when selecting runs:  Adding Analysis Notes.  Select the runs you want to include and then click next.

 

 

 

9)      The next screen will show you all the of the counters that are available to graph.  This will include all counters you collected for the runs that were selected.  Select a few counters and click Finish.

 

 

 

10)   Now the set of reports will be automatically generated.

 

Viewing the Reports

The reports which are generated will be a set of pivot tables and charts.

Here are the set of reports that will be created.

1)      Table of Contents – This will display the name as well as hyper links to each of the reports that were created.

 

 

2)      Runs – This worksheet will show you each of the runs that were selected:

 

 

3)      Reports for counters.  – Then you will have one worksheet for each counter selected.  The x axis is the run number.  So this will give you an idea of how the counter is trending from run to run.

 

 

 

Editing Existing Reports

After generating the report, you can modify a few things.  You can change the selected runs and the selected counters.

1)      Click the Load Test ribbon again.

2)      Click the Edit Runs button.  This will display the runs that are currently displayed.  You can remove runs or maybe you want to add runs which have completed since you generated the report.

 

 

3)      Click the Edit Counters button.  This will allow you to change the set of counters that are displayed.

 

 

 

Sharing Reports

As mentioned above, once you have generated the report, you can share the workbook.  You will only be able to modify the workbook on a machine that has VS installed on it.  Other users will not see the Load Test ribbon, but they will be able to view the workbook.

Hopefully this feature will help with creating reports that can be shared with the rest of your team.  Please provide feedback on this feature.

Using Fiddler to create web tests with transactions

Check out this blog post from Eric Mattingly about extenstions he added to fiddler for creating VSTS web tests with transactions: http://blogs.msdn.com/nexpert/archive/2009/05/04/creating-transactional-web-tests-for-visual-studio-with-fiddler-nexpert.aspx

Creating a custom step load profile

This blog post will show you how to create custom load profiles by walking you thorugh the process of creating one which will step the user load up and then back down: http://blogs.msdn.com/slumley/pages/creating-a-custom-load-profile-which-will-step-user-load-up-and-then-back-down.aspx

Load Test Plug-in Overview and Samples

This blog post gives an overview of the load test plug-in extensibility point and provides a few samples of what they can do: http://blogs.msdn.com/slumley/pages/load-test-plug-ins.aspx

Awesome new Web/Load testing reference guide available

Check out the following guide for answers to many different questions on web and load testing: http://vstt2008qrg.codeplex.com/

This covers the following topics:

SETUP CONSIDERATIONS

WEB TEST CONSIDERATIONS

WEB SERVICE TEST CONSIDERATIONS

UNIT TEST CONSIDERATIONS

LOAD TEST CONSIDERATIONS

LOAD TEST RIG CONSIDERATION

PERFORMANCE DATA COLLECTION AND USAGE

LOAD TEST RESULTS STORE INFORMATION

TEST CUSTOMIZATION

ITEMS CHANGED OR FIXED IN VSTS 2008 SP1

GENERAL COMMANDS AND TRICKS (NOT VSTS SPECIFIC)

How to run Web and Load Tests from the Command-Line

This blog post will show you how you can run web and load tests from the command-line: http://blogs.msdn.com/slumley/pages/running-web-and-load-tests-from-the-command-line.aspx

Modifying Request Properties with a Web Test Plugin

This blog post will show you how you can create a simple web test plugin which can be used to easily modify properties of each request in a web test: http://blogs.msdn.com/slumley/pages/setting-request-properties-with-a-web-test-plug-in.aspx

Scriptable Parameter Values in web tests

This post will show you how to imbed code in a parameter value of a web test without having to create a new plugin: http://blogs.msdn.com/slumley/pages/using-scriptable-parameter-values-in-a-web-test.aspx