Ed Glas's blog on VSTS load testing : Web Test

New Web and Load Test Features in Beta2

edglas — Wed, 21 Oct 2009 20:42:00 GMT

In beta2 we added a number of new minor features, and a couple of major ones.

We also changed our licensing model, which you can read about here.

Here is a list of the minor features we added, along with a brief description of each. The last one in the list, compare two runs in Excel, is probably my most favorite feature in the entire release.

Area

Title

Description

Recorder

Recorder options in tools options

Starting URL supplies a default url to start recording from.

Two new options to control whether or not the recorder does hidden field matching, and whether or not it runs the dynamic parameter detection tool after recording completes.

Recorder

Recorder records all requests by default, fold dependents under page

In 2008, static dependents not found by the dependents parser were not recorded by default. Now they are.

Web test

search/replace

Search and replace in web test requests!

Playback

Add request bytes column

Request byte count is now shown in request list pane in playback (in addition to response byte count)

Playback

PreRequestDataBinding event

This new event fires before data binding occurs, allowing a plugin to dynamically change data bindings in a request.

Data binding

64 bit csv

This fixes a limitation we had in beta1, and now enables csv data sources on 64 bit machines.

Data binding

New APIs on WebTest to control data cursor position and reload data

Give full control over the cursor, enabling tight control of which user is using what data. Also reload enables refreshing a datasource during the test.

Load Test

Scrub counter sets

We refreshed the set of counters being collected, thresholds, and default counters.

Analyzer

Add standard deviation and 99^th percentile stats to load test analyzer

Analyzer

Graph instance 0 for CPU on local runs by default

Fixes the problem documented in this post.

Analyzer

Specify range for a counter (CPU range is 100)

Now you can specify a range for a given counter. Also percentage counters get a range of 100 by default.

Analyzer

Fixed range on perf graph

The performance graph now gets a default range set for it (range is rounded up response time of the slowest measurement)

Analyzer

Computer tags

Computer tags enable you to classify computers by function (e.g. web server, db server, etc.). Also enables run to run comparison even if the runs are on different physical machines.

Reporting

Compare two runs in Excel

This is a new, rich reporting option that enables you to compare performance and resource utilization data in two different runs.

Ed.

New Performance Testing Training Available from RTTS

edglas — Fri, 22 May 2009 13:39:00 GMT

RTTS, a premier testing consultancy and test tools training company, now has training available for performance and load testing with VSTS 2008. RTTS specializes in providing software quality services, and offers training on a variety of tools. It is great to see VSTS in the mix as they see demand for VSTS growing. RTTS offers VSTS performance testing training onsite, over the web, or at their site, as well as consulting services. More details available here.

Ed.

dynaTrace Leverages new Extensibility Points to Provide Deep Integration with Dev10

edglas — Wed, 20 May 2009 20:25:30 GMT

A fantastic example of a partner taking advantage of the new extensibility points in Web test playback and load tests in dev10 is provided by dynaTrace. dynaTrace has developed an extension that provides enables you to drill from a particular request to the PurePath for that request. The PurePath provides a tiered performance breakdown for the request, detailing how much time the request spent at each tier of your application. Check out Andreas’s post on the integration (including some code samples!).

Ed.

Dev10 Beta 1 Available!

edglas — Mon, 18 May 2009 18:15:00 GMT

Beta 1 was released to the web today for MSDN subscribers. If you are a MSDN subscriber, you can download the Beta today from here.

We have been very busy on this release of late (one reason why my blog posts have slowed down lately :))

My team and I will be blogging about all the great new features in dev10 beta 1 over the coming weeks.

As in the past, I will compose a list of links to any posts on the new features and update this blog post with the links as we get them.

New Web Test Features in Dev10:

These features are about making it easier to get web test playback working:

Extensible recorder plugins for modifying recorded web tests (including adding custom dynamic parameter correlation)
Find in playback
View recording log in playback
Jump to request from playback
Add extraction rule from playback
WebTestResult in a separate file for better scalability
New API for web test result
New extensibility API in playback enables UI extensions to playback

File Upload record/playback just works

Uploaded file added to project and deployed
Option for generating a unique file name on each iteration

Looping and branching in web test
New dialog for editing think times, goals, reporting string across all requests
New validation and extraction rules for inner text and select tags
Test level validation rule for pages meeting response time goal

New features in Load Test in Dev10:

Logging and data collection improvements

Log entire test result on test failure or log entire result every n requests
Remote profiler integration
Event Log Data Collection
Extensible data collectors to support more platforms

Analysis Improvements

Load test virtual user activity visualization, also here.

Reporting

Run to run perf reports in Microsoft Excel

Performance testing

Run automated UI tests as performance tests
True network emulation

More flexible load modeling

New Sequential Test Mix
Delay start on scenario
Iteration count on scenario
Select which agent(s) to run the scenario on
Disable during warmup

Multiple load test plugins
64 bit test execution

In addition to continuing our testing and bug fixing efforts, right now we’re working on an additional set of features for beta 2. I’ll have to sit on those until beta2. :)

Ed.

New, comprehensive technical documentation available for Web, Load, and Unit Tests

edglas — Tue, 31 Mar 2009 14:50:00 GMT

The VSTS Ranger team, led by Geoff Gray, just published the VSTT Quick Reference Guide 1.0.

This is a comprehensive collection of technical information on VSTT –- 83 pages of information!!! Many questions you may have on how stuff works in web, load, and unit test are answered in this doc. This doc is a must have for anyone working in VSTT.

Geoff and his team, the Microsoft Services Test Labs, deliver professional services for doing performance testing. They are experts at the tools, and work closely with my team (the product team) both to troubleshoot issues as they come up and to help steer the product in the right direction. Whenever Geoff saw an answer in email, figured it out himself, or saw it on a blog post, he put it in the document. He has been building it up over the past few years. I highly recommend you refer to it any time you are looking for more information or are stuck trying to get something to work in the product.

Here’s an extract from the document:

Congrats Geoff on getting this published! And thanks to others who helped along the way, and to Bijan Javidi for your support in sponsoring this project to get it published.

Ed.

VSTS wins Testers Choice in for .NET Apps in Software Test and Performance Magazine

edglas — Wed, 04 Feb 2009 17:13:00 GMT

The annual testers choice awards are out in STP magazine, and VSTS took top spot in two categories in this years tester's choice poll, winning out over load runner for ".NET Performance Testing" and for "Integrated Test/Performance Suite".

Congrats to the team who worked so hard on VS 2008 product, it's great to see our customers like the tool.

I'm dissapointed with the product write up in the magazine, it doesn't mention our load testing solution at all. Edward Correia, we need to talk! :) For a good overview of our performance testing solution, see Amit Chatterjee's recent blog post.

VS 2008 is a fine release, but there is still plenty of work to do! I expect our product line will continue to rise up in other categories as well with our next release. We have a fantastic set of new products for general testing on the runway, plus a ton of new performance testing features in Visual Studio Team Test. It's good to see TFS get the second place vote for defect tracking, I'm sure that as we expand our product line to appeal to all testers we will become the top choice in this and the test management category as well.

Ed.

Back from PDC

edglas — Tue, 04 Nov 2008 01:14:01 GMT

I spent last week in LA at the PDC. The conference was mainly focused on the new technologies Microsoft is working on bringing to market, such as Windows 7 and Azure.

We also introduced the features in our next VSTS release, which customers were super-excited to see. It's always great to step back at a time like this to take account of what we've been building since shipping the 2008 release. The demos were all using the CTP we just shipped.

I did a perf talk with Steve Carroll that you can watch here.

Some of the features I showed or talked about:

New run to run trend reports in Excel
Data collectors used to collect profiler data from the web server
Using UI tests as performance tests in a "load" test
New details view for visualizing virtual user actions and performance patterns

Cameron Skinner in his lap around VSTS 2010 also showed some of the new test case management features we are working on, you can watch that talk here.

There are more features in the CTP:

detailed logging in a load test
looping and branching in web tests
editing think times, response goals, and reporting name in web tests

These CTP bits were actually cut in July, so we've added a bunch of new features since then. I can't blog on those features yet, but they are also exciting! At this point we're really turning our attention to shutting down feature work to focus on shipping beta.

Ed.

Event ordering in Web Tests

edglas — Thu, 28 Aug 2008 18:49:49 GMT

Web tests have a number of extensibility points before and after a request. What order do they run in?

The hooks are:

WebTestPlugin.PreRequest

WebTestRequestPlugin.PreRequest

ExtractionRule.Extract

ValidationRule.Validate

Here is a sequence diagram that shows what order things fire:

There is no guaranteed order of firing for WebTestPlugin.PreRequest and RequestPlugin.PreRequest. Also if you have >1 Extraction or Validation rule on a request there is no guaranteed order they are run in.

Also note that Pre and Post request events do not fire by default for dependents. You can hook the PostRequest call on a top level request and iterate through dependents to wire up event handlers if you need that functionality.

Ed.

New Fiddler Timeline view is neat!

edglas — Tue, 19 Aug 2008 15:58:00 GMT

Eric Lawrence has added a cool new feature to fiddler to visualize what is happening in your http traffic. Here's a snippet from the online fiddler help:

Timeline

The Timeline Tab shows the transfer timeline of selected HTTP Sessions. NOTE: You should enable Streaming Mode for the highest accuracy in Timeline rendering.
The vertical line indicates the time to first byte of the server's response.
The green circle before the bar shows that a connection was reused; a red circle means that the connection was newly created.
The green circle after the bar shows that the server didn't send a Connection: close header; a red circle means that the server sent Connection: close.
The gray arrow icon indicates that the server's response was a redirect (302). The red ! icon indicates that the server returned an error code (404).

How cool is that?!

Ed.

Updated load test links

edglas — Tue, 19 Aug 2008 11:19:00 GMT

I've updated my links to info on web and load test post with the following links:

New Item Templates for custom rule and plugin development

Data Driven Web Testing With Visual Studio 2008 Team System

InfoPath Forms Services 2007 Web Testing Toolkit

SharePoint Server 2007 Scalability and Performance Whitepaper

Diagnosing OutOfMemoryExceptions that occur when running load tests

Ed.

New templates for plugins and rules

edglas — Tue, 19 Aug 2008 10:30:35 GMT

Inspired by templates developed by Dennis Bass of the services test labs, I added new item templates to the codeplex.com\TeamTestPlugins project.

If you install these on your machine it makes creating new plugins much easier:

If you work with web and load tests you'll find these very handy, and they are xcopy-installed on your machine (you'll have them working in less than 5 minutes).

Now to get these in our next release. :)

Ed.

Masking a 404 error in a dependent request

edglas — Thu, 07 Aug 2008 00:01:02 GMT

There are times when your web test may fail with a 404 error in a dependent request (like a css file or gif), but you may not actually care about this error. For example, it may be something that doesn't really impact the layout or content of the page. Or you might file a bug and temporarily want to suppress the error until it is fixed so they can find other errors.

If you get a 404 error in a dependent resource, here's how you can mask it.

Here's a failing web test, it is very simple. If I expand the top request I can see it is failing due to a failed dependent request.

To mask the request, right-click on the dependent and hit Copy.

Go to the test, right click on the corresponding request in the test and select "Add Dependent Request". Paste the url on the clipboard into the url property for the dependent request:

Now set the expected http status code on the dependent to 404.

Rerun the test to see it pass.

Hope that helps,

Ed.

Web test support for random text extraction in SP1

edglas — Sun, 27 Jul 2008 20:11:45 GMT

In SP1 we added a new Random property on the Extract Text rule. This simple property enables some really powerful scenarios.

The best example is to randomly select a link that is the result of a query. This works nicely if search result urls are uniform and pass the selection as a query string parameter. For example, all search result links are of the form

http://mysite/productdetails.aspx?productId=42

In the source page, this might look like:

<A href="http://mysite/productdetails.aspx?productId=42">Telescope</A>

<A href="http://mysite/productdetails.aspx?productId=43">Microscope</A>

What you can do with the new rules is randomly select one of the products on the search results page by extracting matches that start with productId= and ends with " to extract the 42 or 43, then set the random property to true to randomly select one of these products.

Another neat thing you can do is re-run the correlation tool after adding the extraction rule (Detect Dynamic Parameters command on the web test toolbar). Many times the value extracted (e.g. 42) is reused in other places in your script. The correlation tool will detect that, and make it easy to add the extraction rules and databinding to the right places in your test.

Ed.

Web, Load and Unit Testing Videos on MSDN

edglas — Fri, 25 Jul 2008 14:19:51 GMT

Check out these videos on MSDN. Here's the landing page for VSTS videos.

Here are the testing related videos:

How Do I: Create a Data-Bound Web Test?

How Do I: Use .NET Code to Customize Coded Web Tests?

How Do I: Use the Visual Studio Team System Test Load Agent?

How Do I: Manage Test Results and Test Results Files?

How Do I: Data-Bind a Unit Test?

How Do I: Easily Bind Data to .NET Unit Tests?

- Note this video does not talk about managing load test results.

DB Pro (Visual Studio Team System for Database Professionals) test videos:

How Do I: Generate Test Data using Visual Studio Team System Database Edition?

How Do I: Use the Checksum and ExpectedSchema Database Test Conditions?

Ed.

New rules published to codeplex

edglas — Mon, 21 Jul 2008 14:38:01 GMT

I've published a new project to http://codeplex.com/teamtestplugins. This project is based on VS 2008.

A summer intern we have from Duke, Herodotos Herodotou, has developed a nice set of rules we did not have before.

All of the rules have to do with the inner text of a tag. For example, for a select tag (list box and combo box), the option text is stored in inner text rather than an attribute:

<select name="myselect1">
<option>Milk </option>
<option>Coffee</option>
<option selected="selected">Tea</option>
</select>

In order to extract the value of the list box, we need to parse out the inner text of the selected option. TextArea is another tag that does this, but there's also a lot of other examples in HTML why you want to extract or validate inner text.

The new project has these new rules:

ExtractionRuleInnerText
ExtractionRuleSelectTag
ValidationRuleInnerText
ValidationRuleSelectTag

As well as an inner text and select tag parser.

I've also added the web test plugin for filtering dependents, below.

Ed.