Ed Glas's blog on VSTS load testing : Visual Studio 2008 SP1

VS 2008 SP1 Released

edglas — Tue, 12 Aug 2008 14:14:10 GMT

SP1 has shipped. As I blogged about before here and here, there is a lot of good stuff in SP1 for TeamTest. The English downloads are available here. Note, if you are using a different language VS, the patch is available in all languages too.

I have to note, we found a serious regression in load tests last week that was introduced in SP1, but it was too late to fix it in SP1. The regression is that think times are not applied to requests that redirect.

If you are doing web load testing, this problem will almost certainly impact the pace at which your vUsers run. Bill has documented a web test plugin workaround for this problem, and we will be issuing a QFE for the fix.

Ed.

Web test support for random text extraction in SP1

edglas — Sun, 27 Jul 2008 20:11:45 GMT

In SP1 we added a new Random property on the Extract Text rule. This simple property enables some really powerful scenarios.

The best example is to randomly select a link that is the result of a query. This works nicely if search result urls are uniform and pass the selection as a query string parameter. For example, all search result links are of the form

http://mysite/productdetails.aspx?productId=42

In the source page, this might look like:

<A href="http://mysite/productdetails.aspx?productId=42">Telescope</A>

<A href="http://mysite/productdetails.aspx?productId=43">Microscope</A>

What you can do with the new rules is randomly select one of the products on the search results page by extracting matches that start with productId= and ends with " to extract the 42 or 43, then set the random property to true to randomly select one of these products.

Another neat thing you can do is re-run the correlation tool after adding the extraction rule (Detect Dynamic Parameters command on the web test toolbar). Many times the value extracted (e.g. 42) is reused in other places in your script. The correlation tool will detect that, and make it easy to add the extraction rules and databinding to the right places in your test.

Ed.

More on 2008 SP1

edglas — Wed, 25 Jun 2008 04:34:34 GMT

Bill Barnett has posted a couple of great articles detailing the new APIs we've added in SP1. Check these out to see the powerful extensibility points for controlling load patterns in a load test, and more/better hooks in web tests that enable some key scenarios such as easily filtering some types of dependents and easily capturing page load times via the API.

Since the beta, we added support for IE8. The interesting thing about IE8 is that by default it opens 6 connections to the web server, rather than 2 which is used by previous browsers. FireFox 3 also uses 6 concurrent connections. While this makes playback of a single test faster, it doesn't actually increase throughput in a load test.

As I said before, and Bill details on his blog, the biggest thing to be aware of with SP1 if you are running load tests on an agent is that agent, controller, and VS clients must all be upgraded at the same time. So you can't run SP1 VS against an RTM controller and agents (and visa-versa).

We've also fixed a few additional bugs since the beta (all of these were in the RTM version of 2008, none were regressions in the beta):

Priority	Team Level 1	Title
1	Test Infrastructure Team	VSTS failed to install on top of VS Pro + SP1
1	Load Test Team	Web tests do not accurately emulate and measure the page response time using IE8
2	Load Test Team	Multi-agent -total user load and TestRunning are not calculated correctly
2	Test Infrastructure Team	Dialog clipping - the "Disable background discovery of test methods" string clips on localized versions.
2	Test Infrastructure Team	When a test file (test storage) has dependent files, we do not enumerate tests from it
2	Load Test Team	WebTest: No way for Web test code to get the ContentLength of a request
1	Load Test Team	Recorder: Low level recorder isn't working for multiline textboxes
2	Load Test Team	Web test hidden field correlation feature doesn't work for Ajax update panels
1	Load Test Team	Web test execution: file uploads are posted twice when uploading to sites that require NTLM authentication
1	Unit Test Team	Unit Test inheritance: user experience running unit tests with inheritance from command line is broken (mstest reports an error)

Short of finding a catastrophic error and clearing up one last problem we are seeing with applying the patch, we are now done with SP1 and making great headway on our next release.

Ed.

VS 2008 SP1 Beta is available

edglas — Wed, 28 May 2008 00:39:16 GMT

Incase you haven't seen this, the beta for Visual Studio 2008 SP1 is now available.

We've put a lot of work into this SP1 release, mostly around bug fixes for bugs that you reported on the forums and through tech support (mostly forums), as well as bugs our test and dev teams reported. We've also managed to slip a few extra features into the release.

The download is here.

I highly recommend you install the beta and report back any issues ASAP so we can be sure to get them rolled into the final release.

One issue to be aware of is if you are running using test agents, the agent, controller, and all clients must match the SP1 version. Unfortunately the controller and agent uses .NET remoting object serialization, which makes it very sensitive to any changes in the objects that are being passed back and forth.

New features (we'll be blogging about these in separate posts):

Priority	Team Level 1	Title
1	Load Test Team	Support random extraction of matches using extract text rule
2	Load Test Team	DCR: Add Web test and load test extensibility points to support partners
1	Test Infrastructure Team	Enable Dual NIC controller scenarios

Bugs fixed (I’ve highlighted ones you may find interesting):

Priority	Team Level 1	Title
1	Load Test Team	VS crashes due to InvalidOperationException in load test monitor when running load test that hits many unique URLs
1	Load Test Team	Extra threads created when running load test containing unit tests on multi-agent rig causing out of memory exception
1	Load Test Team	Loadtest DB: Installing the controller should not drop an existing database.
2	Load Test Team	Goal-based profile - user load adjustment is based on 'ideal' user load instead of actual user load
2	Load Test Team	Cookies: We're not handling cookie expirations properly.
2	Load Test Team	LoadTest: When OutOfMemory exception stops load test on one agent in multi-agent test some load test results data is lost.
2	Load Test Team	Incorrect requests being recorded for Ajax site
2	Load Test Team	Form fields can be recorded twice
2	Load Test Team	Loadtest: Test times are inaccurate at less than 15 milliseconds.
2	Load Test Team	DTS: Web Recorder does not record request when Content type is missing
2	Load Test Team	DTS: Webtest Recorded in 2005 will not record in 2008
2	Load Test Team	WebTest.WebProxy property does not work
2	Load Test Team	Hidden Field Name appears as GUID when uploading file to sharepoint
2	Load Test Team	Custom Load Profiles: Having a MinUserCount > MaxUserCount throws the wrong exception.
2	Load Test Team	Performance overhead of new PostPage event is too high
2	Load Test Team	Webtest: Think times set on the first request are ignored.
2	Load Test Team	Orcas Web Test: View response in html editor does not work
2	Load Test Team	LoadTestPlugin can't set the user load higher than the max user load for the scenario
2	Load Test Team	Test View has huge delays
2	Load Test Team	Visual Studio crashes when calling AddCommentToResult in a WebTestRequestPlugin
2	Load Test Team	Web Browser tab in Web Test result shows broken page on JPN OS.
2	Load Test Team	Can't set Outcome on web test or request
2	Load Test Team	Copy/Paste from the webtest playback window on virtual PC crashes orcas.
2	Load Test Team	The default for the load test run setting "Run unit tests in application domain" should be true
2	Load Test Team	File upload parameters with no actual file causes the web test engine to throw an exception.
2	Load Test Team	Wininet recorder: ID field not correlated properly when uploading files to sharepoint.
2	Load Test Team	Copying a Load profile from the UI yields a cast error.
2	Load Test Team	Load test user load ramps up to quickly during warmup period
2	Load Test Team	Load Test should honor the ApartmentState setting in the Test Run Config file
2	Load Test Team	Thinktime that's set via plugin code is ignored
2	Load Test Team	LoadTest aborts when test throws an exception with a quote in the stack trace
2	Load Test Team	Loadtest Analyzer: Error when trying to bring up the loadtest analyzer after a test that stops very quickly.
2	Load Test Team	LoadTest: Goal profile performance counter should be sampled at least every 5 seconds
2	Load Test Team	WebTest: User code can't set values in the WebTestContext if the property has been set in an env variable on in the LoadTestContext
2	Load Test Team	Network emulation doesn't work correctly with compression
1	Test Infrastructure Team	Loc: String "Whidbey" appears in Microsoft.VisualStudio.QualityTools.Resource.dll
1	Test Infrastructure Team	VSPro 2008 won't show test lists created on VSTS2008
1	Test Infrastructure Team	Test runs with load test and any other test type get stuck
1	Test Infrastructure Team	CodeGen for IIS website code failed in generating the private accessor
1	Test Infrastructure Team	Upgrade a Whidbey test project in VSTD2008 will fail
1	Test Infrastructure Team	SP1 patch setup fails on VSTS with controller/agent installed
1	Test Infrastructure Team	Fix perf tests that are failing from time to time in QFE (TechEase ticket 143089)
1	Test Infrastructure Team	Publishing load tests fails in TFS SP1
2	Test Infrastructure Team	Multi-agent: Could not start two load tests in paralell on different agents
2	Test Infrastructure Team	Watson - unhandled TeamFoundationServerUnauthorizedException crashed devenv.exe
2	Test Infrastructure Team	Out of memory error when publishing test cases using MSTest
2	Test Infrastructure Team	MSTest can't run codedwebtests when alone errors with assembly is not trusted
1	Unit Test Team	Invalid AppDomain.CurrentDomain.BaseDirectory in VS 2008 Unit Test Runner
1	Unit Test Team	Unit Testing: App Domain naming creates issues with performance counters
1	Unit Test Team	Unit Test Generation fails for default WCF service
2	Unit Test Team	Ordered tests created in VS2005 are not automatically upgraded in VSTD 2008
2	Unit Test Team	DTS: IDE in 2008 responds slow compared to IDE in 2005 in a test Project with over 9000 unit test
2	Unit Test Team	VSTestHost specificies LoaderOptimization.MultiDomain as the default, causes various problems
2	Unit Test Team	Unit testing fails in asp.net hosted in iis

I hope you find this useful, and by all means please download and use SP1 beta!

Ed.