Ed Glas's blog on VSTS load testing : Visual Studio Team Test

New Performance Testing Training Available from RTTS

edglas — Fri, 22 May 2009 13:39:00 GMT

RTTS, a premier testing consultancy and test tools training company, now has training available for performance and load testing with VSTS 2008. RTTS specializes in providing software quality services, and offers training on a variety of tools. It is great to see VSTS in the mix as they see demand for VSTS growing. RTTS offers VSTS performance testing training onsite, over the web, or at their site, as well as consulting services. More details available here.

Ed.

Are You Reporting Inaccurate Performance Numbers with VS 2008?

edglas — Wed, 20 May 2009 00:04:00 GMT

There’s a hidden problem with VS 2008 load testing that you need be aware if you are running tests locally (as opposed to using a controller and agents).

If you purchase and run a load agent, load agents run on all cores and all CPUs on the machine.

However, when running locally VS limits load generation to one core on the machine. If this core is running at or close to 100% CPU utilization, it will throw off performance measurements and interfere with performance counter collection. In general it really invalidates your test results. This is why we graph agent CPU by default, and have performance counter thresholds that fire to warn you if the test is running too hot. However, by default the test is configured to look at the _Total instance, when it should be looking at the “0” instance.

This screen shot really tells the story:

CPU 0 is clearly pegged:

Yet the CPU counter in the load test is reporting ~50% utilization:

‘

What is going on here is that the counter reported on is the _Total counter.

To get an accurate measurement we should graph the instance 0 counter instead.

To change your tests to graph the instance 0 counter, change the counter instance in the load test editor:

Now if I re-run, alarm bells should go off as thresholds violations. But they don’t!

The reason is the analyzer is still displaying the _Total instance by default. However, you can see the threshold violation is now firing:

To graph instance 0, drill into the counter tree and drag it on the graph:

Now you can see the threshold violations displayed on the graph. To graph the "0" instance by default, edit the .loadtest file in the xml editor and change the instance:

You can fix this for all subsequent load tests you create by editing the <installdir>\Common7\IDE\Templates\LoadTest\CounterSets\AgentCounterSet.CounterSet file and adding a new instance tag for the “0” instance:

<CounterCategory Name="Processor">
<Counters>
    <Counter Name="% Processor Time">
      <ThresholdRules>
        <ThresholdRule Classname="Microsoft.VisualStudio.TestTools.WebStress.Rules.ThresholdRuleCompareConstant, Microsoft.VisualStudio.QualityTools.LoadTest">
          <RuleParameters>
            <RuleParameter Name="AlertIfOver" Value="True" />
            <RuleParameter Name="WarningThreshold" Value="90" />
            <RuleParameter Name="CriticalThreshold" Value="95" />
          </RuleParameters>
        </ThresholdRule>
      </ThresholdRules>
    </Counter>
    <Counter Name="% Privileged Time" />
    <Counter Name="% User Time" />
</Counters>
<Instances>
    <Instance Name="_Total" />
    <Instance Name="0" />
</Instances>
</CounterCategory>

Also change the settings for the default counters to show on the graph:

Hope that helps, and we certainly will get this fixed for dev10 (but the fix did not make it into beta1).

Ed.

Elevating the Role of the Tester with Visual Studio 2010

edglas — Tue, 19 May 2009 19:55:00 GMT

I am very excited and proud to introduce the new testing features you’ll find in Visual Studio 2010. In addition to continuing to work on the performance testing tools, I have also been focusing on delivering a new set of products for the entire test team. Since most testing is done manually, we have really focused on the manual testers with this release. If you do testing or test management you will love the new features. We also are introducing a new solution for UI automation as well as improvements in web, load, and unit testing, and our testing framework.

We have been working on this since before we shipped VS 2008, it is great to (finally) get the beta out. When we set out to define this release, we set forth a set of what we call value props that capture the value in the release. Using these as the frame, we built up the architecture and feature set to match.

In this post, I’ll go over the over-arching themes for the test tools in dev10. Our GM Amit Chatterjee has a terrific set of posts on each of these feature areas on his blog, be sure to check those out (I’ve linked to them throughout this post). Also be sure to check out the beta version of our documentation, which takes a refreshing approach to product documentation.

Introduction to Team Test for Dev10

Visual Studio Team Test (VSTT10) for Dev10 introduces key new functionality for test management, manual testing, test lab management, and automated testing. Specifically,

VSTT10 introduces an entirely new tester-focused UI that will free Manual Testers from what they see as the developer-centric Visual Studio UI. This new UI (code named Camano) will be the console for test case authoring, management, execution and tracking.
VSTT10 takes virtual test lab management to new heights by allowing virtual lab creation, configuration and deployment all integrated into the testing experience.
VSTT10 significantly increases our investment in the Specialist Tester role by improving the capabilities and performance of load and web testing as well as introducing new UI automation capabilities through the Visual Studio IDE.

VSTT10 introduces substantial new value propositions for software testers:

Make manual testers more effective by integrating test activity into the ALM and providing tools for managing, developing, recommending and executing tests.
Enable manual testers to reliably and consistently create actionable bug reports via automatic data gathering, reducing debugging time and eliminating non-reproducible bugs.
Ensure that both requirements and source code are adequately and transparently tested, ultimately answering the question "can we release?"
Enable efficient collaboration between QA and the rest of the extended development team throughout all phases of the application lifecycle.
Foster a collaborative tester community with an extensible platform that makes it easy to create and share new features and components that further enhance a tester’s productivity and effectiveness.

The core VSTT10 areas of test management, manual testing, lab management, and automated testing cut across each of these value propositions and create lasting value for test and development teams.

Getting the Basics Right

VSTT10 lays the foundation to help you the tester be more effective at the essential aspects of your job. The idea is to “get the basics right” and create a toolset that appeals to all test teams as well as the extended development team. You can watch an overview video here.

An Emphasis on Test Management and Manual Testing

In talking to you, our customers, we found that many of you track your testing effort using ad-hoc means and informal tools like Excel or Word. Most testing is performed manually by what we are calling generalist testers. To this end, the main focus for VSTT10 is to offer solid test case management tooling as well as a differentiating experience for manual testing. Because we are targeting generalist testers and test managers, we chose to develop a new client outside of the Visual Studio shell. We really wanted to provide a simple, focused experience for testers, and found we could not effectively do that with VS. We think you are going to really like the new client.

TFS-based Test Planning and Authoring

VSTT10 integrates deeply with TFS to offer rich functionality for test planning and authoring. Test planning features enable the test manager to define which test cases to run as part of a specific test plan or testing effort. VSTT10 also incorporates test case authoring, including the ability to define test steps, attachments and to factor and re-use parts of tests into shared steps. Test cases also allow the manual tester to associate data with the test case making it easy to step through different rows of data within the same test case.

A tester adds structure to their test plans by organizing test cases into hierarchical test suites. Query-based suites are used to automatically include any test case that meets specific criteria.

Test configurations allow the tester to define valid configurations that will be covered in a test plan, and then assign which tests will run on which configurations. Again the recurring theme is to better organize a tester to perform more rigorous testing, and track which configurations have or have not been tested.

Test cases are assigned to different testers on a project and tracked by a test manager to allow for better team organization and management of the test effort during the testing phase.

Linking Test Cases to Requirements and Bugs

Test cases are easily linked to requirements, which enables test planning, test execution, and traceability. Linking requirements to test cases also enables rich reporting, such as the count of test cases per requirement and pass/fail/not run testing status per requirement. A tester can easily add a requirements-based suite to a test plan, which automatically adds any test case linked to the requirement to the test plan.

When a new bug is filed while testing, the bug is automatically linked to the test case that generated it. This enables the developer to quickly open the test case that generated the bug, as well as interesting reports on which test cases generate the most bugs.

Documentation links: Managing New Testing Efforts with Team Test, Defining your testing effort using test plans

A Great Manual Testing Experience

The VSTT10 client offers a differentiating and highly productive manual testing experience. The capabilities around bug capture alone are not to be found in any competitors offering and will substantially increase productivity around bug finding, reporting and triaging. . In addition, the tester can easily capture screen shots, mark tests or steps as pass/fail and manage their bug finding techniques at a level not possible in other tools.

A key differentiator for manual testing is in fast forward manual testing. This innovative feature enables a tester to automatically record during testing, and then replay these actions the next time that same test is run. Automation can also be associated with shared steps, enabling testers to fast-forward through common steps that are repeated across test cases. This will allow testers to quickly fill in form fields and other types of repetitive data entry activity by replaying the form fill actions recorded in previous tests. The idea is to increase productivity by focusing automation on repetitive tasks and freeing a tester to concentrate on finding bugs and gaining better test coverage.

Documentation links:

Creating manual test cases, Running Manual Tests with Test Runner, Recording and Playing Back Manual Tests

Are We Ready to Ship?

A key goal for test managers is answering the question “are we ready to ship?” Arguably, answering this question is the primary mission of a test team, and doing so has many facets. The test team must shine a light in dark places of the application under test in order to find problems before customers do. This becomes as much an exercise in understanding what hasn’t been tested as what has. To that end, the VSTT10 will collect and report data to answer questions like:

Do we have adequate test coverage of our requirements and features?
Have we run all our tests?
Have we tested all of our target configurations adequately?
Have we tested the feature areas adequately?
Have we tested the changes?
What are we missing in our regression suite?
Are we on track to complete our testing?

To answer these questions, VSTT10 makes it easy to:

Define and organize test cases.
Define and manage which test cases are to be executed in a test effort.
Link test cases to requirements and features, and associate them with areas of the application.
Manage which configurations test cases are run on.
Seamlessly track which tests have been authored and run, and for those that have been run track pass/fail.
View which tests have been impacted by code changes.
View reports which surface this data to enable the test manager to answer the questions above.

VSTT10 achieves this by allowing the tester to define relationships of test cases to requirements and bugs in TFS, and to define the set of test cases and configs to run as part of a test plan. VSTT10 then seamlessly captures test results, ultimately storing this data in the TFS warehouse.

A basic set of reports are available through the process template and hosted on the SharePoint portal.Data is surfaced through reports on top of TFS. Since the data is in the TFS warehouse, reporting services or Excel reporting can be used to flexibly mine the data. A very rich set of data is available and can be related to metrics and measures currently captured by a product team. The warehouse stores test results over the life of a project, and those results can be sliced by build, area path, suite, and configuration.

This allows a test manager and stakeholders to generate reports such as the set of tests run for a given plan, the set of tests not run a given plan, and for those that were run what the results were. Historical results are available in the warehouse, providing stake holders rich visualizations of testing activity and quality trends.

Because test data can be related to requirements, a test manager can generate reports that show pass/fail per requirement, the number of bugs per requirement, test results per area, per suite, per build, and so forth. In all a very rich set of information is available in VSTT10. Check out Amit’s blog post on reporting.

Collaboration with Dev

VSTT10 provides tight integration between developers and testers through the free flow of information between them. Not only can developers and testers share the information stored in TFS, but information required by developers from ongoing testing is gathered automatically as described below. Also for the first time developers can get visibility into which areas of the product have and haven’t been tested, and better understand where their features are at.

Actionable Bug Capture

The basic flow of information from testers to developers is primarily through bugs. The effort wasted chasing down bugs and reproducing errant behavior is a severe pain point for developers and testers alike. “It works on my box” is the norm. VSTT10 takes this on head-on through information rich bug capture.

In order to enable this rich flow of information, VSTT10 delivers an innovative technology called data collectors. Data collectors enable testers to collect information from client and server machines in the test lab while they are testing. Data collectors enable a wide range of scenarios, including rich bug capture.

As test environments can be very complicated, and not all testers are experts at configuring the server system, an important aspect of data collectors is that they “just work” out of the box and are easy to configure and use.

A case in point is an essential new technology for bug capture known as Historical Debugging, which enables developers to debug exactly what was going on on a machine at the time a bug occurred.

As useful as historical debugging is to developers, the Action Recorder is a technology useful for developers and testers alike. It is based on VSTT10’s UI automation recorder, and captures user actions the tester takes and records that information as part of the bug report. The Screen Capture tool enables the tester to easily take screen captures, and the Video Recorder records the testers desktop and links the video to the bug.

Documentation links: Setting Up Machines and Diagnostic Information to be Collected Using Test Settings, Creating A Diagnostic Data Adapter to Collect Custom Data or Impact a Test System

The Essential Role of Builds

As bugs represent the primary flow of information form test to dev, the flow from developers to testers is through the build. Testers pick up new builds to get new features and requirements to test and bugs to regress. The developers’ efforts are realized in the build. As such the build becomes fulcrum of the dev/test relationship and takes a first class role in the VSTT10 product. When selecting a new build, the test manager sees which work items were completed as part of the build and act on those. She sees which tests were impacted by changes in the build and assign them to be run. The ability to see impacted tests is especially important in the end-game just before releasing to production, when the test manager and stakeholders need to make sure they have adequately tested the bug fixes that were made in the final builds.

Documentation links: Recommending Tests to Run That are Affected by Code Changes, Determining Which Builds Have Bug Fixes, New Features or Requirements

Preparing the Test Environment

The build is also central to ensuring high quality builds are passed to testers and assists in preparing the test environment for new builds. VSTT10 enables integration with VMM and Windows Workflow for automated provisioning and setting up the test environment as part of the build process. With VSTT10 build and deployment engineers can define a workflow script to provision machines from VMM, install the latest build, and then run functional BVT tests against the installed build. These can then be leverage by both the development and test teams to quickly set up complicated test environments. VS 2008 limited the level of testing to unit testing of assemblies whereas VSTT10 enables full functional build verification testing.

Testers leverage test environments from within the test client to quickly re-use environments. Network fencing enables testers to have more than one instance of the same VM running at the same time. This enables one person on the test or development team to get a test environment set up, and others on the team to quickly clone the environment for testing.

Test Automation in VSTT10

Test automation is a key and strategic investment area for VSTT10. Every test team wants to be more efficient, and automation is one way to increase efficiency. A goal for VSTT10 is to enable teams to define and run a automated regression suites to validate their primary scenarios are still intact. These automated regression suites are then used both as build verification tests and as a final set of regression tests to run before shipping.

The Record and Playback Engine

A primary investment in VSTT10 is the record and playback engine, which is leveraged in both automated tests and the fast forward for manual testing feature. A great deal of engineering has gone into ensuring record and playback “just works”. Record and Playback supports IE, Winforms, and WPF front ends. Silverlight support and playback on Firefox are planned for an out of band release (between dev10 and dev11). The RnP technology also supports a rich extensibility mechanism for extending to custom controls within these UI technologies as well as new browsers and future UI frameworks. This technology provides the underpinnings for our future UI Automation solutions.

UI Automation

VSTT10 includes a new “Coded UI” UI automation test type which is available in Visual Studio. Coded UI tests are squarely targeted at the more sophisticated end of the specialist tester segment, as it requires development skills. The “Coded UI” test type provides an action recorder, validation tool, and comprehensive API.

Coded UI Tests are built on the mstest unit testing framework, providing a familiar and common paradigm for testers and developers. Test authors leverage the full power of VS, the .NET framework, and managed languages to develop their automation.

Managing Automated Tests

Test management capabilities enable testers to track which test cases have or have not been automated, and to track automated test results over time. As part of the ability to scale-up test execution, VSTT10 enables you to define and run automated suites that span across VS solutions.

Documentation links: How to: Associate an Automated Test With a Manual Test Case and Run It, How to: Create Test Cases from an Assembly of Automated Tests

Performance Testing in VSTT10

VSTT10 builds on the success of VSTT 2008 performance testing tools, with innovative new tools that help testers find performance problems and enable development to fix them.

VSTT10 introduces WAN emulation for both manual and automated performance testing. Network emulation enables you to see how your application will perform over a slow link.

New advances in Web Performance Tests (formerly known as Web tests) make it far easier to debug them, and powerful new extensibility hooks in the web test recorder enable custom rules that can be targeted at specific types of sites so that record/playback will “just work” against those sites.

Load tests leverage data collector collectors to collect information from the system under test during a load test. A significant advancement in this area is the ability to remotely collect a profiler trace from the system under test, enabling testers to collect code-level performance data from the server during the test, so developers will have enough information to fix performance bugs.

The timing details view enables the performance tester to visualize virtual user activity, and quickly drill into a complete virtual user session log from a failed iteration. And new advances in reporting will enable you to easily share performance test results and trends with your team.

Advances in MSTest

Dev10 introduces significant advances in mstest extensibility, with new APIs for executing tests and getting results. This enables many new scenarios, including introducing new test clients outside of visual studio.

Dev10 also introduces test categories, which enable you to categorize tests (think tagging them) and dynamically select a set of tests that are in a particular category. A test can belong to multiple categories at a time. This enables you to move away from static vsmdi test lists, providing a richer way of selecting tests for BVTs.

Mapping Value Props to Feature Areas

Make manual testers more effective by integrating test activity into the ALM and providing tools for managing, developing, recommending and executing tests.
Enable manual testers to reliably and consistently create actionable bug reports via automatic data gathering, reducing debugging time and eliminating non-reproducible bugs.
Ensure that both requirements and source code are adequately and transparently tested, ultimately answering the question "can we release?"
Enable efficient collaboration between QA and the rest of the extended development team throughout all phases of the application lifecycle.
Foster a collaborative tester community with an extensible platform that makes it easy to create and share new features and components that further enhance a tester’s productivity and effectiveness.

	Effective Generalist Testers	Actionable Bugs	Can we Release?	Collab with Dev	Partners / Extensibility
Test Planning	Yes		Yes
Test Authoring	Yes	Yes		Yes
Test Case Linking	Yes		Yes	Yes	Yes
Test Runner	Yes	Yes	Yes
Test Reporting	Yes		Yes	Yes	Yes
Data Collectors	Yes	Yes		Yes	Yes
RnP	Yes	Yes		Yes
Test Automation	Yes		Yes	Yes	Yes
Test Impact	Yes		Yes	Yes
Build Integration	Yes			Yes
Lab Management	Yes	Yes		Yes
Public APIs	Yes	Yes	Yes	Yes	Yes

Conclusion

As you can see from this paper, this is a compelling release! But don’t take my word for it, try it out! Download dev10 beta1 today. You’ll need to download and install TFS server. The new test client is installed with VSTS in beta 1 (we are working on a stand-alone installer for beta 2). Once you have the product installed, here is a Quick Start Guide to get you up and running.

Dev10 Beta 1 Available!

edglas — Mon, 18 May 2009 18:15:00 GMT

Beta 1 was released to the web today for MSDN subscribers. If you are a MSDN subscriber, you can download the Beta today from here.

We have been very busy on this release of late (one reason why my blog posts have slowed down lately :))

My team and I will be blogging about all the great new features in dev10 beta 1 over the coming weeks.

As in the past, I will compose a list of links to any posts on the new features and update this blog post with the links as we get them.

New Web Test Features in Dev10:

These features are about making it easier to get web test playback working:

Extensible recorder plugins for modifying recorded web tests (including adding custom dynamic parameter correlation)
Find in playback
View recording log in playback
Jump to request from playback
Add extraction rule from playback
WebTestResult in a separate file for better scalability
New API for web test result
New extensibility API in playback enables UI extensions to playback

File Upload record/playback just works

Uploaded file added to project and deployed
Option for generating a unique file name on each iteration

Looping and branching in web test
New dialog for editing think times, goals, reporting string across all requests
New validation and extraction rules for inner text and select tags
Test level validation rule for pages meeting response time goal

New features in Load Test in Dev10:

Logging and data collection improvements

Log entire test result on test failure or log entire result every n requests
Remote profiler integration
Event Log Data Collection
Extensible data collectors to support more platforms

Analysis Improvements

Load test virtual user activity visualization, also here.

Reporting

Run to run perf reports in Microsoft Excel

Performance testing

Run automated UI tests as performance tests
True network emulation

More flexible load modeling

New Sequential Test Mix
Delay start on scenario
Iteration count on scenario
Select which agent(s) to run the scenario on
Disable during warmup

Multiple load test plugins
64 bit test execution

In addition to continuing our testing and bug fixing efforts, right now we’re working on an additional set of features for beta 2. I’ll have to sit on those until beta2. :)

Ed.

New, comprehensive technical documentation available for Web, Load, and Unit Tests

edglas — Tue, 31 Mar 2009 14:50:00 GMT

The VSTS Ranger team, led by Geoff Gray, just published the VSTT Quick Reference Guide 1.0.

This is a comprehensive collection of technical information on VSTT –- 83 pages of information!!! Many questions you may have on how stuff works in web, load, and unit test are answered in this doc. This doc is a must have for anyone working in VSTT.

Geoff and his team, the Microsoft Services Test Labs, deliver professional services for doing performance testing. They are experts at the tools, and work closely with my team (the product team) both to troubleshoot issues as they come up and to help steer the product in the right direction. Whenever Geoff saw an answer in email, figured it out himself, or saw it on a blog post, he put it in the document. He has been building it up over the past few years. I highly recommend you refer to it any time you are looking for more information or are stuck trying to get something to work in the product.

Here’s an extract from the document:

Congrats Geoff on getting this published! And thanks to others who helped along the way, and to Bijan Javidi for your support in sponsoring this project to get it published.

Ed.

VSTS wins Testers Choice in for .NET Apps in Software Test and Performance Magazine

edglas — Wed, 04 Feb 2009 17:13:00 GMT

The annual testers choice awards are out in STP magazine, and VSTS took top spot in two categories in this years tester's choice poll, winning out over load runner for ".NET Performance Testing" and for "Integrated Test/Performance Suite".

Congrats to the team who worked so hard on VS 2008 product, it's great to see our customers like the tool.

I'm dissapointed with the product write up in the magazine, it doesn't mention our load testing solution at all. Edward Correia, we need to talk! :) For a good overview of our performance testing solution, see Amit Chatterjee's recent blog post.

VS 2008 is a fine release, but there is still plenty of work to do! I expect our product line will continue to rise up in other categories as well with our next release. We have a fantastic set of new products for general testing on the runway, plus a ton of new performance testing features in Visual Studio Team Test. It's good to see TFS get the second place vote for defect tracking, I'm sure that as we expand our product line to appeal to all testers we will become the top choice in this and the test management category as well.

Ed.

Back from PDC

edglas — Tue, 04 Nov 2008 01:14:01 GMT

I spent last week in LA at the PDC. The conference was mainly focused on the new technologies Microsoft is working on bringing to market, such as Windows 7 and Azure.

We also introduced the features in our next VSTS release, which customers were super-excited to see. It's always great to step back at a time like this to take account of what we've been building since shipping the 2008 release. The demos were all using the CTP we just shipped.

I did a perf talk with Steve Carroll that you can watch here.

Some of the features I showed or talked about:

New run to run trend reports in Excel
Data collectors used to collect profiler data from the web server
Using UI tests as performance tests in a "load" test
New details view for visualizing virtual user actions and performance patterns

Cameron Skinner in his lap around VSTS 2010 also showed some of the new test case management features we are working on, you can watch that talk here.

There are more features in the CTP:

detailed logging in a load test
looping and branching in web tests
editing think times, response goals, and reporting name in web tests

These CTP bits were actually cut in July, so we've added a bunch of new features since then. I can't blog on those features yet, but they are also exciting! At this point we're really turning our attention to shutting down feature work to focus on shipping beta.

Ed.

Event ordering in Web Tests

edglas — Thu, 28 Aug 2008 18:49:49 GMT

Web tests have a number of extensibility points before and after a request. What order do they run in?

The hooks are:

WebTestPlugin.PreRequest

WebTestRequestPlugin.PreRequest

ExtractionRule.Extract

ValidationRule.Validate

Here is a sequence diagram that shows what order things fire:

There is no guaranteed order of firing for WebTestPlugin.PreRequest and RequestPlugin.PreRequest. Also if you have >1 Extraction or Validation rule on a request there is no guaranteed order they are run in.

Also note that Pre and Post request events do not fire by default for dependents. You can hook the PostRequest call on a top level request and iterate through dependents to wire up event handlers if you need that functionality.

Ed.

JW post on MS Testing - my observations

edglas — Tue, 19 Aug 2008 15:17:51 GMT

Check out James's recent post with the eye-catching title “if Microsoft is so good at testing, why does your software suck?”

I have another angle on this, which I saw first hand with our Whidbey release. It goes to some of the themes I refer to in my So you want to automate your test cases? post. We hire developers as testers. Our tester role is "Software Development Engineer in Test", or SDET. Our SDETs are awesome, our testers here at Microsoft are way better than testers I have worked with previously. But one problem I've seen is that because of our huge emphasis on test automation at Microsoft, sometimes ad-hoc or exploratory testing suffers because of it. That is, sometimes we're so busy writing automation, automation harnesses, and automation tools, we don't spend enough time in our customers shoes banging on our features and trying to break them. Almost all of the bugs we find are found through manual testing, not through automated testing. On my team we brainstorm on tests to run in order to break new features we're developing, schedule specific tasks in TFS for testers to beat on features, and also have dedicated bug bashes for our features.

Of course we can always do better, the recent load test regression that got through in SP1 is evidence of that. I wish we had had an automated test for that one, we do now. :) The timing of finding this bug was unfortunate, and we had another like it in RTM, was that one of our internal customers from the Services Test Labs found this bug in SP1 beta just as the door closed on SP1. We need to drive more adoption of our betas so we can catch these problems before RTM. The other thing is we were pretty aggressive in SP1 about getting fixes and features in that we heard about from customers. This regression was caused by one of those customer-reported problems ("Thinktime that's set via plugin code is ignored"). I don't see that changing, SPs are a chance to get value back to you, we need to take advantage of that. Of course we want to be smart about that, and make incremental changes that pack a lot of power but do not introduce a lot of risk.

Ed.

Updated load test links

edglas — Tue, 19 Aug 2008 11:19:00 GMT

I've updated my links to info on web and load test post with the following links:

New Item Templates for custom rule and plugin development

Data Driven Web Testing With Visual Studio 2008 Team System

InfoPath Forms Services 2007 Web Testing Toolkit

SharePoint Server 2007 Scalability and Performance Whitepaper

Diagnosing OutOfMemoryExceptions that occur when running load tests

Ed.

New templates for plugins and rules

edglas — Tue, 19 Aug 2008 10:30:35 GMT

Inspired by templates developed by Dennis Bass of the services test labs, I added new item templates to the codeplex.com\TeamTestPlugins project.

If you install these on your machine it makes creating new plugins much easier:

If you work with web and load tests you'll find these very handy, and they are xcopy-installed on your machine (you'll have them working in less than 5 minutes).

Now to get these in our next release. :)

Ed.

dynaTrace is cool!

edglas — Fri, 08 Aug 2008 20:13:29 GMT

dynaTrace just completed a nice integration with VSTS web, unit and load tests that enables them to do end-to-end tracing of performance. In addition to getting overall performance numbers, they have a feature called PurePath that let's you trace, down to a given web test request, what calls that request results in on the mid-tier and data-tier. Being the performance geek that I am, I love this stuff!

And the setup and deployment are a snap. This week Andreas Grabner, a perf architect for dynaTrace, came and helped us get it running to diagnose some TFS server performance problems we are working on. We also brainstormed on some more integration points in web and load test that will enable an even tighter integration in our next release.

Check out the two minute video when you have two minutes: http://www.dynatrace.com/twominutedemo/default.aspx

Ed.

Masking a 404 error in a dependent request

edglas — Thu, 07 Aug 2008 00:01:02 GMT

There are times when your web test may fail with a 404 error in a dependent request (like a css file or gif), but you may not actually care about this error. For example, it may be something that doesn't really impact the layout or content of the page. Or you might file a bug and temporarily want to suppress the error until it is fixed so they can find other errors.

If you get a 404 error in a dependent resource, here's how you can mask it.

Here's a failing web test, it is very simple. If I expand the top request I can see it is failing due to a failed dependent request.

To mask the request, right-click on the dependent and hit Copy.

Go to the test, right click on the corresponding request in the test and select "Add Dependent Request". Paste the url on the clipboard into the url property for the dependent request:

Now set the expected http status code on the dependent to 404.

Rerun the test to see it pass.

Hope that helps,

Ed.

WCF Load Test Tool is hot on codeplex

edglas — Wed, 30 Jul 2008 20:53:38 GMT

Check out the stats for the WCF load test tool on codeplex. The tool is approaching 2000 downloads! Kudos again to Rob Jarratt, who wrote this tool. It's great that we have a platform with the right extensibility hooks that enables us to produce out band solutions to key problems like this one.

I had the chance to actually meet Rob face to face and go to dinner with him last night. Rob is from the UK, so we'd never actually met! What a connected world we live in. We were both spending the day at TechReady, our Microsoft-internal technical field sales conference in Seattle (support, consultants, and technical sales come from around the world to exchange info and learn more about our products).

Ed.

Updated Links to Info

edglas — Sun, 27 Jul 2008 21:24:10 GMT

I updated my Links to Info on Web and Load Tests post with new links. The last update was in March, so it was due. I also moved some links into a separate table that are specific to 2005. Some of the 2005 context still applies to 2008, so I left that in the main table.

Most of the links are to new content my, Bill's, and Sean's blogs. Here's the new stuff:

New Web Test APIs for page boundaries and error handling

New Load Test APIs for load modeling

Support for IE8 and FireFox3 playback emulation

Support for random extraction of text

Description of web test execution

New rules for Inner Text parsing, list, and combo box

Disabling caching of all dependent requests

Collecting Code Coverage when Running Web and Load Tests

Sharing Test Results

How to create a custom counter set

SQL Record/Playback for Visual Studio Load Tests

Ed.