Web and Load Testing with Visual Studio Team System
Continuing the series on Testing with Visual Studio Team System, I devote this blog to our Web and Load Test functionality.
The web test and load test features were initially introduced in Visual Studio 2005, and enhanced in Visual Studio 2008. These tests are available in the Team Test and Team Suite editions of Visual Studio Team System.
Web tests enable you to generate http requests and responses, test for correctness of the responses, and measure response times and throughput. The primary scenario for web tests is to use them in a load test to generate load against a web application and measure web application performance. In order to efficiently generate load, Web tests work at the http layer, they do not drive the browser.
Load tests enable you to simulate many users hitting using an application at the same time, and then measure server response times, throughput, error rates, and resource utilization on the servers under test. Load tests will help you ensure your applications will stand up under load and deliver acceptable performance. In addition to driving web tests to simulate user activity, load tests can also run “unit” tests under load, which enables you to drive load to any server with a .NET API.
Enterprise Ready
This feature set delivers powerful enterprise-class load and performance testing features to the Team System product line. In the years since it was developed it has become the de facto load testing solution used internally by Microsoft. Nearly every server product team and web property now uses Visual Studio load tests. Server teams using this feature set include SQL, Windows, VSTS, and SharePoint, CRM, Dynamics, Media Center, and many more. Web properties include Hot Mail, Live Earth, Ad Center, Cars, Music, and many more. It is also the de facto tool used by our IT teams, who test everything from accounts payable, to the payroll system, to our polling system using VSTS.
It has also been used by a wide variety of customers and load test consultants to test their mission critical systems, with scale to 100,000s of users, on systems as varied as mission critical customer applications, to huge SharePoint deployments, to delivery of television content over phone lines.
Introduction to Web Tests
As I stated above, the primary use of Web tests is to use them in a load test for performance testing.
In order to efficiently drive load, Web tests work at the http layer. So rather than driving IE or another browser, they generate http requests and responses. A key thing web tests do not do is execute java script. Any http requests sent in java script will be recorded and played back, but again by sending http requests, not by running the java script. Under the covers the Web test engine uses the system.net classes in the .NET framework for generating http traffic.
Web tests can also be used as functional test to validate the application server responses, but since they run at the http layer cannot test anything that happens in the browser: java script execution, page layout, or control behaviors.
You start by creating a test project, then adding a web test to the project:
Which launches the recorder:
Which generates a Web test in Visual Studio:
Which you can then playback:
Playback allows you to view response time, error code status, and response size. For a given request/response, you see a preview of the html that was returned, and you can view the actual request and response data. The context parameters shows what values were in the virtual users context, and you can also see the results of any validation or extraction rules.
Web tests do a lot under the covers to do this effectively. When Web tests are run the Web test engine automatically does many of the same things the browser does:
Handle cookies
The web test engine automatically tracks cookies for a given "virtual user", storing cookies in a cookie container when received from the server, then sending them back as required on subsequent requests.
Authentication
NTLM, Kerberos, and forms authentication are all supported.
HTTPS Support
The Web test engine supports both http and https requests.
Automatically Download Resources
Most web pages contain images, style sheets, and java script. The Web test engine will automatically parse html responses and fetch the resources on the page (also called "dependent requests").
Follow Redirects
By default the Web test engine automatically follows page redirects.
Hidden Field Tracking
The web test recorder automatically generates code to extract hidden field values and bind them to subsequent requests.
Validation Rules
A validation rule is used to pass or fail a test. There are a set of canned rules that ship with web tests, such as the validation text rule. This rule will fail if the specified text is not found on the page.
Extraction Rules
An extraction rule is used to scrape data from a response for use in a later request. A good example of this is the hidden field extraction rule which extracts the values of all hidden fields on a page for use in the subsequent post request.
Dynamic Parameter Handling
Dynamic parameters are any parameters that change each time you run hit a web site. A great example is a session id, which is unique each time you create a new session with the web server. You can't simply replay the recorded session id, or the test will fail. Dynamic parameters are often stored in cookies and hidden fields, which are automatically handled as outlined above. Some querystring parameters also contain dynamic parameters, and the dynamic parameter detection tool detects these and automatically correlates them using extraction rules.
Data Driven Tests
For multi-user testing, you typically do not want each user entering the exact same data in the test. Data driven tests provide the ability to bind parameters to data from a data source. This enables you to easily test different data combinations.
Parameterization
You can easily parameterize a web test. A great example of this is using the same test to target different web servers, which you can do using the Parameterize Web Server command. The creates a context parameter with a default value, which can then be overridden from a load test or from an environment variable.
Extensibility
The Web test engine fires events to code before and after the test, and before and after each request, page, and transaction. This provides you with powerful hooks to a wide variety of things in your test. In addition, you can add your own extraction and validation rules.
"Declarative" or Coded Tests
When you create a new web test, a "declarative" style test is generated. Declarative tests are persisted as xml and have a designer. You can also generate a coded web test, which enables you to do looping, conditional execution, and compute parameter values in code. Generally the designer is easier to work with.
Included Test
Web tests can also be broken up into parts, and the parts can be re-used to build other tests. This can make test maintenance easier, since if one part of the app breaks, you only have to fix the web test components that hit that part.
Introduction to Load Tests
Load tests enable to you simulate many users hitting an application at the same time. The primary scenario for load testing is performance testing. How will the server respond to so many users? Will response times be unacceptably slow? Will error rates be acceptable? Another is capacity planning, where you may want to understand how large a server you will need to support the expected user load.
You start out creating a load test by walking through the load test wizard. The wizard will guide you through creating a scenario, which you can think of as a particular group of users. The load test scenario primarily contains:
- the load pattern, which defines how users will running at any given point in the test
- the test mix, which defines which test scripts they will be using. A test should roughly be viewed as a single users walk through the app.
In addition to web tests, load tests can drive "unit" tests, which enables you to drive load to any server that can be called from managed code (which is nearly every server or protocol on the planet).
Counter sets allow you to easily control which counters are collected from the load agents and the system under test. It is very easy to configure the counters to collect on a given machine. Individual counters also contain thresholds, which when triggered fire special alerts in the load test monitor.
The run settings allow you to map which machines to collect counters from, as well as various other properties of the test such as how long to run the test for. You can easily configure different run settings for running the same test against different servers.
Load Test Analyzer
You use the load test analyzer both during and after a run to analyzer the load test results. While the test is running, the analyzer displays performance counters collected from the load agents and system under test. You can control which counters are displayed on the graphs, as well as create new graphs. Once the test completes, you can also view a summary report as well as tabular data for the performance measurements collected during the test.
Once the test is complete, you can drill into specific areas of the test:
You can also view tabular data
And drill into details of requests that fail
Reporting
The summary report supports copy/paste to html format for easy sharing via email, web or msword.
The load test data can be published to Team Foundation Server which houses a summary report. All of the data is stored in SQL server, so you can use your favorite tool to report on the data. A powerful set of reports are available on codeplex.com/loadtestreports.
Scale Out Load Generation
With VSTS and VSTT you are able generate a significant amount of load. If you need to generate more load, you can run the load agent product on multiple computers to achieve scale-out load generation to many 1000s of users.
Resources and Community
There are many resources you can leverage to make your effort with these tools more effective. Ed Glas maintains a list of links which includes links to guidance, how to videos, blog posts, and various codeplex projects with complementary tools for load testing, including tools for WCF recording, SQL recording, and load test reports. There is also a very active community on our Web and load test forum.
Summary
As you can see, Web and load tests provide a rich and capable solution for your load testing needs. Give it a try today! You can download the VSTS trial here, which contains the web and load test features.
