nkamkolkar

Performance Testing in Agile Teams

2010-11-11T06:26:00Z

Here is link to the recording of a session I did last week on Performance Testing in Agile teams

https://www107.livemeeting.com/cc/msevents/view?id=5810&role=attend&pw=ATT5810

Getting Started with Visual Studio 2010 Ultimate - Load and Performance Testing

2010-11-02T12:23:00Z

If you are new to Visual Studio 2010 Ultimate Load Testing - this article is for you. This gives you a quick overview and shows you how you can get started.

Getting Visual Studio 2010

There are a couple of ways you can get access to Visual Studio 2010 to try it out.

Option 1: Download a trial

Download a trial version for Visual Studio 2010 Ultimate Trial from http://www.microsoft.com/visualstudio/en-us/download . The other versions will not work for you, so select the Web Installer or the ISO download for Visual Studio 2010 Ultimate Trial. This download gives you the ability to run 250 users of virtual user load for the trial duration.

If you need to scale up load, you simply buy the Visual Studio Virtual User Pack 2010 through your traditional channel or buy it from the store at http://store.microsoft.com/microsoft/Visual-Studio-2010-Load-Test-Virtual-User-Pack/product/7AFF7BC9 (visit click for approximate pricing).

Once you’ve downloaded your trial, simply install Visual Studio 2010 Ultimate by following the "Typical" install option on a suitable machine and you are ready to try it ou

Option2: Pre-Configured VHD/VPC

You can download a preconfigured VHD that will allow you to focus on trying the product features instead of installing and configuring. If you want to install and configure experience as well you should consider downloading the trial.

To download this VHD/VPC go to the link below and follow the download instructions on the page.

http://blogs.msdn.com/b/briankel/archive/2010/06/25/now-available-visual-studio-2010-rtm-virtual-machine-with-sample-data-and-hands-on-labs.aspx

This download is almost 9GB, so you should plan to kick off the download at night/off time as it’s a sizeable download and use the best practice of using the a download manager software to minimize download interruptions. Once you download it, run the appropriate .exe file (for your platform as in instructions on the above link). I downloaded the VisualStudio2010RTM_Win7VPC and had previously configured my Windows7 machine to have the VPC feature enabled.

For the size of the download, you will not only be able to give load testing a try (needs a couple more extra steps covered below), but also all of the manual, functional automation and other testing and development capabilities available in the Visual Studio 2010 release.

For the rest of the article I will focus on how to go about setting up a working Windows 7 VPC that I have downloaded and configured.

First Time VPC Configuration

Once VPC is configured and running, you should login to the VPC as “Christine Koch (Tester)” and use the password “P2ssw0rd” (without the quotes).

Once you have logged in (only for the first time), you must configure your browser defaults by just launching the browser and following the first time configuration wizard and accepting defaults.

Figure1: VisualStudio2010RTM_Win7VPC running on my machine

Figure 2: First time configuration for Visual Studio 2010 Ultimate

Additional Setup for VPC

Before you can use the downloaded and configured VPC specifically for load testing, you need to

1. Configure the SQL server that is already installed on the machine.

2. Configure Visual Studio 2010 to use the above database.

SQL is pre-installed on the image. You must Launch SQM Management Studio from the start menu, connect to the Database engine and open and run the SQL script located at C:\Program Files \Microsoft Visual Studio 10.0\Common7\IDE7\loadtestresultrepository.sql

Figure 3: Connect to SQM Management Studio

To configure Visual Studio to use the above database, from Visual Studio go to the “Test” menu and select “Manage Test Controllers…” and click on button to configure load test results store.

Figure 4: Configure Visual Studio to use SQL database (configure load test results not shown)

Ensure you use windows authentication and select “LoadTest2010” database from the list.

Figure 5: Configure Load Test Results Store

The image contains a pre-installed and configured sample application for you to use as your application under test. It is called TailSpinToys. It's an ecommerce application that allows end users to browse and buy toys. The application is running at http://Win-GS9GMUJITS8:8000/ or http://localhost:8000.

TIP: Launch the application and spend a few minutes to browse through the application and familiarize yourselves with the target application. You may want to book mark the application home page.

Web Performance Test Basics

Web performance tests enable you to generate http requests and responses, test for correctness of the responses, and measure response times and throughput. The primary scenario for web tests is to use them in a load test to generate load against a web application and measure web application performance. In order to efficiently generate load, Web performance tests work at the http layer, they do not drive the browser. For functional UI automation, you must use the Coded UI test type.

To create your first web test, from Visual Studio choose File \ New \ Project. In the dialog that follows, on the left hand side you see the installed template. Select the appropriate language of your choice on the left (Visual Basic or C#). Expand the node and choose “Test” to create a test project.

At the bottom of the dialog, choose a name for your test project. By default VS will create a new solution for you. In this walk through I have used C# for the language template and called my solution TailSpinToysLoadTestDemo

Figure 6: Create a test project with Visual Studio 2010

Note that you have an option to check in your tests into source control if you have setup Team Foundation Server. Load testing works even if you do have team foundation server installed and configured (although in the image there is a TFS instance running). Also, it is not necessary for you to check in web/load tests.

Visual studio creates a project and opens a Unit Test by default, ignore this.

TIP: If you haven’t used Visual Studio before you need to know that most of time for testing you will spend in the Solution Explorer pane. If this is not open by default, you must go to “View” menu and choose “Solution Explorer”

You must now “Add” a “Web Performance Test” by right clicking on the TailSpinToysLoadTestDemo

Figure 7: Add a Web Performance Test from Visual Studio

This opens the WebTestRecorder. Visual Studio 2010 using a browser helper object and WinINET based recorder to record your web tests. To record your web test, simply navigate to the target application URL http://Win-GS9GMUJITS8:8000/ or http://localhost:8000 and record your scenario. At each step of the way you can add a comment make your web test more readable by others.

Figure 8: Web Test Recorder opens in browser window navigate to target application to record.

In my case, I have done a complete recording of browsing for a model plane and going through a full purchase process. Once you’ve completed your recording, stop the recording and this will close the web test recorder and bring you back to the Visual Studio window with the recorded web test tab with all the recorded URLs. It will automatically re-run the web test to automatically identify and promote dynamic parameters so subsequent requests are parameterized and handled correctly.

For this specific application, and workload, the dynamic playback run fails (and this could happen depending on your application behavior) because the response URL validation test fails. This is a validation rule that is added out of the box for a web test, and the reason this fails is that in tail spin toys the application puts the receipt number as part of the response URL and this will vary each time you run the test; so you cannot compare this with earlier recorded response URLs that contains the receipt URL. For that reason, we can delete this validation for this scenario.

Figure 9: Delete Response URL validation rules

Once you have done this, you can re-run the web test to verify that it is passing.

Figure 10: Replay a web test to debug and fix any errors

TIP: In a real world scenario, there are will scenario’s where you will need to debug your web test to identify and resolve issues. Visit the content index for web and performance testing at my blog http://blogs.msdn.com/nkamkolkar/ and/or follow it to visit the post for debugging techniques http://blogs.msdn.com/b/edglas/archive/2010/03/24/web-test-authoring-and-debugging-techniques-for-visual-studio-2010.aspx

At this point you should have a web test against tail spin toys application that runs consistently and doesn’t fail. Playback allows you to view response time, error code status, and response size for a single user. For a given request/response, you see a preview of the html that was returned, and you can view the actual request and response data. The context parameters shows what values were in the virtual users context, and you can also see the results of any validation or extraction rules.

Your next step is to create a load scenario that can scale up the load, add this web test to it and run the load scenario.

Load Testing with Visual Studio 2010 Ultimate

Load tests enable to you simulate many users hitting an application at the same time. The primary scenario for load testing is performance testing. How will the server respond to so many users? Will response times be unacceptably slow? Will error rates be acceptable? Another is capacity planning, where you may want to understand how large a server you will need to support the expected user load. You start out creating a load test by walking through the load test wizard. The wizard will guide you through creating a scenario, which you can think of as a particular group of users. The load test scenario primarily contains:

1. The load pattern, which defines how users will running at any given point in the test

2. The test mix, which defines which test scripts they will be using. A web test should roughly be viewed as a single users walk through the app.

To keep this article a reasonable length, I will not go into the full details of the elements of the load test wizard since the articles primary goal is to get you started. Perhaps this is a subject of a subsequent article or blog.

When you add a load test, you will walk through a wizard. For getting started if you go through the basic wizard accepting the defaults you should be fine. Below, I will share some specific screens that are relevant to have a manageable load test for the VPC/VHD environment. For more information on load test wizard you can visit: http://msdn.microsoft.com/en-us/library/ms182572(v=VS.90).aspx

Figure 11: Load test wizard allows you to set up load scenario that scales up the load

You can select and/or define your load pattern for the scenario; given you are running this on a VPC/VHD change the default load from 25 users to say 5 virtual users as in figure 10.

Figure 12: Selecting a load pattern

Continue to choose defaults on the Test Mix Model screen. Following that, select “Add” and choose "TailSpinWebTest" or the equivalent name for the web test that you added earlier. Note that the "TestMethod1" is the default test that got created when you opened a test project, you can ignore this.

Figure 13: You can add web tests or unit tests to load scenarios using the load test wizard.

Continue to accept the defaults in the wizard until you to come to the Run Settings screen. Here you determine how your load test should run. Change the default Run duration to 1 minute again; once you deploy this to your test labs you can run 10+ hour runs easily.

Figure 14: Load test run settings

At this point you have a full load test scenario defined which you can run to generate load and then performance analysis. Note that a load test scenario in Visual Studio 2010 appears as “.loadtest” node and a web test appears as a “.webtest” node in the solution explorer under your test project. To edit either one at any time, you can simply double click the appropriate test type.

To run your load test simply hit the green “>” tool box button just below the tab title for the load test. This will start your load test and bring up the load test analyzer which shows your running load tests and performance metrics.

Figure 15: Running a load test

As you deploy the load test in your test lab and run real load tests, you will want to familiarize yourselves with the load test integration with ASP.NET profiler that allows you to not only get down to the exact method level detail for slow performing methods, but also show you the exact SQL queries and associated performance metrics. Visual Studio 2010 Ultimate also integrates with partner technologies for JAVA diagnostics. Below is an example screen shot of what ASP.NET profiler provides you insight into.

Figure 16: ASP.NET profiler integration with Tier Interaction Profiling enabled showing slow SQL queries

Working with Load Test Results Analyzer

Visual Studio 2010 Ultimate has integrated Load Test Analyzer that provides monitoring of the application and system under test in addition to providing visibility into key performance indicators, in addition to average, min, max, standard deviation, and 90th percentile data. You can view error rates, drag and drop metrics and also view innovative new visualizations such as VUSER ACTIVITY .

Figure 17: Load test analyzer monitors during the load test and provides for post run analysis

Load test analyzer also allows you to make analysis notes, generate run to run comparison as excel reports and also provides the ability to export your load test results as a load test archive which you can email to another Visual Studio 2010 Ultimate user for offline analysis.

Summary

Visual Studio 2010 Ultimate includes comprehensive load testing capabilities for your web applications. In this article I hope I have given you a way to quickly experience the basic flow and usage of the product and put you on a path to try out more advanced scenarios. There are many resources available in form of online MSDN documentation, blogs, videos, white papers, and forums to help you get to the next stage of using the product to solve real complex scenarios. You can find more information and resources at my recently started blog http://blogs.msdn.com/b/nkamkolkar

Content Index for Visual Studio 2010 Ultimate - Load and Performance Testing

2010-10-22T07:54:00Z

I am re-posting the content index blog post from Ed Glas' blog. This is a must read for anyone that is just starting out with load and performance testing with Visual Studio 2010 Ultimate.

http://blogs.msdn.com/b/edglas/archive/2007/12/17/content-index-for-web-tests-and-load-tests.aspx

In addition, if you are trying out load testing for RIA applications try the plugins at:

http://teamtestplugins.codeplex.com/

Troubleshooting Network Emulation

2010-06-25T11:15:00Z

Recently we had a customer support issue on trouble shooting the Network Emulation driver in VS2010 Ultimate while doing load testing. I thought a blog on how we troubleshooted and isolated the problem would be helpful, so here it is. In this blog, I discuss the problem, symptoms and also explain how Network Emulation works in 2010. I also suggest specific steps to consider to isolate and narrow down the problem.

Scope

This applies to Visual Studio 2010 Ultimate

Customer Scenario

The trouble shooting in this document is applicable to situations where you are attempting to use the Network emulation capability newly available in VS 2010 Ultimate while creating a new Load Test and in the "Edit Network Mix" screen of the wizard you select any other network type other than LAN.

What is Network Emulation?

Microsoft Visual Studio 2010 uses software-based true network emulation for all test types. This includes load tests. True network emulation simulates network conditions by direct manipulation of the network packets. The true network emulator can emulate the behavior of both wired and wireless networks by using a reliable physical link, such as an Ethernet. The following network attributes are incorporated into true network emulation:

Round-trip time across the network (latency)
The amount of available bandwidth
Queuing behavior
Packet loss
Reordering of packets
Error propagations.

True network emulation also provides flexibility in filtering network packets based on IP addresses or protocols such as TCP, UDP, and ICMP. This can be used by network-based developers and testers to emulate a desired test environment, assess performance, predict the effect of change, or make decisions about technology optimization. When compared to hardware test beds, true network emulation is a much cheaper and more flexible solution.

How Network Emulation Works in VS2010

Network emulation in VS 2010 Ultimate uses a network device driver that was designed and built by the Microsoft Research Labs and is productized in Visual Studio 2010. The technology has been around since 2005 and is widely used within Microsoft across many server product teams.

To use Network Emulation, you will need to install the Visual Studio 2010 Ultimate SKU. Network Emulation is configured as part of Add and new Load Test Type in Visual Studio and following the wizard screens (see above). Once you have set up network emulation following instructions at http://msdn.microsoft.com/en-us/library/dd997557.aspx, you will run your load tests. When the load test starts, it allocates a range of available ports for each of the Network profiles (DSL, 56.K Modem etc.) that you have selected in your network mix. This port range is available to the Network Emulation Driver that is enabled at run time (by default the network emulation driver is disabled).

During load testing, when the load generator sends a request to the application under test it specifies a port from the port range. When the network emulation driver sees this port from the select port range, it is able to associate this port with the network profile that this request should follow. This enables the driver to throttle the load in software ensure it meets the network profile you have selected.

How to know Network Emulation is not working?

Often one of the symptoms you'll see is that load test records socket exceptions in the log such as the one below:

"The requested address is not valid in its context xx.xx.xx.xxx:80"

NOTE: There may be other conditions that maybe causing such socket exceptions as well. The load test may continue to work, but the socket exceptions get logged. The next section will help you isolate and trouble shoot where the problem lies.

How to trouble shoot Network Emulation

To troubleshoot and isolate problems effectively you must ensure that you have done the basic tests.

1. Ensure that you have full network connectivity across all the machines that are participating in your load test.
2. Ensure you have configured the Network Emulation correctly by following the instructions and making sure admin rights are available for the agent.
3. Ensure that any/all firewalls are dis-abled (at least for trouble shooting) to ensure that firewall is NOT blocking specific ports or traffic on the lab network.
- a. Run tcpview (available here) to ensure that any socket connections are actually visible during run time (check for "red" highlights). You may also run your favorite port monitoring tool (portmon is another example)
4. Ensure that there is no virus software on the load generator machine that is possibly obstructing this software.
5. To isolate whether the problem is with the Network Emulation Driver or the Load Test Components you should:
- a. Eliminating the network emulation driver as a cause
  - Run the load test with network emulation configured correctly (even though you may be getting socket exceptions)
  - Ping another host to see whether the output shows network show down and/or higher latency. Check if the delay value matches selected network profile. If the latency values match the profile you have selected, then the network driver is working well.
  - From that agent machine where you are running the load test, attempt a connection to any host outside (like your favorite web page). This test verifies that while the load test is running and network driver is enabled, that external or lab connectivity is NOT a problem. This will isolate your network emulation driver from being a problem area.
- b. Eliminating the Load Test Components as cause
  - You should download and run this sample test program (available as is, not Microsoft supported) on the same machine as the load generator (agent machine). This sample program simulates the exact set of socket connection calls used in the load testing components. If this test program also displays Socket Exceptions (like in the image below) then this eliminates the Load Testing product as a cause for the socket exceptions and indicates the problem lies in the environment, machine, network or something external to the tooling. Please debug the external problem first before trying to run the load test again.
  - If this sample program is working correctly, you will see the output as below and this will confirm that there is a likely problem in the load test program and the environment is not the likely cause. Please contact support or post your query or situation in the forums for further help in this case.

Known Issues

There is a known issue with the Broadcom network cards where packets are dropped under heavy loads. We recommend if you run into this, try another network card until Broadcom addresses this problem.

Also, if IPSEC is enabled, the ports in the network packet are encrypted and as such the network emulation driver will not be able to determine that the packets are from the designated port range as set by the load test engine (described above in "How Network Emulation Works in VS2010"). You must disable IPSEC for network emulation to work.

Additional Resources:

http://msdn.microsoft.com/en-us/library/dd505008(VS.100).aspx

http://blogs.msdn.com/b/lkruger/archive/2009/06/08/introducing-true-network-emulation-in-visual-studio-2010.aspx

Introduction Blog - Load and PerformanceTesting PM

2010-06-25T08:16:00Z

Recently, I took over the role of program management for the Load and Performance testing tools in Visual Studio 2010 Product Family. Having been in the industry for 14 years(at the time of this writing) and spending close to half that time writing code for system level device drivers to managed applications for application performance and root cause diagnosticsI find that my passion for technology and passion for customers has only increased over the years. I am hoping that this blog will allow me to provide another key avenue to pursue customer and community connection to share ideas, learnings, best practices and to give you a peek into my world at work.

Ed Glas, Sean Lumley, Bill Barnett, Yun Tong, Lonny Kruger have over the years provided this window to the world from a product team perspective. You can continue to visit their blogs for all valuable content that is already published on the load and performance testing tools in VS. In fact, if you are getting started on Visual Studio 2010 Ultimate Load and Performance Testing, here is a great starting point from Ed's blog.

http://blogs.msdn.com/b/edglas/archive/2007/12/17/content-index-for-web-tests-and-load-tests.aspx

Also below are blogs from the creators/champions and supporters of the load and performance testing tools that includes valuable insights on various topics.

http://blogs.msdn.com/edglas/ (Ed Glas)

http://blogs.msdn.com/slumley/ (Sean Lumley)

http://blogs.msdn.com/billbar/ (Bill Barnett)

http://blogs.msdn.com/yutong/ (Yun Tong)

http://blogs.msdn.com/mtaute/ (Mike Taute)

http://blogs.msdn.com/lkruger/ (Lonny Kruger)

http://blogs.msdn.com/densto/ (Dennis Stone)

I am very excited about all the things we have done in the 2010 release to help testers and developers collaborate better. I am even more excited as customers and partners alike are becoming aware of what we've built and they try it out. I look forward to being a part of the community here at MSDN.

Neelesh Kamkolkar

Visual Studio 2010 Ultimate Program Manager

Load and Performance Testing