Introducing Cloud-based Load Testing with Team Foundation Service

Introducing Cloud-based Load Testing with Team Foundation Service

Rate This
  • Comments 17

Update: As announced with Visual Studio 2013 launch, cloud based load testing with Visual studio Online is in Commercial Preview. This is open to all Visual Studio Online users. You can read more about it here

 

Performance and load testing is a very important part of the application lifecycle. You’d want to do this every time before you deploy the latest changes to production. However, one of the biggest challenges with performance and load testing today is setting up infrastructure required to run the tests. Acquiring machines, configuring them with the right settings, and managing the test environment– all of these are no easy jobs and require significant time and effort.

Going forward, you do not need to worry about this problem. As announced by Brian Harry this morning, here’s introducing cloud-based load testing as part of Team Foundation Service. The service will take care of setting up the agents, deploying your tests, and running your tests automatically, so you can just focus on what matters the most – finding and fixing performance and scale related problems in your application.

Here are the key features of the service:

  1. Simple to use and get started: All you need to get started is a Team Foundation Service account. You can just connect to this account from Visual Studio and get started instantly. No more pre-requisites.
  2. Capacity on-demand: One of the important aspects of load testing is that you need to generate different amount of load at different phases in your application lifecycle to simulate real-world conditions. The load testing service is elastic and dynamic, and is designed for this scenario. i.e. whether you need to run a load test with 200 concurrent virtual users or 20,000 – you’ll be able to queue a test as per your requirements.
  3. Same tests work on-premises and on the cloud: The other important aspect is that the tests you have authored on-premises just run on the service - this includes the load tests, declarative web performance tests, coded web tests, and unit tests. This means if you have a load test project authored using Visual Studio 2010 or Visual Studio 2012, you can just run the same tests using the service and take advantage of the cloud.


Earlier today, I did a breakout session at TechEd North America where I showed how this capability works and talked a little bit about the architecture and roadmap. Here is the recording of this session to learn more. For now, let us do a quick walkthrough of running load tests using Team Foundation Service.

  1. Configure test settings

Run your existing load test on the service with just one change in test settings. In Visual Studio 2013 Preview, you’ll find a new option to run tests using Team Foundation Service. You can target your load tests to run using the service by choosing this setting. That’s it – you are all set.


 

2. Run tests on the cloud

When you click ‘Run Load Test’, Visual Studio uses the existing connection to a Team Foundation Service account to automatically connect and run your tests on the service. The agents get automatically created and your load test starts running. You can view the progress of the run and monitor performance and throughput counters while the test is in progress.

 

 

3.       Manage tests run using the service

Using the new Load Test Manager, you can view load tests run by any member from your team using the Team Foundation Service and review the results anytime from anywhere.

 

 

Next steps/How do I try?

Can’t wait to get started? You will be able to do so very soon. As announced by Brian, Visual Studio 2013 Preview will be available at the //build/ conference on Jun 26th. At that time, you will be able to try this out. Load testing will be open in Preview mode for all users of Team Foundation Service with a limited quantity per month. Try out the service and if you have scenarios where you'd need higher usage, you can join the early adopter program. Stay tuned for more details on the service and the early adopter program.

We'd love to know what you think. Keep the feedback coming.

Leave a Comment
  • Please add 4 and 2 and type the answer here:
  • Post
  • Great stuff guys!

  • Thank you Tarun

  • Do you also support other test types in the cloud? We have MSTest-based Integration tests. A complete test run takes about 4 hours. It would be great if I could easily run these tests in the cloud.

  • Mathias,

    The answer is no. At the moment, we only support running load tests on the cloud. You can have unit tests as part of the load tests though.

  • Will this work only with sites that are internet-visible, or is there way to set up a VPN?

  • @Aidan, this will work as long as the end-point is publicly accessible. If your end-point isn't public, one option is to set a rule to just avoid traffic in from a specific IP range, for the duration of your testing.

  • Update: Visual Studio 2013 Preview is now available and you can try out load testing with Team Foundation Service. You can find links to the getting started guide and the early adopter program at: http://aka.ms/loadtfs

  • There are useful Apache JMeter tutorial on https://loadfocus.com/blog/, also Load Focus https://loadfocus.com/ offers a cloud load and performance testing service to stress your web app or API with thousands of users with live monitoring also running from tablets.

  • Nice article. You can find useful information in out blog http://blazemeter.com/blog

  • It would be much better to focus on stability instead of adding some features. We experienced a lot of problems with long build in mai (3 days no builds). And now we have again the same problems that builds are not running... I'm really starting to be dissapointed in TFS service as it is extremely unreliable.

  • @Denis,

    We are very sorry about the outage. Yesterday's outage was caused due a problem in Azure storage. We have a fix for this and load test service will be up shortly. We understand how frustrating it can be when services go down and right now - most of our team is focused on improving the reliability and scale for load testing. I know the build team is working on improving reliability as well and I'll pass this feedback to them.

    Once again, I apologize for the inconvenience and you should expect us to get better on reliability in the coming days.

    Thanks.

    -Darshan

  • Are there opportunities to run load tests which are started from different geographical locations?

  • @Phil,

    Currently we only support running load tests from US. However, this item is on our product backlog and we'll enable running load tests from different geographical locations in the future.

  • As requested by Jack Zhai in the thread on the following forum:

    social.msdn.microsoft.com/.../load-testing-and-collecting-performance-counters-on-target-environment-in-vs-2013

    I leave a comment about performance counters, test agent and test controller.

    When I last ran performance test using Team Foundation Service and VS 2013 I noticed that the agent-controller configuration is ignored when executing the test. This is apparently because they are no longer needed. That sounds like a good thing since it removes the need to set up additional machines and configurations during test environment setup. The other big change was that there seemed to be no way of collecting performance counters from target environment when executing load test. This was because team foundation service does not run in the same workgroup/domain as target environment and therefore cannot access this data from target environment.

    The fact that I could not get performance counters anymore led me to set up my test environment in azure using VMs with VS 2012 and installing test controller and agents just like the old setup except that it runs in Azure. Once you set up your virtual network, installed the VMs, made some firewall changes and added endpoints on the machines this works fine and until the load testing features of VS 2013 using team foundation service supports collecting performance counters from target environment I will stick with this solution.

    Thanks for the feedback on my question in the forum. My question was answered.

  • @Jonas,

    Yes, you are right. Currently we have no way of collecting performance counters automatically when you run your load tests thru' Team Foundation Service (now called Visual Studio Online). We are working on adding this feature, please stay tuned for updates.

    The on-premises way of collecting performance counters (by making a direct connection to server) isn't the best way to do this in the cloud, hence we are working on a better cloud-based solution. You can use it for the time-being as a workaround until we deliver the new solution.

    That said, you should check out Application Insights in Visual Studio Online that lets you monitor your applications: www.visualstudio.com/.../find-performance-problems-vs

    Using Application Insights in conjunction with load testing is another way to solve this problem.

    -Darshan

Page 1 of 2 (17 items) 12