Jeff Beehler's Blog : Load testing

Resources to learn more about Visual Studio Team System 2010

jeffbe — Fri, 10 Apr 2009 19:29:40 GMT

While the various Visual Studio teams are working on our first beta release, I thought I’d point out some important resources to learn more about what we’re doing. First is the new weekly video podcast series known as 10-4. This has been going on for a few months now but I finally figured out where the name came from…I just couldn’t get past the CB phrase that was so popular when I was growing up. I now get that it’s a clever combination of VSTS 2010 and NetFx 4. Duh! I guess I’ve been triaging a few too many bugs recently. :-) Anyway, Brian Keller and crew have done a great job making this information very accessible and interesting along the way. Here’s a quick list of the podcasts posted to date:

10-4 Episode 14: Sentient DSLs

10-4 Episode 13: No More Late Surprises

10-4 Episode 12: Simplifying Your Code With C# 4.0

10-4 Episode 11: Bi-Directional Routing with ASP.NET WebForms 4.0

10-4 Episode 10: Making Web Deployment Easier

10-4 Episode 9: Visual Basic 10

10-4 Episode 8: Pure Client-Side Development with ASP.NET AJAX 4.0

10-4 Episode 7: No More Planning Black Box

10-4 Episode 6: Parallel Extensions

10-4 Episode 5: Code Focused in Visual Studio 2010

10-4 Episode 4: No More Parallel Development Pain

10-4 Episode 3: ASP.NET WebForms 4.0

10-4 Episode 2: Welcome to Visual Studio 2010

10-4 Episode 1: Working with the Visual Studio 2010 CTP VPC

The other recent activity in this space is Brian Harry’s new series on VSTS 2010 which he started this week: Team System 2010 Overview. If you haven’t already subscribed to his blog, I’d certainly recommend it. Brian’s always got interesting things to say.

Team Test 2008 Quick Reference Guide now available

jeffbe — Thu, 09 Apr 2009 21:54:21 GMT

The Rangers have shipped again! (Faithful readers know just how much I love saying that). This time, led by Geoff Gray, they have put together the VSTT 2008 Quick Reference Guide, a comprehensive technical document about Visual Studio Team Test functionality, specifically focusing on Web, Load and Unit testing. Geoff is a member of our Services Testing Lab, an arm of Microsoft Consulting Services and he’s help putting together this great reference based on his own personal experience using our products in customer settings. As such, it’s chockfull of very helpful advice and best practices informed by real world usage. If you’re at all interested in using our Testing tools, I’d highly recommend taking a look at this guide.

Rangers ship SQL load testing tool

jeffbe — Sun, 13 Jul 2008 17:37:45 GMT

The Team System Rangers have shipped again! This time they’re have built on the success of the WCF Load Testing Tool and have applied that same approach to SQL. You can find this tool on our Codeplex site: SQL Load Test. Here are the details from that site:

Project Description

This tool takes a SQL Profiler trace file and generates a unit test that replays the same sequence of database calls found in the trace file. The unit test is designed to be used in a Visual Studio Load Test. The code generated is easily modifiable so that data variation can be introduced for the purpose of doing performance testing. The tool generates code for both Visual Studio 2005 and Visual Studio 2008. The source code is a Visual Studio 2005 project. The tool is still in pre-release stage of development.

Why Load Test SQL?

There is a lot of code out there that makes load testing the application very difficult. The most common type is a client application that contains direct database calls. The application however is not structured to allow the business logic to be exercised without the GUI. Another common scenario is a legacy application written in a language such as Visual Basic 6. In cases like these, it can be difficult to load test the application without a GUI test tool, which introduces other disadvantages. GUI test tools are notoriously difficult to use, generally lead to brittle tests and also have limitations in scaling the load injectors sufficiently to exercise the servers. This tool offers one possible alternative. You start with using the client application to generate a trace of the SQL statements which typically represent a usage scenario. You can then turn the trace into equivalent ADO.NET code program that can be used in a load test.

Tool Description

The tool generates a Visual Studio 2005/2008 Unit Test from a SQL Server Profiler trace. It extracts all the SQL statements and stored procedure calls from the trace and turns them into a single Visual Studio Unit Test, which can then be configured as a Visual Studio Load Test. The tool does not interact with the database itself when it analyzes the trace and generates the test code. It can therefore be used in “offline” scenarios. The generated code needs to be customized to include a connection string to the database under test. That code also includes hooks to allow the user to customize the parameters that are passed to the SQL statements and stored procedures, so that variability can be introduced into the data to prevent caching from producing artificially high performance figures. The tool is intended to be used in conjunction with a client program that runs against a database. A trace of the client’s SQL Server activity is captured using the SQL Server Profiler. This represents a test scenario, which is then processed by the tool to produce a Unit Test that replays the scenario.

Congratulations to the various Rangers involved in this project! Please check it out and let us know what you think.

New edition of TFS Times

jeffbe — Mon, 26 May 2008 16:24:00 GMT

Paul Hacker, one of our 72 (!) Team System MVPs, has recently published a new edition of his very popular TFS Times. I’m impressed that even with his very busy personal and professional life that he’s able to publish this newsletter but somehow he pulls it off. This most recent issue focuses extensively on Testing related topics including the announcement of the Load Test Report Generator, details on web testing, and advancements in VSTS 2008 for testers, and much more. Take a look at what he’s put together for this month and let him know what else you’d like to see. I know he’d love to hear from you.

Load Test Report Generator now available on Codeplex

jeffbe — Tue, 13 May 2008 17:07:00 GMT

The Team System Rangers have shipped again!

Last Friday, we published our Ranger Load Test Report Generator to Codeplex. We decided to post this to the existing Load Test Reports Codplex site for customer convenience.

The Load Test Report Generator contains a utility and a set of SQL Server Reporting Services (SSRS) reports that allow users to generate Summary reports, counter comparison reports within and across test runs. The reports are simple but very flexible and can be used to generate & visualize various scenarios that a test team would like to create as a summary report for a set of load test runs.

The tool allows users to author and generate reports in HTML/MHT and Doc formats. It applies a task based approach for creating reports and is based on the feedback received from consultants in professional test labs.

Reports can also be generated using the concept of a template that makes it very easy to reuse predefined reports as templates for new ones. It is very simple to use custom build reports with the tool. All you need to do is publish your custom report in SSRS.

Congratulations to the Team System Rangers for their continued work in accelerating the adoption of Team System by filling the gaps in our current offering.

Team System 2008 Licensing White Paper now available

jeffbe — Tue, 13 May 2008 16:59:30 GMT

Kevin Kelly, a Team System Product Manager, announced today that at long last the Team System 2008 licensing white paper has been published. I’ve lost track of the number of people who have asked about this important update so I’m sure there are many people interested in this announcement. As Kevin says, we continue to simplify this topic and as a result you can expect an update to this whitepaper in August which will include explanations and examples for both Work Item Web Access (WIWA) and the new data synchronization license.

We’re thinking hard about our licensing for upcoming releases at this time. Your feedback on better ways to simplify and communicate our licensing is welcome and timely.

Thanks!

84,248 Ads per second!

jeffbe — Mon, 24 Mar 2008 20:30:13 GMT

I think that the Load Testing capabilities within Team System Test Edition and the Load Test Agent are really still just waiting to be discovered by the majority of our users. When they do, they're in for a very big surprise. As an example of what these tools can do for an organization, Ed Glas recently posted details about the experience of the adCenter Display Ads Platform team and their use of Team System. Pretty impressive results including running 42 agents generating 3,639,518,000 requests. Check out the details here: http://blogs.msdn.com/edglas/archive/2008/03/17/84-248-ads-per-second.aspx.

Load testing Windows Communication Foundation applications

jeffbe — Mon, 28 Jan 2008 04:00:56 GMT

The VSTS Rangers have done it again! They've identified a gap in our current product offering that was blocking it's adoption by a major customer and found a way to deliver tooling to fill the gap. In this case, the challenge at hand was load testing an application that was utilizing the new Windows Communication Foundation (WCF), part of the .NET Framework 3.0.

In response, our team wrote a tool which takes a WCF trace file and a WCF client proxy, or a WCF interface contract, and generates a unit test that replays the same sequence of calls found in the trace file. The code generated is easily modifiable so that data variation can be introduced for the purpose of doing performance testing.

The tool generates code for both Visual Studio 2005 and Visual Studio 2008. It also installs a wizard into both editions of Visual Studio for creating the trace and processing it from inside Visual Studio. If both editions are present the tool is installed into both editions. The source code is a Visual Studio 2005 project.

Features

The tool has the following main features:

Replay of captured scenario in a unit test that can be included in a load test.
Support for the DataContractSerializer.
Support for message contracts.
Support for proxies generated using svcutil.
Support for clients that create proxies at run time from contract interfaces.
Supports calls to multiple services in a single scenario.
Supports multiple calls to the same service operation.
Filtering by SOAP action of which messages in the trace to replay.
Readable and modifiable code is generated.
Automatic association of trace message with proxy method (requires all operations to have a unique SOAP action).
Support for client and server side traces.
A command line tool for processing traces and generating code.
Visual Studio 2005/2008 integration (Team Developer, Team Test, Team Suite and for 2008 also Professional)) that can be used instead of the command line tool.

You can learn more as well as download the contents from here on the Codeplex site.

Congratulations to the Ranger team for this powerful tool and the resulting customer win. Please take a look and let us know what you think.

Woohoo!

Check out the new Tester Center on MSDN

jeffbe — Mon, 22 Oct 2007 23:07:26 GMT

Testers play a critical role in the successful release of most software projects. However, to date, we haven't done a great job providing test specific information on MSDN. Today, with the introduction of the new Tester Center, that all changes. I encourage you to check it out today!

Some of the features I'm most excited about include:

Whiteboard videos - short focused conversations about typical topics that testers encounter on a daily basis. I appreciate the conversational tone and the fact that they're linked directly to forum posts for further discussion.
Testing experts - looking at the list of testing experts the team has amassed to support this effort with new content is stunning. You can expect to see them blogging, writing articles, making videos and answering forum posts.
Test column - ask your questions of the experts and get a detailed explanation from one of our experts.
Forum - everywhere you see this icon , it'll take you to forum discussion specifically on that topic. In addition, you can post your own discussion topics for the community to contribute to.

I'm looking forward to seeing this new Center come to life...with your help it can really turn into something great.

Aggregating popular Team System blogs

jeffbe — Wed, 05 Sep 2007 17:02:53 GMT

Blogs have quickly become a primary means for the Team System team to communicate with our community and as a result we have dozens and dozens of valuable blogs from our team. Because of this distributed approach, it’s hard for our customers to navigate and track all of the blogs related to various topics. We’ve also heard from many customers that they tire of the endless cross posts between our blogs…the echo can sometimes be deafening.

To help us dampen the echo and to provide one place for customers to go to learn about their favorite portions of the product, we are now aggregating the best Team System blog articles on each of the role specific Team Centers. As we learn more about how this approach works (and doesn’t work) for us and our customers, we’ll evolve this program. To get us started, I have identified key posts and posters to help us kick off this program. Please take a look, subscribe to your favorite feeds and let us know if this help you to stay in touch with us better.

Thanks!

jeff

Team System (rss)

Architect (rss)

VSArch

Developer (rss)

Test (rss)

Database (rss)

TFS (rss)

Project Manager (rss)

Team_WIT_Tools

Upcoming Team System webcasts

jeffbe — Wed, 01 Aug 2007 02:36:00 GMT

Adam Gallant recently posted a listing of many Team System webcasts that are scheduled for the next 10 months or so. Some of them look really interesting...maybe I'll listen in and learn something too! His blog has a full listing...I've just included the Team System related broadcasts here:

Whew!

Watch our TechEd 2007 sessions

jeffbe — Fri, 27 Jul 2007 00:29:00 GMT

The Team System team and community delivered quite a few talks at TechEd 2007 last month and I'm happy to announce that the recordings of these sessions, demos and all, are now available on the web for your viewing enjoyment. Here's a quick listing:

DEV347: Unit Testing and Test Driven Development: Brian Randell and Doug Seven
DEV309: Best Practices for Team-Based Software Development: Joel Semeniuk
ARC303: Building Your Own Software Factory: Edward Bakker and Jezz Santos
DEV313: Improving Code Performance with Microsoft Visual Studio Team System: Marc Popkin-Paine and Steve Carroll
DEV320: Writing Maintainable and Robust Applications with Microsoft Visual Studio Team System: Abhijit Rao and Noah Coad
DAT317: Database Schema Versioning: How to Use Microsoft Visual Studio Team System for Database Professionals and Team Foundation Server to Version and Deploy Your Databases: Gert Drapers
DEV322: Microsoft Visual Studio Team Foundation Server (Part 1 of 2): Applying Work Item Tracking and Version Control to Application Lifecycle Management: Brian Harry
DEV336: Microsoft Visual Studio Team Foundation Server (Part 2 of 2): Customizing Team Foundation Server for Application Lifecycle Management: Jeff Beehler
ARC315: Loose Coupling in Practice: Composite UI Application Block (CAB) in the Real World: David Platt
DEV329: Testing .NET 3.0 Applications with Microsoft Visual Studio Team System Test Tools: Mark Michaelis
DEV337: End-to-End Database Development with Microsoft Visual Studio Team Edition for Database Professionals: Duncan Davenport
ARC305: Practice-Centric Software Development with Microsoft Visual Studio Team Foundation Server: David West and Ivar Jacobson
DEV210: Microsoft Solutions Framework 4.0 Core and its Families: Rafal Lukawiecki
DEV335: What's New in Microsoft Visual Studio Team System for Testers, New Features in Visual Studio 2008 and Best Practices for Testing AJAX, SharePoint, and Reporting Services: David R. Williamson
DEV321: Best Practices for Deploying and Migrating Projects to Microsoft Visual Studio Team Foundation Server: Doug Neumann
DEV301: Integrating the Database into the Application Lifecycle Using Microsoft Visual Studio Team System for Database Professionals: Duncan Davenport

I think Noah's talk (with Abhijit demoing) got the highest rating of all of these and of course you've got to watch mine!

Enjoy!

Visual Studio 2008 Beta 2 is available!

jeffbe — Fri, 27 Jul 2007 00:07:49 GMT

I'm sure there will be many more posts (from me and others) but I wanted to get the good word out quickly that Visual Studio Beta 2 is now available for download! It's been a long haul but I'm really excited about what we've achieved. I'm confident it's ready for people to really pick up and use on a daily basis. But what really matters is to hear what you think...please let us hear from you!

For more information, please see Soma's announcement. If you just want to get to the bits, please head to our recently redesigned Developer Center on MSDN.

Woohoo!

Team System testing videos available

jeffbe — Mon, 02 Jul 2007 19:48:00 GMT

Here's a great series of testing oriented videos for Team System hosted on the ASP.net site:

#1 | Introduction to Unit Testing with Team System

(10 minutes, 54 seconds)

#2 | Introduction to Testing Web Applications with Team System

(9 minutes, 43 seconds)

#3 | Introduction to Load Testing Web Applications with Team System

(9 minutes, 8 seconds)

#4 | Introduction to Manual Testing with Team System

(8 minutes, 56 seconds)

#5 | Introduction to Managing and Running Tests with Team System

(8 minutes, 37 seconds)

#6 | Measuring the Business Value of AJAX

(10 minutes, 27 seconds)

#7 | Code Coverage of Automated Tests

(6 minutes, 49 seconds)

#8 | Custom Extraction Rules and Coded Web Tests

(12 minutes, 39 seconds)

#9 | The Effects of Caching

(15 minutes, 36 seconds)

#10 | Using the Load Test Agent

(5 minutes, 43 seconds)

#11 | The Effects of ViewState

(10 minutes, 4 seconds)

#12 | How Do I: Integrate Defect Tracking with Testing?

(8 minutes, 2 seconds)

#13 | How Do I: Create My Own Bug Work Item?

(11 minutes, 23 seconds)

#14 | How Do I: Write Code More Quickly with Unit Tests?

(8 minutes, 40 seconds)

#15 | How Do I: Practice Test-Driven Development?

(8 minutes, 21 seconds)

#16 | How Do I: Load Test a Web Application?

(8 minutes, 31 seconds)

#17 | How Do I: Tune Web Application Performance with Profiling?

(8 minutes, 33 seconds)

Performance testing guide

jeffbe — Tue, 29 May 2007 16:28:06 GMT

JD and his p&p team are on fire. In the space of two days last week, they released both the Beta1 of the TFS Guide and the Beta1 of the Performance Testing Guide. They both look great and are ready for your feedback to make them even better. JD writes the following description of the perf testing guide in a recent blog article:

Today we released our Beta 1 of Performance Testing Guidance for Web Applications Guide. It shows you an end-to-end approach for implementing performance testing, based on lessons learned from applied use in customer scenarios. Whether you're new to performance testing or looking for ways to improve your current approach, you'll find insights you can use.

Good stuff...check it out today!