Jeff Beehler's Blog : TSBT-TST

Rangers ship SQL load testing tool

jeffbe — Sun, 13 Jul 2008 17:37:45 GMT

The Team System Rangers have shipped again! This time they’re have built on the success of the WCF Load Testing Tool and have applied that same approach to SQL. You can find this tool on our Codeplex site: SQL Load Test. Here are the details from that site:

Project Description

This tool takes a SQL Profiler trace file and generates a unit test that replays the same sequence of database calls found in the trace file. The unit test is designed to be used in a Visual Studio Load Test. The code generated is easily modifiable so that data variation can be introduced for the purpose of doing performance testing. The tool generates code for both Visual Studio 2005 and Visual Studio 2008. The source code is a Visual Studio 2005 project. The tool is still in pre-release stage of development.

Why Load Test SQL?

There is a lot of code out there that makes load testing the application very difficult. The most common type is a client application that contains direct database calls. The application however is not structured to allow the business logic to be exercised without the GUI. Another common scenario is a legacy application written in a language such as Visual Basic 6. In cases like these, it can be difficult to load test the application without a GUI test tool, which introduces other disadvantages. GUI test tools are notoriously difficult to use, generally lead to brittle tests and also have limitations in scaling the load injectors sufficiently to exercise the servers. This tool offers one possible alternative. You start with using the client application to generate a trace of the SQL statements which typically represent a usage scenario. You can then turn the trace into equivalent ADO.NET code program that can be used in a load test.

Tool Description

The tool generates a Visual Studio 2005/2008 Unit Test from a SQL Server Profiler trace. It extracts all the SQL statements and stored procedure calls from the trace and turns them into a single Visual Studio Unit Test, which can then be configured as a Visual Studio Load Test. The tool does not interact with the database itself when it analyzes the trace and generates the test code. It can therefore be used in “offline” scenarios. The generated code needs to be customized to include a connection string to the database under test. That code also includes hooks to allow the user to customize the parameters that are passed to the SQL statements and stored procedures, so that variability can be introduced into the data to prevent caching from producing artificially high performance figures. The tool is intended to be used in conjunction with a client program that runs against a database. A trace of the client’s SQL Server activity is captured using the SQL Server Profiler. This represents a test scenario, which is then processed by the tool to produce a Unit Test that replays the scenario.

Congratulations to the various Rangers involved in this project! Please check it out and let us know what you think.

Check out good stuff from TechEd 2008

jeffbe — Fri, 06 Jun 2008 20:44:38 GMT

This was a good week for Team System at TechEd 2008 in Orlando. I would recommend that you check out the TechEd 2008 Online site (http://msdn.microsoft.com/en-us/events/teched/default.aspx) to learn about much of what was covered. In particular, you should check out the BillG keynote in which our very own Brian Harry demonstrated some exciting new features of Rosario (you can even read more about it here). In addition, you should watch the Team System panel when you can watch an interactive session with Brian, Gert Drapers, Neelesh Kamkolkar, Peter Provost, Habib Heydarian and Norman Guadagno.

Enjoy!

New edition of TFS Times

jeffbe — Mon, 26 May 2008 16:24:00 GMT

Paul Hacker, one of our 72 (!) Team System MVPs, has recently published a new edition of his very popular TFS Times. I’m impressed that even with his very busy personal and professional life that he’s able to publish this newsletter but somehow he pulls it off. This most recent issue focuses extensively on Testing related topics including the announcement of the Load Test Report Generator, details on web testing, and advancements in VSTS 2008 for testers, and much more. Take a look at what he’s put together for this month and let him know what else you’d like to see. I know he’d love to hear from you.

Load Test Report Generator now available on Codeplex

jeffbe — Tue, 13 May 2008 17:07:00 GMT

The Team System Rangers have shipped again!

Last Friday, we published our Ranger Load Test Report Generator to Codeplex. We decided to post this to the existing Load Test Reports Codplex site for customer convenience.

The Load Test Report Generator contains a utility and a set of SQL Server Reporting Services (SSRS) reports that allow users to generate Summary reports, counter comparison reports within and across test runs. The reports are simple but very flexible and can be used to generate & visualize various scenarios that a test team would like to create as a summary report for a set of load test runs.

The tool allows users to author and generate reports in HTML/MHT and Doc formats. It applies a task based approach for creating reports and is based on the feedback received from consultants in professional test labs.

Reports can also be generated using the concept of a template that makes it very easy to reuse predefined reports as templates for new ones. It is very simple to use custom build reports with the tool. All you need to do is publish your custom report in SSRS.

Congratulations to the Team System Rangers for their continued work in accelerating the adoption of Team System by filling the gaps in our current offering.

Watch James Whittaker talk about Software Testing

jeffbe — Fri, 09 May 2008 18:11:50 GMT

James Whittaker has recently joined the team responsible for producing Visual Studio Team System Test Edition as an Architect. He’s got years of experience in the testing field and it’s great to have him on board. Brian Keller recently interviewed him for a segment on Channel 9 which is well worth watching. You should also check out the Tester Center which contains even more material from James.

How DevDiv uses TFS

jeffbe — Sun, 20 Apr 2008 17:57:27 GMT

Gregg Boer, a program manager on the TFS work item tracking team has started a series of posts outlining in good detail how we used TFS work item tracking within Developer Division to manage the Visual Studio 2008 project. Based on the popularity of my Channel 9 video covering similar topics, I suspect this will be interesting reading for many of you. Check out the first 4 articles in his occasional series:

Hopefully this explanation of how we have implemented TFS internally will help you with your own implementations. Please let us know what you think.

Thanks!

April Rosario CTP now available!

jeffbe — Fri, 11 Apr 2008 15:36:29 GMT

I'm happy to announce that the next Community Technology Preview of Rosario (officially known as Microsoft® Visual Studio® Team System code name “Rosario” April 2008 CTP) is now available. Customers can immediately begin downloading the VPC images from our download page. Because of the size of the VPC, I'd recommend using a download manager, such as the one referred to here, to download the follow part of this release:

http://download.microsoft.com/download/b/f/9/bf9c1b30-ca2d-4381-8e9e-8b5db9701437/Apr2008RosarioCTP _12PartsTotal.part01.exe
http://download.microsoft.com/download/b/f/9/bf9c1b30-ca2d-4381-8e9e-8b5db9701437/Apr2008RosarioCTP _12PartsTotal.part02.rar
http://download.microsoft.com/download/b/f/9/bf9c1b30-ca2d-4381-8e9e-8b5db9701437/Apr2008RosarioCTP _12PartsTotal.part03.rar
http://download.microsoft.com/download/b/f/9/bf9c1b30-ca2d-4381-8e9e-8b5db9701437/Apr2008RosarioCTP _12PartsTotal.part04.rar
http://download.microsoft.com/download/b/f/9/bf9c1b30-ca2d-4381-8e9e-8b5db9701437/Apr2008RosarioCTP _12PartsTotal.part05.rar
http://download.microsoft.com/download/b/f/9/bf9c1b30-ca2d-4381-8e9e-8b5db9701437/Apr2008RosarioCTP _12PartsTotal.part06.rar
http://download.microsoft.com/download/b/f/9/bf9c1b30-ca2d-4381-8e9e-8b5db9701437/Apr2008RosarioCTP _12PartsTotal.part07.rar
http://download.microsoft.com/download/b/f/9/bf9c1b30-ca2d-4381-8e9e-8b5db9701437/Apr2008RosarioCTP _12PartsTotal.part08.rar
http://download.microsoft.com/download/b/f/9/bf9c1b30-ca2d-4381-8e9e-8b5db9701437/Apr2008RosarioCTP _12PartsTotal.part09.rar
http://download.microsoft.com/download/b/f/9/bf9c1b30-ca2d-4381-8e9e-8b5db9701437/Apr2008RosarioCTP _12PartsTotal.part10.rar
http://download.microsoft.com/download/b/f/9/bf9c1b30-ca2d-4381-8e9e-8b5db9701437/Apr2008RosarioCTP _12PartsTotal.part11.rar
http://download.microsoft.com/download/b/f/9/bf9c1b30-ca2d-4381-8e9e-8b5db9701437/Apr2008RosarioCTP _12PartsTotal.part12.rar

For this release we worked on a number of key scenarios that we'd like you to try out. To help facilitate your evaluation, we've included samples and walkthroughs within the VPC to demonstrate these new features. Here's a list of walkthroughs broken out by product:

Architecture Edition

Exploring the existing code structure
Designing process flow as activities
Designing user interactions with systems
Designing system functionality as components
Visualizing and designing types in systems
Visualizing and designing interaction sequences in systems

Development Edition

Simplify Code Analysis rule selection with rule sets
Find and fix a bug using the historical debugger
Identify the test impact of code changes
Find a bug on a separate machine using the standalone debugger

Database Edition

Building and using an off-line representation of your operational database as a “sandbox” development environment.
Using Data Generation to custom-build data for testing your database application.
Making and unit-testing schema and code changes in an off-line environment.
Performing static code analysis of your programmability objects.

Test Edition

Planning a testing effort
Executing manual test cases
Verify the fix
Automate a manual test and add validation.

Team Foundation Server

Managing an Agile schedule
Easier reporting from Excel
Managing features with the CMMI Process
A new Add Files to Source Control wizard and support for drag and drop from Windows Explorer to Source Control Explorer
An enhanced, non-modal conflict resolution experience, integrated into the pending changes tool window
A new history view that shows labels applied to a file as well as how changes were merged across branches
A new automated build system built on Windows Workflow Foundation, featuring dynamic build machine allocation from a machine pool and distributed build functionality
Rollback for a check in (currently only available at the command line)
Many Source Control Explorer usability enhancements

As you can see, we’ve been busy since our last CTP! As with our previous Rosario CTP, this release is not a beta. We’ve done a reasonable amount of testing on the bits to ensure that most the major features work to some degree and that the key experiences listed above work well but have not done any of the configuration or deep testing that we do for a beta release that we expect customers to use on a regular basis. As such, we encourage you to take the release for a spin and evaluate the new features we’ve added but please don’t expect to use it for any sort of production usage and understand that any data that you enter into the system cannot be migrated to future releases. And by all means, any applications written with these bits may not be redistributed in any way.

We’re releasing this CTP primarily for two key reasons: practice shipping and collecting feedback. Over the past few weeks, we’ve gotten a lot of the first as we’ve worked out the kinks associated with building, testing and releasing these bits to you. Hopefully in the coming weeks and months, we’ll get a lot of the second from you. Feedback in the form of bugs and suggestions can be submitted via http://connect.microsoft.com/visualstudio. Make sure when you submit issues through Connect that you indicate the product version as being “Visual Studio Team System Code Name “Rosario” (April 2008 CTP)”. That will ensure it gets to the right folks for consideration. If you have a question or would like provide feedback to the product team, we've created a special forum specifically focused on Rosario: http://forums.microsoft.com/MSDN/ShowForum.aspx?ForumID=1736&SiteID=1. We look forward to hearing from you.

Please join me in congratulating the Team System team on this release.

Woohoo!

jeff

84,248 Ads per second!

jeffbe — Mon, 24 Mar 2008 20:30:13 GMT

I think that the Load Testing capabilities within Team System Test Edition and the Load Test Agent are really still just waiting to be discovered by the majority of our users. When they do, they're in for a very big surprise. As an example of what these tools can do for an organization, Ed Glas recently posted details about the experience of the adCenter Display Ads Platform team and their use of Team System. Pretty impressive results including running 42 agents generating 3,639,518,000 requests. Check out the details here: http://blogs.msdn.com/edglas/archive/2008/03/17/84-248-ads-per-second.aspx.

Join me at our next Team System Chat

jeffbe — Mon, 24 Mar 2008 17:17:42 GMT

Join members of the Visual Studio Team System product group to discuss features available in Team Foundation Server, Team Suite, Architecture Edition, Development Edition, Database Edition, and Test Edition. In addition, discuss what's new for these editions for Visual Studio 2008.

Add to Calendar

April 9, 2008
10:00 A.M. Pacific Time
Additional Time Zones

Licensing Team System editions for your build machine

jeffbe — Tue, 18 Mar 2008 16:46:21 GMT

David Yack, one of our MS Regional Directors and a .NET MVP, tells me there is considerable confusion around the licensing requirements for the software installed on Team System build machines. The best resource available for these sorts of questions is the VSTS Licensing Whitepaper which goes into numerous licensing scenarios in depth. In the case of if you need a separate license of the Team System Development or Test edition on your build machine, in most cases you don't. Here's the excerpt from the whitepaper:

Those users who are authoring tests and including them as part of a Team Build type must be appropriately licensed with the relevant Team Edition product. For unit tests, Team Edition for Software Developers or Team Edition for Database Professionals or Team Edition for Software Testers or Team Suite would be required by the test author. For load and other test types, Team Edition for Software Testers or Team Suite would be required. Similarly, a user who includes code analysis in a Team Build Type must also be appropriately licensed for Team Edition for Software Developers. Users who merely execute and review Team Builds are only required to have a Team Foundation Server CAL.

The way I think about it, the people that are using the Team editions need to be properly licensed which in turn ensures the that the build machines are covered as well.

Hope this helps in some small way to reduce the confusion on this topic.

Load testing Windows Communication Foundation applications

jeffbe — Mon, 28 Jan 2008 04:00:56 GMT

The VSTS Rangers have done it again! They've identified a gap in our current product offering that was blocking it's adoption by a major customer and found a way to deliver tooling to fill the gap. In this case, the challenge at hand was load testing an application that was utilizing the new Windows Communication Foundation (WCF), part of the .NET Framework 3.0.

In response, our team wrote a tool which takes a WCF trace file and a WCF client proxy, or a WCF interface contract, and generates a unit test that replays the same sequence of calls found in the trace file. The code generated is easily modifiable so that data variation can be introduced for the purpose of doing performance testing.

The tool generates code for both Visual Studio 2005 and Visual Studio 2008. It also installs a wizard into both editions of Visual Studio for creating the trace and processing it from inside Visual Studio. If both editions are present the tool is installed into both editions. The source code is a Visual Studio 2005 project.

Features

The tool has the following main features:

Replay of captured scenario in a unit test that can be included in a load test.
Support for the DataContractSerializer.
Support for message contracts.
Support for proxies generated using svcutil.
Support for clients that create proxies at run time from contract interfaces.
Supports calls to multiple services in a single scenario.
Supports multiple calls to the same service operation.
Filtering by SOAP action of which messages in the trace to replay.
Readable and modifiable code is generated.
Automatic association of trace message with proxy method (requires all operations to have a unique SOAP action).
Support for client and server side traces.
A command line tool for processing traces and generating code.
Visual Studio 2005/2008 integration (Team Developer, Team Test, Team Suite and for 2008 also Professional)) that can be used instead of the command line tool.

You can learn more as well as download the contents from here on the Codeplex site.

Congratulations to the Ranger team for this powerful tool and the resulting customer win. Please take a look and let us know what you think.

Woohoo!

VS2008 Demo and Training kit available

jeffbe — Fri, 23 Nov 2007 17:09:04 GMT

The Visual Studio 2008 Training Kit has been updated for the RTM and has been released on the Microsoft.com Download Center for anyone to download and use. You can download the entire training kit from the download center here: http://go.microsoft.com/?linkid=7602397. After downloading the kit, simply run the installation program to extract the contents to your local machine. Once the installation process is complete, you will see an HTML page that allows you to navigate through the contents of the kit. It contains a full 5-days of technical content including 20 hands-on labs, 28 presentations, and 20 scripted demos.

Here are the specific Team System content items you'll find within this kit:

Presentations:

Introducing Application Lifecycle Management
Value-up software development
The Business Analyst Perspective
The Project Managers Perspective
The Architect Perspective
The Developers Perspective
The Testers Perspective

Demos:

Check-in Policies for Process Enforcement
Code Analysis
Code Metrics and Profiling
Creating and Executing Unit Tests
Process Governance with VSTS
Test Processes
Top Down Design

Props to Grant Holliday (VSTS MVP extraordinaire) for pointing me at this recent release. I love our MVPs...they often seem to know more than those of us on the product team!

December Team System chats

jeffbe — Thu, 22 Nov 2007 16:39:46 GMT

Come chat with the Visual Studio Team System group – December 5th

We will be holding two sessions:

Join the chat on Wednesday, December 5th, 2007 from 10:00am - 11:00am Pacific Time. Add to Calendar | Additional Time Zones

-and-

Join the chat on Wednesday, December 5th, 2007 from 4:00pm - 5:00pm Pacific Time. Add to Calendar | Additional Time Zones

Visual Studio Team System 2008 ships!

jeffbe — Mon, 19 Nov 2007 20:15:00 GMT

What else can I say? The big day has arrived (read the official word here on Soma's blog) and we're all very excited to let everyone know that all parts of the Visual Studio Team System 2008 product family are available today for download from MSDN. If you're not already a subscriber to MSDN, you can download a 90-day trial of Team Suite and TFS instead. We've been working on this release for the past two years so there's a whole lot of new features available, many of which we've talked about during the beta releases. Here's a quick run down of some of the key features in Team System this time around (drop me a note if I missed your favorites):

Development Edition
Code Analysis Checkin Policy
Code Metrics
JavaScript debugging
Performance run comparison
Hot pathing
Code Analysis Rules
Rules Management
Analyze menu
Spelling rules

Database Edition
Integrate TeamData v1 into Team Suite

Test Edition
Auto cleanup of results
Control Load Modeling
New default views
Devices Unit testing
Test performance improvements
Inherited test classes
Load test and profiler integration
Web test correlation
Right-click test execution
Test generation from a compiled assembly
Webtest recorder support for AJAX and popups
Webtest playback improvements

Architecture Edition
Architectural Roles
Conform to WSDL
Top-down system design
Web application projects

Team Foundation Server
SharePoint 3.0 / 2007 support
Improved deployment experience
Better offline support
Continuous integration
Build queuing & scheduled builds
Drop Management
Annotate
Folder Diff
Get latest on checkout
Performance and scale improvements (Work Item Tracking / Version Control)
Destroy files
Improved source control merging
Workspace mapping improvements
Team System Web Access (power tool)

Many, many congratulations to all the people that worked so hard over the past couple of years on this release. And more importantly, many, many thanks to our customers for providing us feedback on Team System 2005 as well as pre-releases of Team System 2008. You help us do a better job!

Please try it out and let us know how Visual Studio Team System 2008 works for you!

Woohoo!

Update on expiring VS2008 Beta2 VPCs

jeffbe — Sat, 27 Oct 2007 20:25:45 GMT

Over the past few days I've been doing a lot of research around the issue of the VPCs issued with VS2008 / TFS 2008 Beta2 expiring on November 1, 2007. I wanted to share my findings with everyone.

First, because it wasn't clear to me what would happen to the VPCs after November 1st, I've emulated the timeout experience on my machine. Turns out the behavior wasn't as bad as I had first imaged. After the OS expiration date passed, the VPC continued to run normally for an hour. Then a message box was displayed notifying me that the evaluation period for Windows had expired and that the VPC would continue to run for another hour before rebooting:

During this time, the system continued to run normally, meaning that you should have plenty of time to retrieve your data and move it to another system. After the system shut down, I was able to restart it and have another two hours to use it until it shut down again. As far as I can tell, I could repeat that cycle as many times as I wanted.

For some, like those using the VPC for demo purposes, this may be a tolerable situation and no additional action will be required. For those that need to use the VPC for longer than 2 hrs at a time, you can upgrade your installation of the OS as long as you have a valid license for Windows Server 2003. Turns out we have a few of those lying around here so I took one and upgraded my copy to be sure that worked okay. Anyone that has an MSDN subscription probably has valid licenses available to use for an upgrade.

After putting the DVD in (and using the CD menu to "Use Physical Drive D:") I got the following dialog

After typing in my Product ID (no, I won't be showing that dialog), the upgrade proceeded and completed in a couple hours. I verified with 'winver' that the OS was properly upgraded and that the timeout had been removed. Afterwards, I checked to make sure the installation of Visual Studio and TFS still worked correctly.

So, based on this evaluation, I see three possible alternatives:

Do nothing: if you don't need to use your VPC for more than an hour or so at at time (for instance for demos), then you can probably tolerate the behavior of the timed out OS.
Upgrade OS to fully licensed version: if you have access to a fully licensed version of Windows Server, follow the steps above and upgrade the expired OS.
Use new VPC images: if you can't upgrade the VPC OS and want to continue to use VS2008 / TFS2008 Beta2 on a regular basis, you'll probably want to use the new VPC images which we'll publish early the week of October 29. If you have data in TFS, you'll need to follow the instructions on moving TFS servers.

So, while it's an unfortunate situation that our VPCs are expiring early, I'm relieved that there are several alternative approaches for people to follow depending on their situation.

We're nearly done with the process of publishing the new VPC images. When that's complete, I'll post another note to let everyone know.