Jason Anderson's WebLog

Enterprise Development and Test tools

Why Unit Testing in Visual Studio Team System

My buddy James Newkirk (of NUnit fame) forwarded the following question to me from Ron Green:

 

Sender: Ron Green

=====================================

It appears from what I am reading that Unit testing is included as part of VSTS only. I don't understand why this wouldn't be included with Visual Studio itself. Unit testing really has nothing to do with the size of the development group. Unit testing is unit testing whether it's a group of 30 or a single developer.

 

Can you shed some light on this?

=====================================

 

First, thanks for the question Ron… I really appreciate you taking the time to write in and provide us with your feedback and insight.

 

Indeed, the current thinking is to offer the Unit Testing features in Visual Studio Team Developer and Visual Studio Team Tester which, as you correctly point out, are both parts of Visual Studio Team System. Today the placement of Unit Testing support in Team System is more a function of integration and less of a statement about team size.

 

Our view of testing revolves around the concept of Quality Assurance as a first class and productive member of the product lifecycle. To truly bring the entire value of Quality tools to our customer and to really provide the most value to the product lifecycle in general, we have to go beyond making great test tools. We have to make great test tools that are integrated with each other and integrated with other Team System tools that are fundamental parts of the product lifecycle family. For example, we see a lot of value with the integration of Unit Testing and Code Coverage. The shared scenario of being able to write and execute your Unit Tests integrated with the ability to examine Code Coverage data concerning the effectiveness of those Unit Tests is remarkable. Additionally, we see a huge value in the integration of Unit Testing with Load Testing. Adding Unit Tests with other tests types (such as Web Tests) to a Load scenario and turning the dials to make a custom load environment gives developers and testers an extreme amount of testing power.

 

I agree with you that Unit Testing has enough value to stand as a feature across all versions of Visual Studio. However, several of our tools in Team System fit into that category: Profiling, Static Analysis, etc. Our approach wasn’t really to ask, “Is this tool valuable enough to help all developers?”… We build most of our tools with that target in mind. Instead, we tried to ask the question, “Where can this tool provide the most value?” and in our current thinking the answer is along the lines of “as a feature integrated with the rest of the product lifecycle tools found in Team System.” Over time, we may revisit this decision, but for the 2005 version, we feel confident that this is the right choice.

 

Ron… thanks for the great question!

Published Friday, June 04, 2004 12:55 AM by Jason_Anderson

Comments

 

RichB said:

While I don't necessarily agree with Ron, I do believe that the Burton team is slightly off base in this area. Microsoft is well known for having QA people who develop code, when in fact most companies have QA people who know the business inside out but don't necessarilly know how to code. These people are called Functional QA testers.

Now, as most testers in the wider world are not developers they won't have a copy of Visual Studio on their machine and they also will not know what a code editor looks like. If they don't know what a code editor looks like, they will not need unit testing tools. This is where I believe Microsoft is going wrong.

There's a separate class of testers who are much smaller in number, but more closely resemble Microsoft's warped view of the general testing community. Those people are test harness testers and performance testers. They tend to know less about the business and more about code - and they definately do require tools which will enable them to reproducibly run tests against code (call that unit testing if you will, although it's not common usage).

Therefore, Visual Studio Team Tester targets a small percentage of the QA teams in the wide world.

Now, if Microsoft were to produce a set of functionality which "profiled" code and allowed a functional tester to see a straight percentage of how much of the codebase they had exercised during a black-box QA session and linked this with test scripts (English, verbal, scripts) and a bug database - it would be much more useful to most of the QA teams out there. Imagine setting a "profile" flag in an ASP.Net website and then getting the QA team to test a build of the website. After the QA session ends, the profile gets attached to the test management application for that particular test run. The developer logs onto the test management application to see the results of the test run - and sees graphically which parts of the code got executed, which caused exceptions, which caused crashed and bugs that were entered. If Microsoft can deliver this functionality, then they have hit the holy grail.
June 4, 2004 1:42 AM
 

Battaile Fauber said:

I agree with the above poster. While our development team has been trying to incorporate TDD and could benefit from these tools, our QA department will be able to make zero use of unit testing tools. Its somewhat ironic since microsofts certification tests seem to understand this distinction.

Battaile Fauber
Senior Developer
Protective Life
MCSD for .NET & VS6
June 4, 2004 9:12 AM
 

Chris Lucas's WebLog said:

June 4, 2004 12:56 PM
 

Craig Eddy said:

Well, I just sat through a WebCast by Microsoft employee James Newkirk where he advocates (and I agree) that DEVELOPERS write unit tests, not QA engineers.

Indeed, here is the description of the webcast from James' blog (http://weblogs.asp.net/jamesnewkirk/archive/2004/06/02/147304.aspx):
---------------------------------------------
In Kent Beck's book titled "Test-Driven Development, By Example" he defines Test-Driven Development (TDD) as driving software development with automated tests. He goes further by stating that TDD is governed by two simple rules: Write new code only if an automated test has failed and eliminate duplication. The implications of these two simple rules can proffer a profound change to the way that software is written. Most of the literature to date has bundled TDD along with Extreme Programming (XP). However, the benefits of using TDD are not limited to XP, and can be realized in any programming methodology. This webcast will provide an introduction into TDD, demonstrating how it works and what benefits it provides when used with Microsoft® .NET.
-----------------------------------

So why, again, is unit testing limited to VSTS??
June 4, 2004 10:22 AM
 

Visual Studio 2005 Team System said:

June 4, 2004 1:30 PM
 

Chris Lucas said:

Craig -

Visual Studio Team Developer is the Visual Studio Team System SKU focused on the developer; and Visual Studio Team Developer does include unit testing and code coverage. At a technology level the same testing framework can be used to do both unit testing and other forms of automated testing. For example, our QA team uses this feature. They use the same attribute based framework to write test code that exercises the interfaces between components in Visual Studio as our developers use to write their unit tests. Strictly speaking, what our QA team does is not "unit testing", but it does use the unit test framework. It's because the same technology can be put to multiple uses that the unit test tools are included in BOTH the developer and the tester SKU's. But, you're right, "unit testing" is a developer activity.

Chris
June 4, 2004 11:20 AM
 

Mike Kozlowski said:

It seems like all the reasons you're giving are marketing reasons, not technical ones, so a short answer to this would seem to be "Unit testing is cool, and of much value to everyone, but for marketing and sales reasons, it's in the Expensive Version, not part of the normal-price version", yeah?

While that's understandable, I think it's short-sighted. NUnit exists, and if you don't include MSUnit in regular VS.Net, a lot of people are going to continue using NUnit. (I'm tacitly assuming here that "Team System" is ridiculously pricy, and only big companies are going to buy it.) And once you've got oodles of test code written in NUnit, you're not going to rewrite it all to use MSUnit -- which means that you have less reason to upgrade to Team System, because you won't get to benefit from all the cool integration features of MSUnit, having locked into NUnit.
June 4, 2004 1:27 PM
 

Ken Brubaker said:

Q: Is unit testing a Development or QA Function? A: Both!
June 5, 2004 10:45 AM
 

Ken Brubaker said:

A lot of people seem confused on the relationship of QA and unit testing. Read my trackback above to understand why it makes sense to integrate unit testing with VSTS.
June 5, 2004 7:50 AM
 

Prashant Sridharan said:

While we have not yet finalized pricing, it is wrong to assume that Team System will be "ridiculously pricey". The current crop of "enterprise lifecycle tools" is indeed "ridiculously pricey", but the Team System will not be. More to come in a few months.

All of your points are well taken, but the value of these features is not their existence in and of themselves. The value is in how well they are integrated with one another. As Jason correctly points out, there is a tremendous amount of value in unit testing associated with load testing or code coverage. This is our approach with the Team System: we will not build individual features, rather we will build a coherent set of integrated tools.
June 6, 2004 1:16 PM
 

Sami Vaaraniemi said:

If running a full or partial build cycle is QA then I agree that unit testing is (also) part of QA. My dev team does a form of continuous integration which usually includes a local build every day (or at least before a check-in). The cycle is simplified from what you listed above: refresh sources from version control, build, run unit tests. In addition to getting blue faces from too much unit testing :), we find bugs and integration problems early in the cycle. We also have the daily master build.

There is admittably an inherent race condition with continuous integration but with the fairly small teams I've applied it with it has worked pretty well.

In my experience you get most value from unit tests by running them early and often (continuous integration) and also as part of a more formal daily build. It works like multi-level defence against bugs. Unit tests integrated with coverage and load testing is very nice to have too.

In the end of the day, I personally don't care if unit testing is dev or QA function. The thing that matters for me is I hope that VSTS allows for continuous integration.

If this counts as "really failing to understand unit testing", then I guess I'm a hopeless case.
June 7, 2004 7:17 AM
 

Sami Vaaraniemi said:

Oops. My previous comment was actually meant for Ken's thread...
June 7, 2004 7:22 AM
 

James Newkirk's Blog said:

June 7, 2004 6:29 PM
 

James Newkirk's Blog said:

June 7, 2004 6:30 PM
 

Brad Wilson said:

My answer is: who cares? Use NUnit!

I have to say that I agree that the post above just reeks of marketing. There's no good reason, other than "we don't wanna". It's a fundamental mis-understanding of what UNIT testing means. QA people don't write unit test; hell, they don't even RUN unit tests. Developers write them, developers (and continuous integration servers) run them.

I don't think they would've made this mistake had they consulted Jim Newkirk. But such is life... he's not on the developement tools team.
June 7, 2004 3:40 PM
 

Ron Green said:

Brad Wilson's cooment is exactly the point I was trying to address with the question. Unit testing is something devs do not QA. and if it's something devs do then it should be incorporated in all versions of Visual Studio.
June 7, 2004 4:17 PM
 

Robert Hurlbut said:

Well said, Brad. This is not "Unit Testing" as developers understand it. I write unit tests as a developer, and QA writes test cases to make sure an application meets the business requirements. These are not the same thing!
June 7, 2004 5:57 PM
 

Jim Argeropoulos said:

This is off topic of the rest of the comments, but...

Here here to the idea of unit testing and code coverage working together
June 8, 2004 6:26 AM
 

Ron Green's Weblog said:

June 13, 2004 2:10 PM
 

Being Scott Densmore said:

June 13, 2004 8:41 PM
 

Being Scott Densmore said:

June 13, 2004 8:42 PM
 

Jim Sather said:

It's obvious that passions run deep around who should be doing what and for what purpose. One of the fundamental goals of VSTS is to allow a wide range of flexibility and to allow choice. Don't mistake this to mean unstructured. VSTS allows you to tailor the structure and the process to meet your needs. Microsoft could certainly choose to ship unit testing in the developer SKU only, but that seems pretty restrictive and it violates the goal of flexibility and choice.

Absolutely, developers are the ones that quite often author and execute unit tests. Unit tests are often checkin requirements, regardless of whether you subscribe to XP/TDD/Agile or some other methodology. It’s a requirement in my own Product Unit.

Consider though, what happens when the development process is over. What happens when the component is handed off, when development staffing might not exits, and/or the component is in a sustaining mode. Consider if patches or small incremental changes need to be validated. A new platform, browser, OS update, server component, or if any dependency needs to be validated. It is not always the case that the job of executing those tests goes back to the original developer or development team.

Exhaustive testing should be able to include unit tests too, not just tester tests. Sometimes it's a business requirement for auditing reasons. When thinking of test prioritization, in some cases the highest yield tests to validate change, will be unit tests.

Finally, there are some very technical testers out there, and even some not so technical testers, that can use and author within a unit testing framework. One of the great things about full integration and making unit testing easy, is that it lowers the barrier to entry. It's more approachable by non-developers. A combination of white box and black box techniques is a solid approach for any Quality Assurance team.

I hope this helps explain why unit testing is currently slated for both VSTS Developer and VSTS Test.

-Jim
June 14, 2004 12:17 PM
 

Rob Caron's Blog said:

June 14, 2004 5:25 PM
 

Jeff said:

This whole question is stupid. If Unit Testing has enough value to stand as a feature across all versions of Visual Studio, and so does Profiling, Static Analysis, etc, then they ALL belong in ALL versions. Doing without them here in the goddamn 21st century is like doing without syntax highlighting.
June 14, 2004 3:40 PM
 

James Newkirk's Blog said:

June 15, 2004 4:12 AM
 

buk said:

"Today the placement of Unit Testing support in Team System is more a function of integration and less of a statement about team size."

I would then expect pricing options to reflect that its more a function of integration and less about team size. Amortizing the licensing fees for the server portions over 5 developers will not work. When you have a team of 50+ it makes sense. How will the Team System licensing work for small teams?
June 15, 2004 7:42 AM
 

Ron Jeffries said:

I got a demo of the unit testing stuff in VS, at the Agile Development Conference last week. Almost good.

Now I hear that it is only released as part of the Team System. This concerns me a lot. NUnit, on which the stuff I saw is clearly patterned, is a developer tool, a pair tool if you're pair programming. It is used by the developer to write tests in test-driven style, as he goes.

It is in the test-driven style that the real value comes out. We in the agile and XP communities have spent ages getting people to realize this. If you look at Java, which like it or not is the big competitor to C# and .NET, //all// the major IDEs have jUnit integrated at the developer level.

If you look at VS.NET today, NUnit is a plug in that can be hooked in by any developer anywhere.

Sure, unit testing and all those things belong at the team level /as well/. But test-driven development, the real reason why the xUnit programs exist, is a one-person-one-machine (or two-person if you're pairing) activity.

How do I get on your Beta list, or whatever it would take to get some ears at Microsoft? Or is it too late?
July 7, 2004 4:25 AM
 

scooblog by josh ledgard said:

July 16, 2004 6:53 PM
 

Kevin said:

I agree with the first poster. My QA team can't write code and has to work very closely with developers to write functional tests that matter. Team Edition will only go on a developers desktop. This is simply marketing!!!
July 21, 2004 1:48 PM
 

Jason Anderson said:

Hi Kevin,

Please note that the Test features in Team System are not simply about Unit Testing. Team System also includes Manual Testing and Web UI Testing - both of which won't require anyone on your team to write code. Check out the videos on VSTS posted on http://channel9.msdn.com to get a closer look at Test specific features beyond Unit Testing.

Thanks,

Jason
July 21, 2004 6:47 PM
 

Alan Dean said:

The Next Step? Parameterized Unit Tests
May 22, 2005 6:58 AM
 

ISerializable - Roy Osherove's Blog said:

MbUnit is gaining some momentum as one of the nicest unit testing frameworks for .NET out there. I just
September 23, 2006 8:51 PM
 

ISerializable - Roy Osherove's Blog said:

Update : There's a new Unit test framework called XUnit.NET. More info here. MbUnit is gaining some momentum

September 22, 2007 12:16 PM
 

Free Download Mp3 Song » Jason Anderson-The Wreath said:

PingBack from http://buyonlinemp3.myweblog.com/2007/11/07/jason-anderson-the-wreath/

November 7, 2007 6:42 AM
 

Download Music Online » Blog Archive » Jason Anderson-The Wreath said:

PingBack from http://monstablogs.info/downloadmusiconline/2007/11/21/jason-anderson-the-wreath/

November 21, 2007 4:35 PM
 

Jason Anderson s WebLog Why Unit Testing in Visual Studio Team System | Shed Kits said:

PingBack from http://backyardshed.info/story.php?title=jason-anderson-s-weblog-why-unit-testing-in-visual-studio-team-system

May 26, 2009 11:39 PM
 

Jason Anderson s WebLog Why Unit Testing in Visual Studio Team System | Wood TV Stand said:

PingBack from http://woodtvstand.info/story.php?id=9217

May 31, 2009 7:14 PM
 

Jason Anderson s WebLog Why Unit Testing in Visual Studio Team System | Quick Diets said:

PingBack from http://quickdietsite.info/story.php?id=2984

June 13, 2009 10:21 AM
Anonymous comments are disabled

This Blog

Syndication

Tags

No tags have been created or used yet.

© 2009 Microsoft Corporation. All rights reserved. Terms of Use  |  Trademarks  |  Privacy Statement
Microsoft
Page view tracker