Why Unit Testing in Visual Studio Team System

re: Why Unit Testing in Visual Studio Team System

RichB — Fri, 04 Jun 2004 08:42:00 GMT

While I don't necessarily agree with Ron, I do believe that the Burton team is slightly off base in this area. Microsoft is well known for having QA people who develop code, when in fact most companies have QA people who know the business inside out but don't necessarilly know how to code. These people are called Functional QA testers.

Now, as most testers in the wider world are not developers they won't have a copy of Visual Studio on their machine and they also will not know what a code editor looks like. If they don't know what a code editor looks like, they will not need unit testing tools. This is where I believe Microsoft is going wrong.

There's a separate class of testers who are much smaller in number, but more closely resemble Microsoft's warped view of the general testing community. Those people are test harness testers and performance testers. They tend to know less about the business and more about code - and they definately do require tools which will enable them to reproducibly run tests against code (call that unit testing if you will, although it's not common usage).

Therefore, Visual Studio Team Tester targets a small percentage of the QA teams in the wide world.

Now, if Microsoft were to produce a set of functionality which "profiled" code and allowed a functional tester to see a straight percentage of how much of the codebase they had exercised during a black-box QA session and linked this with test scripts (English, verbal, scripts) and a bug database - it would be much more useful to most of the QA teams out there. Imagine setting a "profile" flag in an ASP.Net website and then getting the QA team to test a build of the website. After the QA session ends, the profile gets attached to the test management application for that particular test run. The developer logs onto the test management application to see the results of the test run - and sees graphically which parts of the code got executed, which caused exceptions, which caused crashed and bugs that were entered. If Microsoft can deliver this functionality, then they have hit the holy grail.

re: Why Unit Testing in Visual Studio Team System

Battaile Fauber — Fri, 04 Jun 2004 16:12:00 GMT

I agree with the above poster. While our development team has been trying to incorporate TDD and could benefit from these tools, our QA department will be able to make zero use of unit testing tools. Its somewhat ironic since microsofts certification tests seem to understand this distinction.

Battaile Fauber
Senior Developer
Protective Life
MCSD for .NET & VS6

Jason Anderson talks about SKU planning and unit testing...

Chris Lucas's WebLog — Fri, 04 Jun 2004 19:56:00 GMT

re: Why Unit Testing in Visual Studio Team System

Craig Eddy — Fri, 04 Jun 2004 17:22:00 GMT

Well, I just sat through a WebCast by Microsoft employee James Newkirk where he advocates (and I agree) that DEVELOPERS write unit tests, not QA engineers.

Indeed, here is the description of the webcast from James' blog (http://weblogs.asp.net/jamesnewkirk/archive/2004/06/02/147304.aspx):
---------------------------------------------
In Kent Beck's book titled "Test-Driven Development, By Example" he defines Test-Driven Development (TDD) as driving software development with automated tests. He goes further by stating that TDD is governed by two simple rules: Write new code only if an automated test has failed and eliminate duplication. The implications of these two simple rules can proffer a profound change to the way that software is written. Most of the literature to date has bundled TDD along with Extreme Programming (XP). However, the benefits of using TDD are not limited to XP, and can be realized in any programming methodology. This webcast will provide an introduction into TDD, demonstrating how it works and what benefits it provides when used with Microsoft® .NET.
-----------------------------------

So why, again, is unit testing limited to VSTS??

re: Announcing Visual Studio 2005 Team System

Visual Studio 2005 Team System — Fri, 04 Jun 2004 20:30:00 GMT

re: Why Unit Testing in Visual Studio Team System

Chris Lucas — Fri, 04 Jun 2004 18:20:00 GMT

Craig -

Visual Studio Team Developer is the Visual Studio Team System SKU focused on the developer; and Visual Studio Team Developer does include unit testing and code coverage. At a technology level the same testing framework can be used to do both unit testing and other forms of automated testing. For example, our QA team uses this feature. They use the same attribute based framework to write test code that exercises the interfaces between components in Visual Studio as our developers use to write their unit tests. Strictly speaking, what our QA team does is not "unit testing", but it does use the unit test framework. It's because the same technology can be put to multiple uses that the unit test tools are included in BOTH the developer and the tester SKU's. But, you're right, "unit testing" is a developer activity.

Chris

re: Why Unit Testing in Visual Studio Team System

Mike Kozlowski — Fri, 04 Jun 2004 20:27:00 GMT

It seems like all the reasons you're giving are marketing reasons, not technical ones, so a short answer to this would seem to be "Unit testing is cool, and of much value to everyone, but for marketing and sales reasons, it's in the Expensive Version, not part of the normal-price version", yeah?

While that's understandable, I think it's short-sighted. NUnit exists, and if you don't include MSUnit in regular VS.Net, a lot of people are going to continue using NUnit. (I'm tacitly assuming here that "Team System" is ridiculously pricy, and only big companies are going to buy it.) And once you've got oodles of test code written in NUnit, you're not going to rewrite it all to use MSUnit -- which means that you have less reason to upgrade to Team System, because you won't get to benefit from all the cool integration features of MSUnit, having locked into NUnit.

VSTS: Is unit testing a Development or QA Function?

Ken Brubaker — Sat, 05 Jun 2004 17:45:00 GMT

Q: Is unit testing a Development or QA Function? A: Both!

re: Why Unit Testing in Visual Studio Team System

Ken Brubaker — Sat, 05 Jun 2004 14:50:00 GMT

A lot of people seem confused on the relationship of QA and unit testing. Read my trackback above to understand why it makes sense to integrate unit testing with VSTS.

re: Why Unit Testing in Visual Studio Team System

Prashant Sridharan — Sun, 06 Jun 2004 20:16:00 GMT

While we have not yet finalized pricing, it is wrong to assume that Team System will be "ridiculously pricey". The current crop of "enterprise lifecycle tools" is indeed "ridiculously pricey", but the Team System will not be. More to come in a few months.

All of your points are well taken, but the value of these features is not their existence in and of themselves. The value is in how well they are integrated with one another. As Jason correctly points out, there is a tremendous amount of value in unit testing associated with load testing or code coverage. This is our approach with the Team System: we will not build individual features, rather we will build a coherent set of integrated tools.

re: Why Unit Testing in Visual Studio Team System

Sami Vaaraniemi — Mon, 07 Jun 2004 14:17:00 GMT

If running a full or partial build cycle is QA then I agree that unit testing is (also) part of QA. My dev team does a form of continuous integration which usually includes a local build every day (or at least before a check-in). The cycle is simplified from what you listed above: refresh sources from version control, build, run unit tests. In addition to getting blue faces from too much unit testing :), we find bugs and integration problems early in the cycle. We also have the daily master build.

There is admittably an inherent race condition with continuous integration but with the fairly small teams I've applied it with it has worked pretty well.

In my experience you get most value from unit tests by running them early and often (continuous integration) and also as part of a more formal daily build. It works like multi-level defence against bugs. Unit tests integrated with coverage and load testing is very nice to have too.

In the end of the day, I personally don't care if unit testing is dev or QA function. The thing that matters for me is I hope that VSTS allows for continuous integration.

If this counts as "really failing to understand unit testing", then I guess I'm a hopeless case.

re: Why Unit Testing in Visual Studio Team System

Sami Vaaraniemi — Mon, 07 Jun 2004 14:22:00 GMT

Oops. My previous comment was actually meant for Ken's thread...

Why unit testing in Visual Studio Team System?

James Newkirk's Blog — Tue, 08 Jun 2004 01:29:00 GMT

Why unit testing in Visual Studio Team System?

James Newkirk's Blog — Tue, 08 Jun 2004 01:30:00 GMT

re: Why Unit Testing in Visual Studio Team System

Brad Wilson — Mon, 07 Jun 2004 22:40:00 GMT

My answer is: who cares? Use NUnit!

I have to say that I agree that the post above just reeks of marketing. There's no good reason, other than "we don't wanna". It's a fundamental mis-understanding of what UNIT testing means. QA people don't write unit test; hell, they don't even RUN unit tests. Developers write them, developers (and continuous integration servers) run them.

I don't think they would've made this mistake had they consulted Jim Newkirk. But such is life... he's not on the developement tools team.

re: Why Unit Testing in Visual Studio Team System

Ron Green — Mon, 07 Jun 2004 23:17:00 GMT

Brad Wilson's cooment is exactly the point I was trying to address with the question. Unit testing is something devs do not QA. and if it's something devs do then it should be incorporated in all versions of Visual Studio.

re: Why Unit Testing in Visual Studio Team System

Robert Hurlbut — Tue, 08 Jun 2004 00:57:00 GMT

Well said, Brad. This is not "Unit Testing" as developers understand it. I write unit tests as a developer, and QA writes test cases to make sure an application meets the business requirements. These are not the same thing!

re: Why Unit Testing in Visual Studio Team System

Jim Argeropoulos — Tue, 08 Jun 2004 13:26:00 GMT

This is off topic of the rest of the comments, but...

Here here to the idea of unit testing and code coverage working together

Unit Testing in Visual Studio 2005

Ron Green's Weblog — Sun, 13 Jun 2004 21:10:00 GMT

To Unit Test or Not to Unit Test... which Visual Studio Version?

Being Scott Densmore — Mon, 14 Jun 2004 03:41:00 GMT

To Unit Test or Not to Unit Test... which Visual Studio Version?

Being Scott Densmore — Mon, 14 Jun 2004 03:42:00 GMT

re: Why Unit Testing in Visual Studio Team System

Jim Sather — Mon, 14 Jun 2004 19:17:00 GMT

It's obvious that passions run deep around who should be doing what and for what purpose. One of the fundamental goals of VSTS is to allow a wide range of flexibility and to allow choice. Don't mistake this to mean unstructured. VSTS allows you to tailor the structure and the process to meet your needs. Microsoft could certainly choose to ship unit testing in the developer SKU only, but that seems pretty restrictive and it violates the goal of flexibility and choice.

Absolutely, developers are the ones that quite often author and execute unit tests. Unit tests are often checkin requirements, regardless of whether you subscribe to XP/TDD/Agile or some other methodology. It’s a requirement in my own Product Unit.

Consider though, what happens when the development process is over. What happens when the component is handed off, when development staffing might not exits, and/or the component is in a sustaining mode. Consider if patches or small incremental changes need to be validated. A new platform, browser, OS update, server component, or if any dependency needs to be validated. It is not always the case that the job of executing those tests goes back to the original developer or development team.

Exhaustive testing should be able to include unit tests too, not just tester tests. Sometimes it's a business requirement for auditing reasons. When thinking of test prioritization, in some cases the highest yield tests to validate change, will be unit tests.

Finally, there are some very technical testers out there, and even some not so technical testers, that can use and author within a unit testing framework. One of the great things about full integration and making unit testing easy, is that it lowers the barrier to entry. It's more approachable by non-developers. A combination of white box and black box techniques is a solid approach for any Quality Assurance team.

I hope this helps explain why unit testing is currently slated for both VSTS Developer and VSTS Test.

-Jim

Branching, Merging and Unit Testing

Rob Caron's Blog — Tue, 15 Jun 2004 00:25:00 GMT

re: Why Unit Testing in Visual Studio Team System

Jeff — Mon, 14 Jun 2004 22:40:00 GMT

This whole question is stupid. If Unit Testing has enough value to stand as a feature across all versions of Visual Studio, and so does Profiling, Static Analysis, etc, then they ALL belong in ALL versions. Doing without them here in the goddamn 21st century is like doing without syntax highlighting.

Why unit testing in Visual Studio Team System - Again

James Newkirk's Blog — Tue, 15 Jun 2004 11:12:00 GMT

re: Why Unit Testing in Visual Studio Team System

buk — Tue, 15 Jun 2004 14:42:00 GMT

"Today the placement of Unit Testing support in Team System is more a function of integration and less of a statement about team size."

I would then expect pricing options to reflect that its more a function of integration and less about team size. Amortizing the licensing fees for the server portions over 5 developers will not work. When you have a team of 50+ it makes sense. How will the Team System licensing work for small teams?

re: Why Unit Testing in Visual Studio Team System

Ron Jeffries — Wed, 07 Jul 2004 11:25:00 GMT

I got a demo of the unit testing stuff in VS, at the Agile Development Conference last week. Almost good.

Now I hear that it is only released as part of the Team System. This concerns me a lot. NUnit, on which the stuff I saw is clearly patterned, is a developer tool, a pair tool if you're pair programming. It is used by the developer to write tests in test-driven style, as he goes.

It is in the test-driven style that the real value comes out. We in the agile and XP communities have spent ages getting people to realize this. If you look at Java, which like it or not is the big competitor to C# and .NET, //all// the major IDEs have jUnit integrated at the developer level.

If you look at VS.NET today, NUnit is a plug in that can be hooked in by any developer anywhere.

Sure, unit testing and all those things belong at the team level /as well/. But test-driven development, the real reason why the xUnit programs exist, is a one-person-one-machine (or two-person if you're pairing) activity.

How do I get on your Beta list, or whatever it would take to get some ears at Microsoft? Or is it too late?

Providing Timely Responses

scooblog by josh ledgard — Sat, 17 Jul 2004 01:53:00 GMT

re: Why Unit Testing in Visual Studio Team System

Kevin — Wed, 21 Jul 2004 20:48:00 GMT

I agree with the first poster. My QA team can't write code and has to work very closely with developers to write functional tests that matter. Team Edition will only go on a developers desktop. This is simply marketing!!!

re: Why Unit Testing in Visual Studio Team System

Jason Anderson — Thu, 22 Jul 2004 01:47:00 GMT

Hi Kevin,

Please note that the Test features in Team System are not simply about Unit Testing. Team System also includes Manual Testing and Web UI Testing - both of which won't require anyone on your team to write code. Check out the videos on VSTS posted on http://channel9.msdn.com to get a closer look at Test specific features beyond Unit Testing.

Thanks,

Jason

The Next Step? Parameterized Unit Tests

Alan Dean — Sun, 22 May 2005 13:58:49 GMT

The Next Step? Parameterized Unit Tests

MbUnit vs. NUnit Vs. Team System Unit Testing - Choosing a unit test framework

ISerializable - Roy Osherove's Blog — Sun, 24 Sep 2006 03:51:42 GMT

MbUnit is gaining some momentum as one of the nicest unit testing frameworks for .NET out there. I just

MbUnit vs. NUnit Vs. Team System Unit Testing - Choosing a unit test framework

ISerializable - Roy Osherove's Blog — Sat, 22 Sep 2007 19:16:06 GMT

Update : There's a new Unit test framework called XUnit.NET. More info here. MbUnit is gaining some momentum

Free Download Mp3 Song » Jason Anderson-The Wreath

Free Download Mp3 Song » Jason Anderson-The Wreath — Wed, 07 Nov 2007 14:42:28 GMT

PingBack from http://buyonlinemp3.myweblog.com/2007/11/07/jason-anderson-the-wreath/

Download Music Online » Blog Archive » Jason Anderson-The Wreath

Download Music Online » Blog Archive » Jason Anderson-The Wreath — Thu, 22 Nov 2007 00:35:05 GMT

PingBack from http://monstablogs.info/downloadmusiconline/2007/11/21/jason-anderson-the-wreath/

Jason Anderson s WebLog Why Unit Testing in Visual Studio Team System | Shed Kits

Wed, 27 May 2009 06:39:11 GMT

PingBack from http://backyardshed.info/story.php?title=jason-anderson-s-weblog-why-unit-testing-in-visual-studio-team-system

Jason Anderson s WebLog Why Unit Testing in Visual Studio Team System | Wood TV Stand

Mon, 01 Jun 2009 02:14:34 GMT

PingBack from http://woodtvstand.info/story.php?id=9217

Jason Anderson s WebLog Why Unit Testing in Visual Studio Team System | Quick Diets

Sat, 13 Jun 2009 17:21:14 GMT

PingBack from http://quickdietsite.info/story.php?id=2984