Jason Anderson's WebLog

Static Analysis Question and Answer

Jason_Anderson — Sat, 18 Sep 2004 02:00:00 GMT

Having trouble with the feedback section on my blog... Posting a cusotmer question and the answer...

Question From Karl:

re: Visual Studio Team System: Overview of Static Analysis
Nice presentation. FXCop is a great tool, having such solid IDE integration is great.

Q. I take it excludes/comments are part of the project? so when someone does a "get latest" he or she will also see the exclude message? or is it developer/machine-specific

Answer from Sean Sandys, GURU of all things concerning Static Analysis:

A. We are moving to an in source suppression technique to manage exclusions. In this strategy, a developer will be able to add an SuppressMessage attribute to their source. This means that the exclusions will live with the source that generated the warning.

Thanks,

Sean.

Q&A on Unit Testing in VSTS

Jason_Anderson — Thu, 16 Sep 2004 12:51:00 GMT

This morning, I wanted to address some of the questions and concerns I've received on some of the Unit Testing features in Visual Studio Team System 2005.

Thank You for Your Feedback

The response to VSTS has been amazing. The questions coming in the form of blogs, newsgroup posts, email questions, etc are very insightful and much appreciated. Thanks for taking the time to voice your concerns and help us build the best product possible.

One of the largest contributors to the set of questions I address here is jarnold in his blog: http://weblogs.asp.net/jarnold/archive/2004/09/15/229893.aspx

Question: Why bother with unit testing in VSTS?
Answer: On one level, Visual Studio Team System is about expanding Visual Studio so that roles beyond the developer and functions beyond standard code development can be addressed. One of many roles we added support for is the Tester and one of the many roles we expanded functionality for is the Developer. Adding built-in support for Testers and treating software testing as a first class citizen and expanding Developer functionality to include unit testing are great things for the software engineering community. The overwhelming customer response we’ve received, and one we agree with whole-heartily, is ‘it’s about time’ these features made their way into Microsoft products.

Question: What about other tools?
Answer: By adding Unit Testing, Web Testing, Load Testing, Code Analysis, Profiling, Manual Testing, Centralized Reporting, etc, etc, etc into Visual Studio Team System we’re trying to give many roles involved in the product lifecycle a great set of tools. This is not to say we believe Microsoft can and will create every tool you’ll need. It’s clear that our customers want a diverse set of functionality and the tools any one vendor can release will only be part of the story. This is why Visual Studio has always had an extensibility story and it’s why we expanded our extensibility story in Visual Studio Team System. Adding unit testing into VSTS was not a statement about the validity or relevance of any other unit testing tool in the community. Adding unit testing into VSTS was about empowering our customers.

Question: Isn’t NUnit cool?
Answer: Yes. We think NUnit is a great tool and we believe NUnit has a place in the community. Additionally, we’re not trying to stop NUnit or any other tool from plugging into Visual Studio or Visual Studio Team System. You should use the tools that provide you the most value. We hope our tools integrated into Visual Studio are compelling. If something else works for you, and you use VS or VSTS for some other support, we’ll do are best to provide a platform so 3rd party tools can target integration into Visual Studio Team System and take advantage of our tools ecosystem.

Question: Is Microsoft going to provide some guidance around unit testing and unit testing features?
Answer: Yes. We have our MSF guidance, and we’re adding Test and Development specific sections to MSF along with help and other forms of guidance around our features. Check out: http://msdn.microsoft.com/library/default.asp?url=/library/en-us/dnvsent/html/vsts-msf.asp

Question: Does the Unit Test Generation Dialog assume a One-to-One Mapping between Development Code and Unit Test Code?
Answer: The Unit Test Generation dialog lets you decide which elements of your code you want to generate tests for. Yes, you can get very specific and generate a single test per code element. That said, I’d stop short of saying the tool ‘assumes that there is a one-to-one mapping’ and instead say the tool allows you to generate tests at a one-to-one mapping granularity. The Unit Test Generation feature isn’t trying to force the way you scope your unit tests. The Unit Test Generation feature is trying to support as much functionality as it can while still be useful for most users. In that pursuit, the feature enables customers to get very granular about code element selection.

Question: Does the Unit Test Generation dialog promote a test-last culture?
Answer: The Unit Test Generation dialog helps you if you have an existing code base and you want to generate a set of unit tests quickly. The Unit Test Generation feature is not about promoting one methodology over another (Test-First vs. Test-Last). Instead, the feature is intended to help customers that have an existing code base and want to generate a set of tests quickly. Those customers that prefer ‘Test-First’ may not use the Unit Test Generation feature. Those customers that prefer ‘Test-Last’ or some how end up with an existing code base and need to use a ‘Test Last’ strategy will have this feature to use.

Question: Is Microsoft making a statement about the way I should test (features vs. implementation)?
Answer: We try to give our customers a useful set of features without dictating a process. Each customer needs to pick the right methodology \ approach to development and test that makes sense.

Question: Does coupling your tests to private implementation hinder or facilitate refactoring?
Answer: My general rule of thumb is that the more dependencies a piece of code has, the higher the cost of refactoring (or at least the higher the percentage chance of hitting issues). As a result, if a customer chooses to test a private method, there most likely will be some overhead in terms of refactoring. In some cases, this overhead may not be worth it. In other cases, the customer may want to pay the cost because the gain is significant enough.

Question: Isn’t offering a Unit Test Code Generation feature based off of existing development code a step backward in terms of Test Driven Development?
Answer: I think many customers face a mixed environment in terms of ‘Test-First’ vs. ‘Test-Last’. In certain cases, Test-First simply doesn’t make sense (an existing code base without unit tests). In other cases, some customers may simply prefer ‘Test-Last’ to ‘Test-First’. The Unit Test Code Generation feature is certainly helpful when Test-Last is used. However, I don’t think having the technology is a step-back in terms of the TDD community.

Question: The refactoring support in VS isn’t good enough… Will it get better?
Answer: We are very aware of the other products on the market that do more in terms of refactoring and code generation from test code to development code. This is something we are planning on improving and is certainly an evolutionary process. In that vein, customers have indicated to us that Generate Method Stub is a great beginning.

Question: The default names in the Unit Test Code Generation dialog don’t work for me… How can I change them?
Answer: The Unit Test Code Generation dialog allows you to enter specific names for specific tests. It also allows you to make use of little helpers like ‘[Class]’ and ‘[Method]’ to reference the Class of the development code or the Method under test.

Overview of Static Analysis in VSTS

Jason_Anderson — Sun, 05 Sep 2004 20:11:00 GMT

After some subtle (and others not so subtle) feedback, I've moved the larger post 'Visual Studio Team System: Overview of Static Analysis' off of the blogs.msdn.com main page.

If you are interested in development tools in Visual Studio Team System or a fan of Static Analysis, it might be worth the download time (The screenshots are 800x600 and are in jpg format…)

Original Post: http://blogs.msdn.com/jason_anderson/archive/2004/09/05/225798.aspx

Here’s an excerpt:

...

If you poke around MSDN and search for terms like Static Analysis, FXCop, PreFast, Code Analysis, etc, you’ll find articles about some of the early versions of Static Analysis tools from Microsoft. In fact, in various forms, Static Analysis tools have been available for a while from Microsoft SDKs, downloads, etc. With Visual Studio Team System, we are finally bringing Static Analysis to you as a fully integrated feature set in Visual Studio Team Developer. This is great stuff, check it out:

The Standard Development Process

Generally speaking, taking your code from source to binary is pretty straightforward:

[[Figure 1: Standard process of building code]]

We start with some source, we compile it, the compiler does preprocessing - checks the grammar - raises errors if it finds any and if it does not - we get a binary.

If we do get compiler errors, Visual Studio (and development tools) helps us find the error in our code and we figure out what went wrong and usually fix it in short order.

Getting a compiler error in our development environment is one of the cheapest types of errors we get in the software development process. This is because we fix the issue without getting others involved and without incurring much overhead in terms of process. Other types of bugs typically require at least one other person, and thus are more expensive. Generally speaking, the more people and process involved in finding and fixing a bug, the more expensive the issue is. Here’s a basic example:

Case 1: Developer writes code, gets a compiler error, finds the issue with the help of development tools, fixes the issue, and recompiles.

Case 2: Developer writes code, gets through compilation, and gets the new code into the daily build of the product. Tester gets the daily build of the product, writes a test against the new code, and finds a bug. Tester writes a bug report, lists the repro steps to find the bug. Developer gets the bug, uses the repro steps to see the problem, investigates the issue, finds the problem, fixes the code and checks the fix back into the daily build. The Tester gets the daily build of the product, verifies the fix and finally closes the bug.

Case 3: Product ships to customers and customers install the product. Product is used and a bug surfaces that some how made it through testing. Many customers call product support and product support specialists try to walk customers through working around the issue. Because the software is flawed (not a problem with configuration, etc) product support logs a bug about the product. Meanwhile, many more customers continue calling and having the problem with the software. In some cases, spending hours on the phone or in email communicating; only to eventually find out the issue is simply a bug. Eventually, a developer on the product team gets the bug report and investigates the issue. The issue is eventually resolved and a new build of the product is done that contains the bug fix. Now testing has to verify that the fix hasn’t caused more problems and has to do a complete testing pass on the product to make sure something unforeseen hasn’t occurred while applying the bug fix. Once the fix is approved, the product update has to be made available to all customers and all customers have to down load the fix. Each time the product is purchased, the new customer still gets a build of the product with the bug until they download the patch (or take advantage of an update mechanism).

Of course, the three sample cases above can get much more complex and complete, use other methodologies and be much more or less expensive… but I think these illustrate the point: The further from the desktop development process we get, the more expensive the process of making quality code.

The utopian case is clear: Write error free code. However, this isn’t very practical. So beyond perfectly written code, if we can get to a point where we detect bugs earlier in the software development lifecycle, we save time and money. Finding bugs earlier in the development process is what Static Analysis is all about.

...

Here's the link: http://blogs.msdn.com/jason_anderson/archive/2004/09/05/225798.aspx

Videos of VSTS on Channel 9!!!

Jason_Anderson — Tue, 20 Jul 2004 18:14:00 GMT

Check out the late night talk video taped on MSDN: http://channel9.msdn.com/default.aspx...

Tom Arnold and I gave this presentation totally unplanned in the middle night for a group of curious customers. All the video parts are now posted, there are 4 total and we walk through many of the Developer and Test tools... Enjoy!

Thanks to Robert Scoble for pulling the Videos together and getting this thing posted!!!

VSTS on Channel 9!

Jason_Anderson — Thu, 15 Jul 2004 22:22:00 GMT

Check out the late night talk video taped on MSDN: http://channel9.msdn.com/default.aspx...

Thanks to Robert Scoble for pulling the Video together and getting this thing posted.

Late Night Talk on Visual Studio Team System

Jason_Anderson — Wed, 23 Jun 2004 02:17:00 GMT

Check Out Steve Maine's blog about a late night talk from Microsoft campus on Visual Studio Team System:

http://hyperthink.net/blog/default,date,2004-06-22.aspx

Chasing the Holy Grail

Jason_Anderson — Fri, 04 Jun 2004 17:14:00 GMT

RichB posted the following comment:

...

Now, if Microsoft were to produce a set of functionality which “profiled”; code and allowed a functional tester to see a straight percentage of how much of the codebase they had exercised during a black-box QA session and linked this with test scripts (English, verbal, scripts) and a bug database - it would be much more useful to most of the QA teams out there. Imagine setting a “profile”; flag in an ASP.Net website and then getting the QA team to test a build of the website. After the QA session ends, the profile gets attached to the test management application for that particular test run. The developer logs onto the test management application to see the results of the test run - and sees graphically which parts of the code got executed, which caused exceptions, which caused crashed and bugs that were entered. If Microsoft can deliver this functionality, then they have hit the holy grail.

Rich, thanks again for taking the time to post your comments… I really appreciate your feedback.

The scenario you mention above will be possible in Visual Studio Team System in two ways [YES! We can do this for you!]:

ASP.NET Profiling: As you mention you instrument the code used by the ASP.NET site, point testers or automated test scenarios at the site, and when the cycle is over collect performance data concerning how the ASP.NET application performed.

ASP.NET Code Coverage: Again, as you note above, you instrument the code used by the ASP.NET application, run your functional tests or have testers use the application, and when the cycle is over collect information specifically concerning exactly what code in your ASP.NET application was touched by your test activity.

The great thing is that this functionality, Profiling and Code Coverage, are both offered for all types of tests and all types of applications. Profiling enables developers to see exactly how much time a specific method took to execute along with many other performance related information. Code Coverage gives you percentages and roll ups of what was hit by a test or a scenario and actually can show you visually what code was touched in your application vs. missed. Check out the great article by my partner-in-crime Tom Arnold and Paul Schafer: http://msdn.microsoft.com/library/default.asp?url=/library/en-us/dnvsent/html/vsts-test.asp for an actual screen shot (though the example provided is not ASP.NET specific).

ASP.NET specific functionality was not included in the bits that we made available at Tech Ed... Code Coverage and Profiling work - but not with ASP.NET at this point. This ASP.NET specific functionality will be there for Final Release.

Visual Studio Team System Test Support: More than just Unit Testing

Jason_Anderson — Fri, 04 Jun 2004 16:57:00 GMT

Feedback from RichB noted the following:

-----------------------------------------------

...

Now, as most testers in the wider world are not developers they won't have a copy of Visual Studio on their machine and they also will not know what a code editor looks like. If they don't know what a code editor looks like, they will not need unit testing tools. This is where I believe Microsoft is going wrong.

There's a separate class of testers who are much smaller in number, but more closely resemble Microsoft's warped view of the general testing community. Those people are test harness testers and performance testers. They tend to know less about the business and more about code - and they definately do require tools which will enable them to reproducibly run tests against code (call that unit testing if you will, although it's not common usage).

Therefore, Visual Studio Team Tester targets a small percentage of the QA teams in the wide world.

...

-----------------------------------------------

Rich, thanks for the comment – you are officially the first viewer response I’ve gotten on my blog!!

I completely agree with your characterization of a segment of the testing community who are not necessarily users of Visual Studio today. As you note, these testers really don’t use pure ‘code editors’ and as a result don’t have much use for today’s Visual Studio and won’t have much use for a unit testing feature set.

One of the goals of Visual Studio Team System is to broaden the functionality of the Visual Studio product family. We’re introducing tools that address not just code specific tasks; we’re introducing functionality targeting the Project Manager, IT Architect, Developers, Testers, Functional QA Tester, and many other roles in the product lifecycle. With VSTS, we expect go beyond the Visual Studio's traditional role as a powerful code editor.

The Unit Testing support I mentioned in my earlier post is only one feature of our Team Tester and Team Developer products. As you correctly point out, our Unit Testing support is specifically targeted at developers and the segment of the test community you referred to as ‘test harness testers’ or sometimes called White Box Testers (people who know the code).

Take for example our Web Tests, these tests allow a user to navigate to a website and browse naturally. Meanwhile, our toolset records all gestures and saves these in the test for the user. The user does not need to write a single line of code. Of course, we allow the user to go into the code if desired, but this is not required. Using our Test Execution features, the user can run this test and literally get the earlier recorded functional scenario in an automated test environment.

Another example non-code specific behavior is in our Manual Tests, these tests enable the user to simply write manual steps for a scenario test using Word or a text editor. These Manual tests (as can all of our test types) can then be assigned to others via our Work Item functionality, integrated with bugs via our defect tracking functionality, or simply saved and used by the single scenario tester in the future.

Also, our partner Compuware has already integrated their functional test product Test Partner into Visual Studio Team System.

Visual Studio Team System really is targeting the wide range of users and roles traditionally found in the product lifecycle. We’ve added features that span both the deeply technical and deeply functional skill sets.

I’ll try to post another response to the rest of your comments later…

For more information, check out:

Our Team site: http://msdn.microsoft.com/vstudio/teamsystem/

Some of our Testing Demos: http://www.microsoft.com/downloads/details.aspx?FamilyId=88F7CB8B-473B-4ED5-BA47-ABBC06D0048E&displaylang=en

Check out the great article by my partner-in-crime Tom Arnold and Paul Schafer: http://msdn.microsoft.com/library/default.asp?url=/library/en-us/dnvsent/html/vsts-test.asp

Why Unit Testing in Visual Studio Team System

Jason_Anderson — Fri, 04 Jun 2004 07:55:00 GMT

My buddy James Newkirk (of NUnit fame) forwarded the following question to me from Ron Green:

Sender: Ron Green

=====================================

It appears from what I am reading that Unit testing is included as part of VSTS only. I don't understand why this wouldn't be included with Visual Studio itself. Unit testing really has nothing to do with the size of the development group. Unit testing is unit testing whether it's a group of 30 or a single developer.

Can you shed some light on this?

=====================================

First, thanks for the question Ron… I really appreciate you taking the time to write in and provide us with your feedback and insight.

Indeed, the current thinking is to offer the Unit Testing features in Visual Studio Team Developer and Visual Studio Team Tester which, as you correctly point out, are both parts of Visual Studio Team System. Today the placement of Unit Testing support in Team System is more a function of integration and less of a statement about team size.

Our view of testing revolves around the concept of Quality Assurance as a first class and productive member of the product lifecycle. To truly bring the entire value of Quality tools to our customer and to really provide the most value to the product lifecycle in general, we have to go beyond making great test tools. We have to make great test tools that are integrated with each other and integrated with other Team System tools that are fundamental parts of the product lifecycle family. For example, we see a lot of value with the integration of Unit Testing and Code Coverage. The shared scenario of being able to write and execute your Unit Tests integrated with the ability to examine Code Coverage data concerning the effectiveness of those Unit Tests is remarkable. Additionally, we see a huge value in the integration of Unit Testing with Load Testing. Adding Unit Tests with other tests types (such as Web Tests) to a Load scenario and turning the dials to make a custom load environment gives developers and testers an extreme amount of testing power.

I agree with you that Unit Testing has enough value to stand as a feature across all versions of Visual Studio. However, several of our tools in Team System fit into that category: Profiling, Static Analysis, etc. Our approach wasn’t really to ask, “Is this tool valuable enough to help all developers?”… We build most of our tools with that target in mind. Instead, we tried to ask the question, “Where can this tool provide the most value?” and in our current thinking the answer is along the lines of “as a feature integrated with the rest of the product lifecycle tools found in Team System.” Over time, we may revisit this decision, but for the 2005 version, we feel confident that this is the right choice.

Ron… thanks for the great question!

Introduction

Jason_Anderson — Thu, 03 Jun 2004 07:07:00 GMT

I’m Jason Anderson, a Microsoft Program Manager on the Enterprise Development and Test (EDT) team; if you don’t know our team… you may recognize our product Visual Studio 2005 Team System. EDT will ship a wide range of tools including static analysis, profiling, and most things related to test. We just announced our features at Tech Ed San Diego, so if you didn’t have a chance to make it to the conference, be sure to check out our site or the various articles and blogs concerning Team System around the web.