In earlier posts I've written about some of the VSTS features around code quality from a testing perspective. This time around I wanted to go into improving code quality from a development standpoint… specifically, Static Analysis.
If you poke around MSDN and search for terms like Static Analysis, FXCop, PreFast, Code Analysis, etc, you’ll find articles about some of the early versions of Static Analysis tools from Microsoft. In fact, in various forms, Static Analysis tools have been available for a while from Microsoft SDKs, downloads, etc. With Visual Studio Team System, we are finally bringing Static Analysis to you as a fully integrated feature set in Visual Studio Team Developer. This is great stuff, check it out:
The Standard Development Process
Generally speaking, taking your code from source to binary is pretty straightforward:
Figure 1: Standard process of building code
We start with some source, we compile it, the compiler does preprocessing - checks the grammar - raises errors if it finds any and if it does not - we get a binary.
If we do get compiler errors, Visual Studio (and development tools) helps us find the error in our code and we figure out what went wrong and usually fix it in short order.
Getting a compiler error in our development environment is one of the cheapest types of errors we get in the software development process. This is because we fix the issue without getting others involved and without incurring much overhead in terms of process. Other types of bugs typically require at least one other person, and thus are more expensive. Generally speaking, the more people and process involved in finding and fixing a bug, the more expensive the issue is. Here’s a basic example:
Case 1: Developer writes code, gets a compiler error, finds the issue with the help of development tools, fixes the issue, and recompiles.
Case 2: Developer writes code, gets through compilation, and gets the new code into the daily build of the product. Tester gets the daily build of the product, writes a test against the new code, and finds a bug. Tester writes a bug report, lists the repro steps to find the bug. Developer gets the bug, uses the repro steps to see the problem, investigates the issue, finds the problem, fixes the code and checks the fix back into the daily build. The Tester gets the daily build of the product, verifies the fix and finally closes the bug.
Case 3: Product ships to customers and customers install the product. Product is used and a bug surfaces that some how made it through testing. Many customers call product support and product support specialists try to walk customers through working around the issue. Because the software is flawed (not a problem with configuration, etc) product support logs a bug about the product. Meanwhile, many more customers continue calling and having the problem with the software. In some cases, spending hours on the phone or in email communicating; only to eventually find out the issue is simply a bug. Eventually, a developer on the product team gets the bug report and investigates the issue. The issue is eventually resolved and a new build of the product is done that contains the bug fix. Now testing has to verify that the fix hasn’t caused more problems and has to do a complete testing pass on the product to make sure something unforeseen hasn’t occurred while applying the bug fix. Once the fix is approved, the product update has to be made available to all customers and all customers have to down load the fix. Each time the product is purchased, the new customer still gets a build of the product with the bug until they download the patch (or take advantage of an update mechanism).
Of course, the three sample cases above can get much more complex and complete, use other methodologies and be much more or less expensive… but I think these illustrate the point: The further from the desktop development process we get, the more expensive the process of making quality code.
The utopian case is clear: Write error free code. However, this isn’t very practical. So beyond perfectly written code, if we can get to a point where we detect bugs earlier in the software development lifecycle, we save time and money. Finding bugs earlier in the development process is what Static Analysis is all about.
The Development Process with Static Analysis
Adding Static Analysis to the development process is very simple: Once you compile your code, you use Static Analysis technology to do a more extensive set of checks than what the normal compiler can do:Figure 2: Standard process of building code with Static Analysis
Static Analysis analyzes your code, checks your binary against an extensible set of rules, and raises errors or warnings if any violations are found. In other words, using Static Analysis is like having a built-in code review that helps you analyze your code - checking for everything from style to code correctness to security issues.
Let’s take a look at a trivial code case to see a tiny example of how Static Analysis can help us:
This code takes a string that has previously been checked to ensure it is not null… The first thing this method does is to check to see if the argument is an empty string. If the string is an empty string, the code knows the User Name is not valid and returns false. If the string is not empty, the code calls another method ‘CheckUserAccount’ in a try catch clause. If CheckUserAccount does not throw an exception this method returns true. If CheckUserAccount does throw an exception, the code catches the exception and returns false.
This code compiles with 0 warnings and 0 errors.
However, using Static Analysis technology, we would get several suggestions of issues to check into. Of course, what suggestions we get depends on what we tell Static Analysis to check against (we’ll explore this in more detail below). Here’s a couple of examples of issues Static Analysis could find with this code:
In the first case, the Static Analysis rule is pointing out that I’m violating a performance rule by checking to see if strUserName is equal to “” rather than just checking to see if strUserName.Length == 0. Changing my code in the way that the Static Analysis rule suggests would accomplish the same functionality, but would help my program’s performance.
In the second case, the Static Analysis rule is warning me about the dangers of catching a generic exception rather than catching a specific exception. In some cases, generally catching all exceptions might be what we want… But in general, it’s a dangerous practice that could hide every exception happening as a result of the call.
Enabling Static Analysis in Visual Studio Team System
Using Static Analysis in VSTS is very easy as Static Analysis is treated as a add-on to the normal development process and is fully integrated with Visual Studio.
Here’s a screen-by-screen example of enabling Static Analysis:
First, we start with the code in the editor…
Next, we build the solution to see what the standard compiler settings are going to catch with this code:
We see the standard compiler settings do not find any issues with this code:Figure 6
Now we want to turn on Static Analysis, so we go to the settings of the project. We can do this in a couple of different ways. We could go to the Solution Explorer and right click on the project and select ‘Properties’. Or we could use the Project menu:Figure 7
Visual Studio now displays the settings of the project:Figure 8
The specific set of settings we’re interested in are those listed on the FxCop tab:
FxCop is the name of Microsoft’s static analysis (a.k.a. Code Analysis) tool for managed code. PREfast is the name of Microsoft’s static analysis tool for unmanaged code.
To enable Static Analysis, all we have to do is check the Enable FxCop checkbox:
Now we save our settings (an extra step I’m doing just to be safe):
Now, we go back to our code:
We follow the same steps to once again build our code:
This time both compilation and Static Analysis occurs:
As you can see, Static Analysis has found the issues described in our original example.
Using Static Analysis, More Detail
Enabling More Rule Sets, Rules, and Changing the Rule Status between Warning and Error
Let’s check out the Static Analysis rules from Figure 10 (to get back to the Static Analysis Rules screen, you can use the Project Menu, click on [Your Project] Properties, and then click the FXCop Tab].
In addition to turning on and off Static Analysis, you can turn on and off whole rule sets. For example, to enable the Static Analysis engine to check your code against Security Rules, just enable the check box next to Security Rules.
In addition to enabling entire rule sets, the Static Analysis User Interface helps us to get very specific about the rules we want to check our code against. Notice the difference between the Security Rules checkbox vs. the checkboxes for Design Rules and Performance Rules (in Figure 15). The solid check box next to Security Rules means that all Security Rules are enabled, while the grey checkbox denotes that some Rules are selected while others are not selected under Design Rules and Performance Rules. To see the specific Rules of any Rule Set, we just click the [+] next to the Rule Set:
Figure 16: A tiny subset of the Security Rules that ship with the Static Analysis technology in Visual Studio Team System
To disable a rule, we just uncheck its corresponding checkbox.Figure 17
Notice that the Security Rules checkbox is now grey, to indicate that some rules under Security Rules are not enabled.
Beyond enabling and disabling Rules and Rule Sets, we can select if we want the issues found by the Static Analysis engine to be reported as Errors or Warnings. Let’s check out the Security Rules once again:
As you can see, these Rules are going to indicate a Warning if a violation is found. To make a specific Rule report an error, we click the corresponding Warning label and icon:
Now the ‘Consider reviewing Deny or PermitOnly permission usage’ Rule will report an Error while all other Security Rules will still report Warnings. Also, notice that the User Interface notes that some rules report Warnings while others report Errors by noting the Security Rules Rule Status as (Mixed).
Just like we can enable and disable an entire Rule Set, we can also change an entire Rule Set’s Rule Status by clicking on the Rule Sets status (in Figure 19, Security Rules has a Status of (Mixed). If we take this action, it’s easy to change all Security Rules to issue Errors if found:
Which enables us to see a distinct difference between Security Rules violations than Performance Rules and Design Rules.Figure 21
Let's close the Project Settings and get back to the code editor and Error List dialog.
Looking at Figure 14, we can see the two issues found in the sample code. What if we only really cared about one of the issues? Do we have to go back into the project settings and find the rule to disable this warning? No. We can simply exclude the Rule.
To exclude the Rule we right-click on the rule violation and click ‘Exclude warning’:
Now when we rebuild our project, Static Analysis will only report the rule violation we did not exclude.
Excluding instances of Rules gives us a per project customization that allows us to get even more specific about the Rules violations we care about and cut down on the ‘noise’ factor.
A Couple More Static Analysis Points
Q. Can Static Analysis be extended?
A. Yes. FXCop has an SDK that has been available in the past and a new version is coming that will let you extend the rule set in Visual Studio Team System
Q. Can I make Static Analysis run as part of my product’s daily build system?
A. Yes. Visual Studio Team System includes a Build component for building an entire product that spans multiple projects and solutions. This Build component enables Static Analysis and Tests to be add executed as part of the build process.
Q. Can I force developers in my organization to run Static Analysis?
A. To some reasonable degree… <smile>. Visual Studio Team System enables a team to declare Policy across all solutions and projects. Policy rules can include running Static Analysis before checking code in and \ or running a specific set of tests before a check in. The developer is still able to override this policy; however, the system records the policy override and will report policy override.
Q. Can Static Analysis work with unmanaged code?
A. Yes. The FXCop technology is for managed code while the PREFast technology is for unmanaged code. You get Static Analysis in both cases.
Q. Where can I get more information about Static Analysis?
A. Check out:
FXCop Team BLOG: http://weblogs.asp.net/fxcop/archive/2004/01/19/60319.aspx
Also search MSDN for FXCop and PREFast and you’ll see some articles on the older versions of these technologies.
Static Analysis is a powerful tool for developers and is now included in Visual Studio Team System in the VS Team Developer box. Static Analysis rocks because: