Visual C++ Team Blog : Test

How to redistribute the Visual C++ Libraries with your application

vcblog — Sat, 13 Oct 2007 00:33:00 GMT

Hello again, this is Ben Anderson, SDET on the Visual C++ libraries team.

One of the most common questions we get from customers on the forums and elsewhere is “My app needs the Visual C++ Libraries (CRT, ATL, MFC, OpenMP or some combination thereof) – how do I get them on my customers’ machines?” It’s also something we see in the wild done, if not incorrectly, at least non-optimally fairly frequently. The help documentation in MSDN is correct, but there is no one stop-shopping explanation of all your options. This blog post will attempt to explain what to do. (In case you’re looking for the short answer, almost always, the correct thing to do to distribute the Visual C++ libraries is to add the Visual C++ redistributable MSMs, or “Merge Modules”, for the libraries you use to your application’s setup.) I’ve tried to outline below the various methods of redistributing the Visual C++ library DLLs based on what your deployment story may be.

In most cases, folks deploy their applications using a standard Windows setup. In these cases, you probably build an .msi file using some toolset (such as a Visual Studio setup project) which is then wrapped in an .exe file by your tool chain. End users run this .exe file and your application is installed. If you don’t already have a setup for your application, it’s very easy to create one using Visual Studio’s setup project (found by right clicking your solution, clicking Add -> New Project… -> Other Project Types -> Setup Project). You can then right click your setup project, click “Add->Project Output…”, then select Primary Output. You can then add Start Menu items and tweak your setup to meet your needs.

In order to redistribute the Visual C++ libraries, all you need to do is include the appropriate .MSM file and its accompanying policy .MSM to distribute the library you need. If you are creating a setup project as part of your solution as described above, Visual Studio will attempt to detect which libraries you depend on and will add MSMs as appropriate. If you are creating your setup project with another tool, or not using the “Add project output” option, you will have to manually add the MSMs for any libraries you need. These libraries are found in “%ProgramFiles(x86)%\Common Files\Merge Modules”. For example, on my VS 2005 SP1 system, if I had an x86 MFC App, I would add the following files as Merge Modules to my setup project:

“C:\Program Files (x86)\Common Files\Merge Modules\Microsoft_VC80_CRT_x86.msm”
“C:\Program Files (x86)\Common Files\Merge Modules\Microsoft_VC80_MFC_x86.msm”
“C:\Program Files (x86)\Common Files\Merge Modules\policy_8_0_Microsoft_VC80_CRT_x86.msm”,
and
“C:\Program Files (x86)\Common Files\Merge Modules\policy_8_0_Microsoft_VC80_MFC_x86.msm”

These files are then consumed by your setup tool, and their contents are dropped as part of your MSI on your users’ systems. They contain components which install to Windows Side by Side the DLLs and the redirection policies [see footnote 1] for the libraries you select. These components are ref counted so that every time an app using these MSMs installs, the ref count is incremented, and every time one of these apps uninstalls it is decremented. Once the ref count hits zero, the DLLs and policy are uninstalled.

There are a few cases in which MSM installation may not work for you. In one case, you may have to deploy your app on systems where the user has no administrator privileges and so cannot run a setup. There may also be some other reason you cannot use an MSI to install your application – for instance, users may run your binaries directly from a network share.

In these cases, you can do an “app-local” deployment, which is sometimes called deploying the DLLs as “private assemblies”. All you need to do in this case is provide a copy of the DLLs you need, and their accompanying manifest in the same directory as every .exe, .dll or .ocx file in your application. To deploy in this way, simply copy the entire contents of the appropriate folder under <Visual Studio install dir>\VC\redist [see footnote 2] into the all folders which contain binaries which use those libraries.

The advantage of this approach is that you do not need to create an install for your application. This means you can deploy and run without requiring your users to elevate to administrator privileges. All your users need to do is copy your application folder onto their systems or run your .exe directly from its current location. The disadvantage is that you must put a separate copy of the libraries you need in every single directory in which your binaries reside. For a simple application, this may not be a problem, but for a large app which might have many subdirectories with many tools and DLLs, this is a lot of file duplication.

Finally, there is one additional scenario for redistributing the Visual C++ libraries DLLs. This scenario is if you are using “Click Once” deployment. In this case, “Click Once” will use a custom built installer package called “VCRedist_<arch>.exe” to install the libraries for you. DO NOT use the VCRedist_<arch>.exe installer packages for any other purpose.

The VCRedist packages are simply MSI’s built by consuming all the MSMs from “%ProgramFiles(x86)%\Common Files\Merge Modules” as well as the MSDIA DLL (used for debugging). However, MSIs are not ref counted like the components in the MSMs, so if you install it, you can never uninstall it because you do not know who else might be using it in addition to your app. Further, your users cannot uninstall it because they do not know which of their applications may be using it. Additionally, your users may not realize what it is when they see the entry in Add/Remove program files. Imagine a user trying to free up space on their machine, seeing the entry for VCRedist which they do not recognize, uninstalling it, then some time later (maybe months), trying your application again. It will not work! Your user will probably not connect the action of uninstalling VCRedist at some point in the past, and will either be broken without a fix, or use your support center’s time trying to find out why your app stopped working. What’s more, it’s very likely that you are not using every single Visual C++ library, and installing the whole of VCRedist is unnecessary. Alternately, a poorly written installer for another application which used VCRedist to redistribute the Visual C++ libraries may (incorrectly) uninstall VCRedist when that app uninstalls.

A better option if for some reason you cannot incorporate the MSMs into an MSI which installs your application is to use Visual Studio or another tool to build a tiny MSI installing just the MSMs, and only those that you require. Since this MSI is unique to your product, and can be named whatever you like, you can uninstall it when your application is removed, and you can name it in such a way that your user recognizes it as part of your application and will not uninstall it inappropriately (name it say “MyApp Prerequisites). By using your own MSI, you also guarantee that no other application which uses the VCRedist package will interfere with your app by incorrectly uninstalling it during that app’s uninstallation.

Again, just to emphasize – do not use VCRedist*.exe unless you are using Click Once to deploy your application.

In addition to all the methods described above of distributing the Visual C++ libraries DLLs, there is one last option for building your application which does not require you to distribute the DLLs. However, this option only works for native-only code (it is not supported with /clr) and leaves your customers seriously vulnerable to any security holes as well as adds a significant burden upon yourself to patch all customer systems should a vulnerability be found in any of the libraries. This option is to statically link in the libraries as .lib files instead of dynamically loading them as DLLs. You do this by using the /MT flag on the cl.exe command line (vs /MD), or selecting the appropriate option in your project properties through Visual Studio. You may wish to use this option when testing early debug builds of your application on test machines before you start working on setup. [See footnote 3]

However, I can think of no scenarios in which this is actually the right thing to do when shipping your product to customers. Basically, what this approach does is pulls in the binary code needed from .LIB files at compile time, making it a part of your .exe or .dll files. It increases the size of your application, and there is no way to update the libraries apart from recompiling your application with new .LIBs and redistributing your application all over again. What this means is that unless you go touch every single machine which has installed your application every time there is a security vulnerability found in the Visual C++ libraries and completely reinstall your updated binaries, you will be leaving your customers vulnerable to attack. If instead you use the DLLs, every time there is a security vulnerability found in the Visual C++ libraries, Microsoft will install the update centrally into the WinSxS folder via Windows Update and all requests for the DLLs will be redirected to the updated version. This removes all servicing burden on your side and also allows the user to install one small update which will touch all their applications instead of replacing every installed exe and DLL on their system. Please, do not distribute an application built by linking statically against the Visual C++ libraries unless you have a system in place for updating every customer machine and also have a very good reason to do so. At this time, I can think of no circumstance under which this would be the right thing to do for a shipping application.

Well, hopefully this article has helped you out in understanding how to redistribute the Visual C++ libraries onto your customer’s machines. If you have additional questions, you can find the documentation for deploying Visual C++ built applications here:

http://msdn2.microsoft.com/en-us/library/zebw5zk9(VS.80).aspx

If you still have questions, you can post comments here (I will check back for a few weeks), or you can post your question in the Visual C++ forums here:

http://forums.microsoft.com/MSDN/ShowForum.aspx?ForumID=29&SiteID=1

Thanks,
Ben Anderson
Visual C++ Libraries Team

[1] The redirection policy always redirects requests for the Visual C++ dlls to the latest installed versions, even if the application requesting the dlls has used “app local” deployment to drop the dlls as private assemblies – this way, if a security issue comes along, Windows Update can drop fixed dlls into Windows SxS and all affected applications will be fixed. The Visual C++ team maintains a strong binary compatibility guarantee that applications built against an earlier version of the library will work against all later versions with few exceptions (exploitable usage may be broken to prevent customer machines from being hacked).

[2] Please note that the files in this directory are not updated in QFE patches, and some of the manifest files in this directory were not updated as part of SP1 of Visual Studio 2005. As a workaround, you can find the appropriate version of the files in the WinSxS directory of your Visual Studio development box by typing “c:\windows\winsxs> dir *VC80*”, identifying the correct directory based on version numbers, then copying the contents of that directory into your application directories instead.

[3] A better option would be to create a simple setup project and include all the Visual C++ MSMs and install this on all your target machines.

What Does a VC++ QA Do Daily?

vcblog — Wed, 12 Sep 2007 18:52:00 GMT

Hello everyone, this is Smile Wei, an SDET (QA) in the VC Compiler team. This is not only my first blog on vcblog, but also my first ever blog.

As you might have guessed, I’m a new face around here, joining Microsoft only 3 ½ months ago. Here I’d like to give a brief introduction to the daily life as a VC QA and, to answer the following two questions that my friends kept asking me:

How do I like life at Microsoft, especially on the VC team?

What do I usually do every day?

For the first question, I love working at Microsoft, especially as a QA in the VC Compiler team. Usually people might think a QA may not be as technical as a dev, but this is not true in our team. As a QA in the VC compiler team, the test input are not simply numerical data or operations, in most cases they are C/C++ source code. When you write a test, you need to worry about not only how to write test processes with script languages like Perl, but also how to write test input code in C/C++ to capture both the correct behavior (positive test) and the wrong behavior (negative behavior) of the compiler. Same situation when you write regression tests. Creating a new test usually requires a lot of discussions with devs, because we need to make sure our tests will exactly reflect the new features or changes in the VC compiler parser, even some subtle ones.

Having source code as test inputs also increases the difficulty of bug investigations and reproduction. Because of the complexity of the compiler, some compiler bugs only fail under certain types of machine architectures (x86, amd64, or ia64), some bugs only fail under certain runtime environment (native, /clr, or /clr pure) on a particular architecture, while some bugs only fail with a particular cl.exe parameter (among hundreds of parameter combinations) under a particular runtime environment on a particular machine architecture. Usually you’ll feel that half the work is done once a bug is successfully identified and reproduced. That’s why a large portion of my time is spent on bug investigations.

Although this position is so challenging, I love it so much because I can learn so much about the VC compiler. All the details I learn from this position perfectly complement my previous high-level computer knowledge learned in college. Can you imagine how proud I am to easily find the bug in my friend’s code while it has harassed him for several days? That’s why I love this job. J

For the second question, my major responsibility includes the following tasks:

1. Bug investigations on test runs. Microsoft has daily test runs on every product with all existing test suites not only before the product is shipped, but also after. Any failures in the test runs will be investigated as the 1^st priority task and get resolved in the earliest possible version.

2. Test fixes and writing regressions. This is a follow-up task of the first one. Once a VC compiler dev fixes a bug, he’ll publish his private built to me for a more comprehensive test even after he has already performed a sniff testing on his own machine. This double assurance mechanism further reduces the risk to introduce new regression bugs by the bug fix. Some extremely crucial bug fix may involve many tests with substantial complexity. For example, to verify a recent fix, I ran a 4 X 3 X 3 X 2 X 2 X 2 test matrix, and each test run in the matrix involved creating and testing a large 23MB PCH. In Microsoft, a regression MUST be written to address the fix if no previous tests were targeting that source code. With more and more regression tests being added, a complex product like Visual Studio might have huge number of tests associated with it. For example, I own a test suite called the “compiler suite”, which contains about 20,000 test cases, this is only one of the 10 test suites of the VC++ compiler team. Please notice that these are only the tests for the VC++ compiler team, not including the tests for the VC++ library team, IDE team, backend team and release team, not to mention other teams in Visual Studio such as the VB and C# teams. The daily test runs on huge number of tests go some way to guaranteeing the quality of Microsoft products.

3. Verify QFE (Quick Fix Engineering) fixes. Yes, your understanding is correct, the QFEs are those hotfixes occasionally popped out and automatically installed on your machine. The usual QFE process is that the support team collets bug reports from our customers, finds a bug and passes it to the dev team. After the dev team fixes the bug, they alert the QA team so we can test it. The QA team will need to perform a two-round verification. The 1^st round is before sending the fix back to the customer who identified the bug, and the 2^nd round is after the customer verifies the bug fix, to perform a final verification before it’s officially published into hotfixes. Testing QFE fixes is always our 1^st priority tasks to solve customers’ problems in the 1^st place.

4. Provide cross-team test support. Occasionally I need to provide test support for other teams - to give a different test prospective for the other team. For example, I recently finished two projects falling into that category and did find some bugs. One project was to test the VS 2008 installation process for the release team, and the other one is to test VS 2008 MSDN samples for the library team. I believe this type of test may help find those bugs that might not have be found by the original test team because of their testers’ familiarity with the correct behaviors and processes of their product. This exactly matches the spirit of a well-known Chinese saying “You won’t have the overview of the mountain if you are already inside it”.

5. Develop internal test tools. As I mentioned in the first two responsibilities, Microsoft performs daily test runs on all of its products, and each test run may execute hundreds of thousands or even millions of tests. How does Microsoft achieve this? The trick is that the whole test execution, test result collection and comparison are fully automated with the support of a large set of test tools. Microsoft employees are encouraged to develop all kinds of internal test tools that can improve the test efficiency and some tools may be adopted and widely used in the whole division or even the whole company.

6. Write tests for new features. Currently the VS team has started working on the next delivery of Visual Studio (the version after VS 2008). A QA will be assigned to work out a whole set of tests targeting each new feature. The test process starts in a very early phase of the product cycle rather than after all the coding is done, which I believe sticks to the agile model instead of the traditional waterfall model. The reason we use an agile model is because the later a bug is found in the product cycle, the more cost will be incurred. Currently most of my responsibilities focus on the first five tasks and I believe in my next vcblog you may see more of my work on new features.

7. Other job responsibilities. This includes everything else on my job commitments or not on my commitments. For example, support campus recruiting, make training slides and videos for new employees, interviewing candidates, and yes, write vcblogs. J

Hope you are not bored by my long introduction. J If you happen to have some interest, comments, or suggestions, why not reply to this blog entry – I look forward to interacting with you? If you happen to have more interest on this challenging position, why not come join us? J

Really appreciate your patience and hope all of you and your families are happy and healthy. Thanks.

Smile Wei

VC Compiler Team

"Build On The Fly" Testing

vcblog — Mon, 10 Sep 2007 20:23:00 GMT

Hello. This is Pengpeng again. I wrote a blog not long ago talking about security testing for the Visual C++ IDE as a parser. The IDE is definitely a lot more than just a parser. Among other things, it provides an integrated software development experience with browsing, go to definition, go to declaration, quick info, auto completion … There is just so much you can do with the UI.

To guarantee great features with great quality, as testers, we have to be innovative and customer focused. This implies that we need to test a lot of end-to-end user scenarios. For instance, a user creates some new Visual C++ projects, defines new classes and functions, declares and instantiates new objects and makes function calls. During this process, for correctness, she/he has to reference the class/function definitions using go to definition, or class view, or quick info, or code definition window … Consequently, this makes the tests really complicated and calls for innovations in our testing infrastructures. Today, I am going to talk about one such innovation called “Build On The Fly (BOTF)”, which changes the way we automate our tests from building and deploying the test binaries all at one time to, as the name implies, on the fly.

A little bit of background on our current test execution architecture. It comprises a driver program and the test codes. The test codes for a testcase are: the test binary, a .dll file, and the test metadata given in an xml file. The metadata also provides the test execution information such as the entry point of a testcase in the .dll. When a test executes, the driver is called with the test xml file as the parameter. The driver reads the xml, find the .dll test binary specified in the xml file, drives the test according to the test data in the xml file, and log appropriately the test results.

Before the era of BOTF, the test binaries were generated daily and aggregated into several big msi’s using a dedicated build machine. During test automation, the msi’s were first copied to, and then installed, on the test machines. As a result, all the test .xml files and .dll files were generated in advance. This approach becomes really painfully: besides the maintenance of the build machine, often times, test issues are convoluted with build issues, and it comes really complicated to figure out what exactly went wrong during test failures. In addition, it brings a lot of overhead: any test issue fix has to wait for the daily build to be ready, and to be deployed, in order to verify.

BOTF solves the problem by building the test binaries on demand. First, in the test metadata, the xml files, the tests are labeled as BOTF. During the setup stages, the raw test codes, the .vcproj, .h and .cpp files are copied to the test machines. The test execution process is exactly as before: the driver is called with the test xml filename as the parameter. Underlying, now the driver will first build the needed test binary and then execute it. With this innovation, all the tests self contained: build or test issue can be easily isolated. All the test fixes are also “on the fly” and we do not need to wait for the build machine any more.

Thanks.

Pengpeng

Testing Compiler Diagnostics...and a Cool IDE Tip!

vcblog — Fri, 03 Aug 2007 23:28:00 GMT

Hello everyone, this is Jamie Eckman. The last time I posted I was an SDET (tester) on the Libraries team. I’ve since moved to the Compiler team, still as an SDET. One thing we see a lot of on both teams is compiler diagnostics: warnings and errors.

One of our Libraries developers, Stephan T. Lavavej, wrote this last year on one of our internal discussion lists: Figuring out the mapping from “what the compiler is complaining about” to “what’s actually wrong with my code, if anything” is like 75% of learning how to program. Depending on how you learned to program, 75% may or may not be an overstatement, but the point is clear. Everyone writes bad code sooner or later, and in the case of new programmers it’s sooner (and often). For this reason it’s important that the errors the compiler emits are correct and clear enough to diagnose the problem. Clarity is something we’re making a significant investment into improving over the next few releases of Visual Studio. For now I’ll mostly stick to talking about correctness.

When I was still on the Libraries team the only direct contact with diagnostics I had was some maintenance of the ATL Server attributes error tests. But for the most part I looked at compiler errors from the end-user perspective: This test I wrote doesn’t compile, why? Now that I’m on the Compiler team, which owns the bulk of the error text you’re likely to see while writing C or C++ code, I’ve had a great opportunity to look at up close how our diagnostics work and how we test them.

The compiler has on the order of 1,000 errors and warnings it can emit. We test these like you might expect, each warning or error has one or more test cases associated with it. The test cases verify that the diagnostic that’s expected is actually emitted and is emitted on the correct line. Of course, the compiler is a hugely complex mechanism and problems are bound to slip through the cracks.

A colleague on our UE team (User Education, responsible for the content on MSDN, among other things) recently came to me with a small list of code snippets from the C/C++ Build Errors section of MSDN that weren’t producing the promised errors. These snippets are pretty useful when you’re trying to figure out what a particular error or warning means. Oftentimes the simplified code in the snippet can help you spot the problem in your own code.

So, Tim from the UE team asked me to look at these snippets that weren’t working and figure out what was wrong. A compiler bug, an expected change we forgot to tell UE about, or something else? As an example, here’s the first one I looked at:

// C2062_b.cpp

// compile with: /c

class DataSent {};

struct MyDataController {

MyDataController(int age, DataSet* j) {} // C2062

// try the following line instead

// MyDataController(int age, DataSent* j) {}

};

This is a simple error, although perhaps not easy to spot. The class name “DataSent” has been mistyped “DataSet”. The expected error is C2062, “type ‘DataSet’ unexpected”. The error that actually gets emitted (with a recent Orcas compiler) is C2061, “syntax error : identifier ‘DataSet’”. Here we see that correctness and clarity often have an interesting relationship when it comes to error messages. Although we didn’t get the error we expected, the error we did get nonetheless points us directly at the problem. I might be able to make the case that this is a compiler bug, but because the new error is functionally just as good as the old error, it’s unlikely to be fixed at this stage in the Orcas release cycle.

Now, here’s a tip that has absolutely nothing to do with the rest of this post. Some of you may already know this, but I haven’t seen it posted on vcblog yet. I recently started working in a new codebase with a pretty strict set of coding conventions. One of the conventions is that nothing should go past column 79. It’s pretty annoying to watch the column display in the status bar every time you get close to the end of a line, but fortunately there’s a nifty hidden feature of Visual Studio that can help. Column guides. This involves editing the registry, so the standard disclaimer applies: Modify the registry at your own risk!

Find this key in your registry:

HKEY_CURRENT_USER\Software\Microsoft\VisualStudio\9.0\Text Editor

For Visual Studio 2005 you’ll want to use “8.0” instead of “9.0”. Create a new String Value in the Text Editor key and call it “Guides”. Modify the data for Guides and add something like this:

RGB(0, 128, 0) 79

This will put a green line at column 79. You can have multiple guides:

RGB(0, 128, 0) 79, RGB(128, 0, 0) 49

Make sure you restart Visual Studio after you add these values for the changes to take effect.

Enjoy!

- Jamie

Testing the IntelliSense engine

vcblog — Thu, 05 Jul 2007 19:01:00 GMT

Hi, I am James Wang, an SDET on the VC++ Compiler Front End team. Currently, I am working on designing the test architecture for the new IntelliSense engine. I am responsible to design tests that make sure the IntelliSense engine gives correct answer for quick info, member list, parameter help, and etc.

Currently we have a test suite that directs testing the IntelliSense feature. But one of the drawbacks is that it only targets a very limited amount of scenarios. Testing converge becomes a major issue. But due to vast varieties of C/C++ constructs, testing every scenario becomes an impossible or at least a very time consuming task. In the meantime, we do have ~100,000 of tests that targeting compiler / libraries that covers a majority of C/C++ constructs. Would it be better to adapt these tests for the purpose of IntelliSense engine testing?

To accomplish this, I wrote a tool that will analysis any arbitrary C/C++ code and generate IntelliSense test site. The tool will also generate the expected results for each test site. So for any arbitrary C/C++ code, we can generate a set of self verifying IntelliSense tests on quick info, member list, parameter help, and etc. As the result, we can reuse the ~100,000 tests for the purpose of IntelliSense testing.

Two things make this possible:

1. Better intermediate language (IL) representation. After parsing the source code, the new IntelliSense engine generate a better IL and expose a wealth of APIs that allows traversing and querying various compiler artifacts. This allows me to figure out where to create IntelliSense test site and what the expected results should be. For example: while traversing through the IL, if I see a field operator (->), I may want to generate a member list tests. For the expected results, I can query the type for the left operand of the field operator and retrieved the expected member list.

2. Componentized the IntelliSense engine. The new IntelliSense engine exposes API that allows engine level testing. So I can do IntelliSense testing without the IDE. Otherwise, running a large amount of IntelliSense tests become impractical due to the time needed to start / shut down the IDE for each tests.

Hopefully, this new approach will improve the test coverage for the new IntelliSense engine and drive up its quality.

Thanks,

James