Unit-Testing Native C++ with Visual Studio

re: Unit-Testing Native C++ with Visual Studio

John Socha-Leialoha - MSFT — Tue, 04 Jan 2011 17:28:29 GMT

@Joe. I don't agree. I was introduced to TDD about 5 years ago when I was writing Windows Mobile code. In that case, I would run the unit tests all the time on the desktop, and every now and then on the mobile device. Since Windows Mobile is a subset of Windows, there was no gaurantee that the behavior would be the same between the desktop and the device. However, even though there wasn't complete fidelity, our productivity gain was huge.

What I'm doing now, and what resulted in my article, is testing desktop software on the desktop. Because my tests are true unit tests, not component or integration tests, they 100% valid. I have yet to run into a case where the behavior was any different between the testing environment and the production code. The actual code that I'm testing is always running as native code. At this point I have almost 400 unit tests, and they've saved us a number of times from making breaking changes.

re: Unit-Testing Native C++ with Visual Studio

H Malthouse — Thu, 30 Dec 2010 02:31:01 GMT

Thanks for linking this Diego. (I'm assuming it was you as you're answering all the Qs)

When I first looked at using VS Test I recoiled at the unfamiliar C++/CLI code, but John's post is a great getting started guide. Very clear and to the point. I think I'll give it a go with my next project.

re: Unit-Testing Native C++ with Visual Studio

Diego Dagum — Wed, 29 Dec 2010 20:52:13 GMT

@Michael Hamilton, your point is certainly valid. In the cross-platform scenario you mentioned this out-of-box approach is not adequate and probably one of those frameworks some other readers said above we'll be better suited. The good news about those alternative testing frameworks is that many are enabled to be hooked into VS as extensions.

@Joe, I wish that were the reason of our buggy code: it's just a matter of swtching to CppUnit and we are leading the race again! ;-D

Seriously, if you found a definition that specifies that the validity of a unit test depends on the fact that the tester portion is built in the same environment than the tested part, I'll conceal you're right. Otherwise, as long as it fails when it's expected to fail, works fine when it's expected so, sorry for disagreeing here, friend. Related question: if you had to unit test for correctness a stored procedure, should the test itself be another stored procedure?

re: Unit-Testing Native C++ with Visual Studio

Joe — Tue, 28 Dec 2010 07:29:55 GMT

Sorry, Diego, but your wrong. You test native code with native code, anything else is NOT testing, but self-indulgent crap. Perhaps this explains so much buggy Microsoft code.

re: Unit-Testing Native C++ with Visual Studio

Michael Hamilton — Tue, 28 Dec 2010 06:49:19 GMT

"Being VS an already OS-dependent tool, getting its out-of-box testing mechanism based in an OS-dependent technology like C++/CLI doesn’t add/subtract anything, does it? ;-)"

It is fairly common to want to port a project to other platforms which may require other compilers, tools, and environments with the same code base. If you want to open source anything you cannot control what people are going to be running things on. If you're building libraries for commercial release it's a bit clunky to have a unit test so environment specific.

Not being able to bring unit tests across easily, let's say to a unix based system, seems like a fairly major stopping point. Even if you are certain you don't need to port your project today it's a little harder to say you'll never need to. If you can't easily bring your unit tests across that's a personal deal breaker. This is beyond the method itself feeling kind of clunky and contrived and setup heavy.

While this method may be "interesting" I think I'll be sticking with one of several in-language implementations. Recently been using boost's testing framework with some success, still playing with others though.

re: Unit-Testing Native C++ with Visual Studio

Diego Dagum — Tue, 28 Dec 2010 05:57:24 GMT

@Joe While desirable to keep the testing context as much production-like as possible, “unit testing” per-se is all about correctness proving for a single, isolated functionality (like a function or method) without much said about the environment (not at least in its formal definition). To put a couple of examples, unit testing for mobile application components aren’t usually being tested in the mobile device itself –not event in a device-emulated environment-. Likewise, we typically use XML files to mimic databases or web services responses, thus avoiding dealing with connectivity issues, etc, and that mocking technique doesn’t invalidate unit tests either.

Yet, your point about potential failures in the interop mechanism are valid although, do we really know about those interop inconsistencies today? My point here is about giving some likelihood to the current status of this IJW interop technique instead of discarding it right away: it has been out there for years, having matured with every new version of VS, being today the preferred approach for managed / unmanaged interoperability.

@kdprw: Being VS an already OS-dependent tool, getting its out-of-box testing mechanism based in an OS-dependent technology like C++/CLI doesn’t add/subtract anything, does it? ;-)

re: Unit-Testing Native C++ with Visual Studio

kdprw — Sat, 25 Dec 2010 20:47:33 GMT

What a great news: in order to write unit tests for your C++ code in VS, all you need is to use some non C++ (and effectively OS-dependent?) framework which, based on the comments here and the actual blog post, doesn't even seem to be suitable for testing C++ code.

re: Unit-Testing Native C++ with Visual Studio

Joe — Sat, 25 Dec 2010 18:31:54 GMT

This is still mostly bogus unless your DLL is intended to be used from .NET. Unit testing does not mean you more or less test the code as intended, but test it actually as intended. There is a big difference. So, for example, how do you know whether an error was caused by your code or by .NET interop? (I use a .NET interface to my native DLLs to do system tests, but not unit tests else the testing would be hopelessly incomplete.)

re: Unit-Testing Native C++ with Visual Studio

Diego Dagum — Thu, 23 Dec 2010 19:10:26 GMT

You're right, Ricardo. In fact, that consideration is common in C++/CLI - native interoperability. If you take a look to the official documentation about nullptr in C++/CLI (available at msdn.microsoft.com/.../4ex65770(v=VS.100).aspx ) you'll find

"The __nullptr keyword is a Microsoft-specific keyword that has the same meaning as nullptr, but applies to only native code. If you use nullptr with native C/C++ code and then compile with the /clr compiler option, the compiler cannot determine whether nullptr indicates a native or managed null pointer value. To make your intention clear to the compiler, use nullptr to specify a managed value or __nullptr to specify a native value."

I want to remind that the native tested functionality, not its managed testing counterpart, is not to be compiled as /clr and therefore any reference to native nullptr (the traditional NULL or (void*)0 before C++0x) is to remain as nullptr. In other words, despite the consideration, you don't need to touch your logic being tested to make it suitable for this testing framework.

re: Unit-Testing Native C++ with Visual Studio

Ricardo Costa — Thu, 23 Dec 2010 07:45:08 GMT

I've been using this technique for about a year now and it seems to be working good so far. The only recent downside I've met was the introduction of the C++0x nullptr keyword, which we cannot use because the unit tests compiled with /clr interpret nullptr as a managed keyword. We would have to use __nullptr instead on our code to avoid breaking the unit tests.