During this last product cycle the C# test team faced some interesting testing challenges.  Below is a recap from some of tools written by the C# test team for Visual C# and Visual Studio 2005.


CompilerFX:
One of the big challenges of testing a compiler and other language related features of the IDE is that numerous combinations of syntax that are possible to combine.  To help solve this problem, we created CompilerFx.  CompilerFx is an internal test tool that enables programmatic access to the C# AST (abstract syntax tree).  It is written in C# and exposes nodes, the symbol table and annotated parse tree through a rich object model.  It has various helper APIs that enables querying the code context easily.  The AST information is dumped by the C# Compiler; this enables the tool to have access to the latest language features as soon as they are available.
Having the C# AST opened up the opportunity for automatically generating tests.  Knowing the code context by querying the AST, we are able to recognize interesting code points such as a method invocation and its return type and then generate test cases based on that.  This is powerful when it can be done across an existing test suite of 20,000 compiler tests.  Since these new compiler tests are compiled and run, we would expect test execution to continue to pass, since the semantics of the code should change.  During the VS2005 product cycle CompilerFX based tools found close to 100 bugs across features including the compiler, the refactoring engine, edit and continue, and code formatting.   Testing using the AST has a lot of potential and is a place we’ll continue to invest.


TAO: Test Automation Object
An important part of what a QA team does is automating test scenarios, many of which require manipulating the user interface. Over time we have created internal tools based mostly on Active Accessibility to do so. A common and significant problem with these tools is synchronizing the tests with the target application; writing very robust tests using this method has not always been easy. When the product UI changes, it often breaks tests and simple focus issues on the test machine can cause false positives.
Trying to circumvent this problem the C# test team created TAO (Test Automation Object).  TAO is basically a test object in the language services that is only instantiated in a test environment.  TAO differs from previous efforts in the following ways:

·         Separation between tests and execution engine: TAO tests are written in XML making them independent of the engine that executes them.   This is really a data driven test.  This abstraction also enables us to use the same tests to verify a feature through the UI (with a more traditional UI manipulation technique) or through an API.

·         Bypass the UI:  TAO does not interact with the product through the traditional approach using Windows messages or Active Accessibility talking to the UI controls.  Instead it drives the product through a lower level test interface that enables it to reach functionality without having to mimic a user’s actions. Through these APIs tests can also obtain more detailed information on the product state including events and internal data structures.

·         Performance:  You might not consider performance as a critical feature of a test; however, we are finding more and more that the usefulness of a test depends a lot on how quickly it can run.  TAO tests execute about 4 times faster than the traditional UI based tests.

·         Efficient to write:  TAO tests take about half as long to write as a traditional UI test.


DELTA: Debugger Engine Level Test Automation
During VS2002, VS2003 and most of VS2005 the Debugger was tested mainly using an UI-based approach, driving the product through the UI exercising the test scenarios. DELTA is an engine level test library developed in C# that communicates with and drives the lower level debugger components through a set of API’s exposed by the Microsoft.VisualStudio.Debugger.Interop.dll. DELTA creates an instance of the Session Debug Manager (“SDM”) component of the debugger which is a dispatcher between the UI and the various lower level components called “debug engines”.  DELTA uses this instance of the SDM to perform various debugging actions such as launching an executable or attaching to an already running one, controlling the execution flow through stepping or breakpoints or manipulating the current state through locals, call stack or threads information. DELTA tests have all the benefits of non-UI automation such as very fast execution and significantly increased robustness.  We found that the cost of writing a Delta test was about the same as the UI approach, however, the DELTA test was about 30 times faster to execute. This makes sense when you consider that it’s only loading a small fraction of the DLL’s loaded in VS2005.   By targeting the lower level debugger components, DELTA enables us to specifically target components in our testing.  One of the biggest benefits is that these test are very robust and fast, which make them useful for checkin tests in addition to running as part of distributed test automation runs.


If you passionate about writing tools and implementing creative ways to test, check out
C# job openings (external)You can send your résumé to vcsjobs@microsoft.com or directly to me, I would love to hear your feedback.