Nikolai Tillmann's Blog

Pex 0.18 Released: New Tutorials, Moles work with Visual Studio 2010

nikolait — Thu, 15 Oct 2009 18:23:09 GMT

We just released a maintenance update of Pex. Together with this release come new tutorials, and support for Moles in Visual Studio 2010. As usual, this release also contains a number of bug fixes and small feature improvements, most of which address your suggestions from the Pex forum and bug reports sent to pexbug@microsoft.com.

New Tutorials: Stubs and Moles, and Unit Testing SharePoint with Pex

If you felt that you didn’t know how to get started with Stubs, Moles, and Pex, then you will like the following two tutorials. They are bundled in the Pex installer, and you can also get them from our website:

“Stubs - Lightweight Test Stubs And Moles for .NET - Getting Started Tutorial”
This document provides a short, step-by-step tutorial for the Stubs framework.
http://research.microsoft.com/en-us/projects/stubs/stubstutorial.pdf
“Unit Testing SharePoint Services with Pex”
This document provides a short, step-by-step tutorial on how to write isolated unit test for SharePoint Services using the Stubs framework and apply Pex to generate a test suite with code coverage.
http://research.microsoft.com/en-us/projects/pex/pexsharepoint.pdf

Moles for Visual Studio 2010

The [HostType(Pex)] attribute, which is required to execute generated, or hand-written parameter-less test cases that Moles, now also works with unit tests in Visual Studio 2010.

Breaking Changes

The code generation of Moles has changed again. This might mean that you will have to recompile your solution, and adapt all existing uses of Moles.

How do you use Pex in Visual Studio? Your Input Is Needed!

nikolait — Thu, 10 Sep 2009 04:44:00 GMT

Since it’s been almost a year that we’ve released Pex, we have started a little survey on our forums. We are looking for feedback on how you use Pex and how you would like to use it. If you are interested, please voice yourself on our mini survey at

http://social.msdn.microsoft.com/Forums/en/pex/thread/27c703ea-3928-41ef-b767-638c39966b99

The questions are:

1. Are you using Pex already? A) From Visual Studio, B) the command-line, C) other, D) no
2. Which version of Visual Studio are you using? A) Professional, B) Team System, C) other
3. How many people are in your team?
4. Do you write A) Parameterized Unit Tests (PUTs), or B) only use the Wizard-generated PUT stubs?

Any other comments are appreciated as well, about the license terms, or required Visual Studio versions…
Thanks, The Pex Team

Pex 0.13 Released: Unit Tests as Inputs, Fuzzing and Visual Studio 2010 Beta 1, F# Beta 1 support

nikolait — Fri, 29 May 2009 03:05:04 GMT

We just released an update of Pex that brings a very cool new feature, Unit Tests as Inputs and updates for Visual Studio 2010 Beta 1, Code Contracts and F# Beta 1. As usual, this release also contains a number of bug fixes and small feature improvements, most of which address your suggestions from the Pex forum.

Seeding the Exploration Part 1: Unit Tests as Inputs

Before using heavy-weight constraint solving to explore hard-to-reach execution paths, Pex can now leverage already existing unit tests that call a parameterized unit tests: Pex scans their body to extract the parameter values, and then Pex uses these values to seed the exploration. (In the past, Pex would have seeded the exploration by simply using the default values for all parameters, and nothing else.)

Here is an example, where Pex starts the exploration of the parameterized unit test Multiply by reusing the values mentioned in the unit test MultiplySeed, as you can see in the first row of the table of generated tests.

As a (intended) side effect, Pex also reuses the tests that Pex might have generated during earlier explorations. In effect, Pex never has to start from scratch again, but instead can reuse knowledge from earlier explorations.

There is a big limitation in this first version of the new feature: Pex can only use values of primitive types, and arrays of primitive types as seeds, and the values must be passed as parameters, not via PexChoose. The unit tests must be in the same class as the parameterized unit test, marked as a unit test, and have a simple structure, i.e., they must just create the input data, and the first method call must be to a parameterized unit test.

Seeding the Exploration Part 2: [PexArgument(…)] to Fuzz Inputs and Help Pex

Pex can not only extract values from existing unit tests, but there is in fact a general API through which seed values can be given. In particular, you can use the new PexArgumentAttribute (and a bunch of related new attributes) to provide seed values.

Here is an example:

During the subsequent exploration, Pex will often reuse parts of the seeded values, which may make constraint solving easier for Pex. Also, when Pex cannot solve all constraints to leap beyond a difficult branch, then you can use this feature to help Pex by providing the values yourself.

Object Creation with Invariants

Code Contracts enable the specification of class invariants by writing an “invariant method” decorated with the ContractInvariantMethod attribute. For example, for a typical ArrayList implementation, this invariant method might read as follows:

    public class ArrayList {
        private Object[] _items;
        private int _size;
        ...
        [ContractInvariantMethod] // attribute comes with Code Contracts, not Pex
        protected void Invariant() {
            Contract.Invariant(this._items != null);
            Contract.Invariant(this._size >= 0);
            Contract.Invariant(this._items.Length >= this._size);
        }

If you have such an invariant, and if you correctly configure runtime checking of contracts, then Pex will create instances of this type by first creating a raw instance, setting all private fields directly, and then making sure that the invariant holds. A generated test case for the Add method of the ArrayList class might then read as follows:

[TestMethod]
public void Add01() {
    object[] os = new object[0];
    ArrayList arrayList = PexInvariant.CreateInstance<ArrayList>();   // creates raw instance
    PexInvariant.SetField<object[]>(arrayList, "_items", os); // sets private field via reflection
    PexInvariant.SetField<int>(arrayList, "_size", 0);        // same
    PexInvariant.CheckInvariant(arrayList);                   // invokes invariant method via reflection
    arrayList.Add(null);                                      // call to method-under-test
}

A limitation of the current implementation is that the type itself must be public, and the types of all fields must be public (but not the fields themselves.)

Surviving unhandled exceptions in Finalizers

Finalizers are especially problematic for test runners: there is no guarantee from the runtime when they will be executed, and when they throw an exception, the process terminates. This is not an uncommon scenario when you apply Pex to (buggy) code with Finalizers: It seems as if Pex suddenly crashes.

In this version, we have improved the robustness of Pex with respect to those failures. Pex effectively rewrites all finalizers of instrumented types with a catch handler that swallows all exceptions and sends it to Pex’ logging infrastructure.

Visual Studio 2010 Beta 1 support

We’ve updated Pex for Visual Studio 2010 Beta 1, the latest Code Contracts version and F# Beta 1.

Send Feedback at your fingertips

We made it even easier for you to ask questions about Pex, or to just tell us what you think, right from Visual Studio:

Bug fixes and other small features

The Stubs framework generates stubs for types containing obsolete members. This behavior can be disabled.
The wizard can generate tests for inherited members by adding the /gim flag or from the wizard settings/options.
The default test assembly name suffix string, i.e., the suffix used to generate the test assembly name, can be specified on the wizard command line, e.g. /tans:.PexTests, or in the Visual Studio options (Tools –> Options –> Pex –> Wizard –> Test Assembly Name Format String).
int ChooseIndexValue<T>(string, T[]) was added to the choice API to choose valid index values.
We added new overloads in PexAssume/PexAssert that support IEnumerable<T>, i.e. for AreElementsEqual etc…
Better understanding of _.GetType(), and conditions involving System.Type values.
The Stubs framework can assign values to ref parameters.

Pex 0.12 Released: Exploration Tree View, Partial Stubs…

nikolait — Sat, 02 May 2009 06:36:00 GMT

Peli just announced the latest Pex Release,version 0.12.40430.3. Get all the details on his blog.

Pex 0.11 Released: Delegates, Exception Trees, and Stubs

nikolait — Wed, 22 Apr 2009 02:38:00 GMT

We just released Pex v0.11.40421.0, which you can download here. This release brings support for Delegates as Parameters, a new Exception Tree View, Stubbed Events, and Recursive Stubs. This also release fixes a blocking issue (incorrect registration of the Stubs Visual Studio Add-in), and it brings Pex in sync with the latest Code Contracts release.

Update: This release may still require a separate download of OpenMP dlls, which part of the vcredist.

Delegate As Parameters

Pex now understands Parameterized Unit Tests that take delegates as parameters. Pex will automatically generate a delegate instance and its behavior: if the delegate returns a value, Pex will generate a new symbolic value for it, track its use, and then generate different values depending on the program behavior. Let’s consider a simple method to illustrate this new feature. I wrote the following Parameterized Test method which takes a Func<int> delegate and throws an exception if the delegate returns a particular value:

When executing Pex on Test, Pex will generate a Func<int> which ‘asks’ Pex to choose the returned int (it uses PexChoose under the hood). Therefore, for each call to that delegate, Pex has the liberty to return a diferent value. Based on how it is used, Pex will generate different values to increase coverage. In this case, Pex ‘chooses’ to return 123 on the first call, which is exactly what we need to cover the exception branch. The following code is generated by Pex; it starts by setting up the desired values, and then it calls the parameterized unit test.

Pex Explorer: Exception Tree View

We have started to work on improving the experience when applying Pex to a large number of explorations. To this end, a new window called Pex Explorer will show various views over the events produced by Pex. The Exception Tree View provides the tree of exception types that Pex found. This is really helpful to quickly drill through the (really) bad exceptions first.

Pex Explorer: Contract Failures Tree View

If you are using Code Contracts, Pex also provides a specialized view to sort the contract failures kind.

Events in Stubs

Stubs now support events: the stubs simply expose the backing delegate fields (which hold the event delegate) as a public fields. As a result, one can clear, set, and invoke the event as any other member. Let’s see this with an example:

In order to raise the SomeEvent in a test, we would simply have to invoke the backing delegate field, which happens to have the same name as the event:

It’s that simple.

Recursive Stubs

Another common feature of mock/stub frameworks is to support nested invocation of members. Stubs now lets you recursively invoke property getters. Instead of assigning the property getter delegate, you can use new helper methods ending in ‘AsStub’ that take care of allocating the nested stub, assigning it to the property getter and returning it. Let’s see this with an example: assume we want to test the little snippet below.

In order to set up our stub, we would need to have IParent.Child return a stub of IChild, then set up IChild.Name to return ‘joe’. For property getters, Stubs generates a helper method that does just that:

The ChildGetAsStub call instantiated a SIChild instance, assigned to the Child property and returned it. It’s that simple.

Fixed Bugs

.Stubx fail to generate the stubs. We fixed a bug with the registration of Stubs in Visual Studio.

Breaking Changes

Microsoft.Stubs.dll has been renamed to Microsoft.Stubs.Framework.dll. The public Stubs type has been moved under the Microsoft.Stubs.Framework namespace.
The generated stub class names are built by prepending ‘S’ to the type name (‘IFoo’ –> ‘SIFoo’). In the previous version, the ‘I’ of interface name was trimmed which sometimes created name clashes.

Pex 0.10 Released / ICSE Tutorial on Parameterized Unit Testing

nikolait — Fri, 10 Apr 2009 22:42:45 GMT

Peli just announced that we have released Pex version 0.10. (You didn’t think that after version 0.9 we would go straight to version 1.0, did you?)

There are many exciting improvements, including better support for other test frameworks, VB.NET, F#, assertion generation, Code Contracts, and Stubs. Many of the changes are thanks to your feedback on our MSDN Pex Forum.

Also, together with Tao Xie, we are going to give a Tutorial on Parameterized Unit Testing at the International Conference of Software Engineering (ICSE) conference next month on May 17th in Vancouver, Canada. Go and register as long as slots are available. This is a great introduction if you are interested in using Pex or other test input generation tools to test your code, but you were struggling with the new concepts such as Parameterized Unit Tests.

Pex now a DevLabs project

nikolait — Fri, 24 Oct 2008 10:52:00 GMT

Soma just announced DevLabs (http://msdn.microsoft.com/DevLabs), "a site dedicated to [...] software innovations for the developer community".

We are proud that Pex is one of the first projects hosted on DevLabs.

What does that mean for Pex?

Pex for Visual Studio Team System 2010 CTP

At the same time as Pex became a DevLabs project, we made Pex available as a Microsoft Download for Visual Studio Team System 2010. But before you go there to download the latest version of Pex, keep reading and let us explain this in more detail: Right now, if you download Pex from that link, it will only install if you have VS2010 installed. The first CTP ("Community Technology Preview") of VS2010 will be distributed to attendees of this month's Professional Developers Conference (PDC) in Los Angeles. Another reason to go to PDC. As an alternative, you can still use the Microsoft Research Download which does not require VS2010, and works great with VS2008; it is identical in terms of features.

The new Microsoft Download comes under a Microsoft Pre-Release Software License for Pex. This license is different from the Microsoft Research License agreement: It does not explicitly rule out all forms of commercial use.

An excerpt from the new license: "PRE-RELEASE SOFTWARE. This software is a pre-release version. It may not work the way a final version of the software will. We may change it for the final, commercial version. We also may not release a commercial version."

Experiment, evaluate, and then join us in the conversation

As Soma said: "help us determine the direction that these projects should eventually head". As an early adopter, now is the best time for you to try out Pex and give us feedback. The Pex team is very open to suggestions, as Pex is still rapidly evolving.

MSDN Forum for Pex

Pex has now an MSDN Forum, where you can ask questions and start discussions.

Pex 0.8 Released, featuring Code Digger and Stubs

nikolait — Wed, 22 Oct 2008 23:53:00 GMT

We just released the most exciting iteration of Pex to this date: version 0.8.

Update: Pex is now a DevLabs project, and available as a Microsoft Download for Visual Studio 2010 CTP. (It won't install without it.)

The Microsoft Research download still comes with the Microsoft Research license, so you cannot use it for commercial purposes, but it's perfect if you just want to try it out, or use it for academic purposes; it targets Visual Studio 2008 Professional, but all you really need is Windows XP with .NET 2.0.

Besides lots of bug fixes and little improvements, this release comes with two huge innovations: Code Digger, and Stubs.

New Experience: Code Digger

Did you ever write some new code, change some old code, or simply look at existing code, and you were not quite sure what it was capable of? Which inputs would trigger a successful "happy" path, and which ones would be rejected? Maybe you were thinking: "If only I had a test suite..."

You can have that test suite! (And you won't have to sweat for it.)

We have added an entirely new way of using Pex: Just right-click somewhere in the code you are currently editing, and then Pex will analyze all the branches in the code.

Pex shows you a table of interesting input/output values, and you can save all findings as a unit test suite with a single button click.

There you are. Enjoy your small test suite with high code coverage that integrates with VIsual Studio Team Test!

We call this new experience Code Digger. If the short description above didn't quite make sense to you, read my longer post on Code Digger, and check out the full Code Digger tutorial (it already refers to the forthcoming VS2010, but don't let that stop you).

New Feature: Stubs

Also shipping in this release: Stubs - A Simple Framework For .NET Test Stubs; also comes with a manual.

It's a no-nonsense framework to easily create and maintain lightweight implementations of interfaces and abstract classes.

Stay tuned for more: PDC is coming

There will be further announcements.

And don't forget: Pex is at PDC! Go to our talk at PDC and get the latest demo at the Microsoft Research booth!

Sneak Preview: Code Digger — The New Pex Experience

nikolait — Tue, 21 Oct 2008 21:59:00 GMT

Do you control your code?

Well, be our guest, or rather Pex' guest: You can have that test suite! (And you won't have to sweat for it.)

Update: You can now download Code Digger as part of the latest Pex release. And just as a reminder: Register for PDC, go to our talk and watch the latest demo at the Microsoft Research booth!

If you want to read more, we have just completed the full Code Digger tutorial. It already refers to VS2010, but the Microsoft Research download also works with VS2008.

Code Digging with Pex

Ok, back to your code. Let's say you are looking at a method that capitalizes words in a string. With horror, you’ve just realized that you totally forgot to write tests for it. In fact, you even don't remember why the method has so many if-statements that treat special cases. It seems that the situations is hopeless, but wait... All you have to do is right-click on your code in the Visual Studio editor, and select "Run Pex Explorations"!

That will trigger a deep code analysis using the Pex engine.

And let us repeat the main point: You start the analysis from the code, not from an already existing Unit Test, or even Parameterized Unit Test! No test suite needed. No test project needed. The only reason we are using the word "test" so often because we want to stress that you don't need any.

So, what do we get? A table with inputs and outputs that cover all the successful and failing corner cases. To this end, Pex analyzed all the branches in the code to figure out what interested values are. The code we were "digging" here only distinguished lower-case characters, upper-case characters, and punctuation characters. (This is the same kind of table that Pex produces when you analyze Parameterized Unit Tests.)

Okay, that's nice. You can look at all the cases and check whether they make sense to you. You can use all the already existing Pex features. For example, with two mouse clicks, you could let Pex fix your code to throw an ArgumentNullException instead of the NullReferenceException.

But there is more. Remember, we promised you a test suite...

Code Digging for Gold: Test Suites for Free!

You can select one, several, or simply all rows, and click on the "Save..." button.

Pex will create an entire test project, and generate C# code for all selected rows as unit test.

Integration with Visual Studio Team Test

And what's best: The generated tests integrate perfectly with the Visual Studio Team Test unit test framework, showing up in Visual Studio's Test View window.

Now you can leverage all unit testing features that come with Visual Studio. For example, you can let Visual Studio collect code coverage information. Or, if you wonder how a line in the code can be triggered, you can set a breakpoint in that line, and run the test suite under the debugger. It's that easy. With the Pex Code Digger, you don't have to write a test driver yourself!

Regression Test Suite

You might wonder: Besides debugging issues right away, what is this generated test suite actually good for? We thought about that too. Each of the generated tests contains a line that asserts the behavior that Pex observed today, for example:

   1: [TestMethod]   2: [PexGeneratedBy(typeof(StringExtensionsTest))]   3: public void Capitalize11()   4: {   5:     string s = this.Capitalize("p:p");   6:     Assert.AreEqual<string>("P_P", s);   7: }

After saving today's tests, when you ever change the code again in the future, you will have an excellent regression test suite that will identify the breaking changes.

Does this replace traditional testing?

Not really. With Code Digger, you have to manually inspect each generated test case to check whether your code does what you want it do to. In contrast, Unit Tests, or even better, Parameterized Unit Test, are an excellent way to describe programmatically what your code is supposed to do. Code Digger and (Parameterized) Unit Testing are complementary, and one doesn't make the other obsolete.

The end

That's all for this post; keep watching for further announcements, or read the full Code Digger tutorial.

Pex at PDC2008

nikolait — Sat, 04 Oct 2008 10:19:02 GMT

We are very excited to announce that Pex has a session at PDC 2008. The session combines two topics: how to write and statically check code contracts in the spirit of Spec#, and how to generate test cases with Pex directly from your product code. We'll demo new cool features you haven't seen before.

Book it now in your conference agenda! (look for Research in the session list, or search for Pex).

See you there and don’t forget to swing by the Microsoft Research booth where we'll demo Pex just for you.

Pex 0.7 Released

nikolait — Tue, 23 Sep 2008 01:35:33 GMT

We just published Pex 0.7. This update is mainly a maintenance release. We have been working on fixing bugs, those reported by our users, as well as many others we found while using Pex internally. There are a few breaking changes (mainly renamings), and a few improvements under the cover.

Patterns for Parameterized Unit Testing

Writing good parameterized unit tests is hard. We have started to collect a set of patterns how to write good parameterized unit tests, and avoid bad ones. Peli gives an introduction to our new paper with patterns in his blog.

Better Reasoning

Pex now knows a bit more about enums, so that you can write assumptions such as the following.

        1: [PexMethod]

       2: public void TestWithEnumParameter(MyEnum value)

       3: {

       4:     PexAssume.IsTrue(Enum.IsDefined(typeof(MyEnum), value));

       5:     // ... use value in test ...

       6: }

The assumption above filters out those enum values which are not defined.

Better Exploration Strategies

In addition to such improvements in local reasoning, we also worked on better global reasoning, i.e., searching more effectively for interesting feasible execution paths.

Together with Tao Xie, who visited us from North Carolina State University, we developed a search strategy based on the idea of fitness. The result is a paper [PDF], and we also published the source code on CodePlex. If you dare, you could try to write your search strategy, plug it into Pex, and see if your strategy solve problems that Pex couldn't solve before!

For example, before we integrated the the fitness-based search strategy, Pex had a really hard time to find the "bug" hidden in the following test.

       1: [PexMethod]

       2: public void FitnessTest([PexAssumeNotNull]int[] x)

       3: {

       4:     int y = 0;

       5:     for (int i = 0; i < x.Length; i++)

       6:         if (x[i] == i)

       7:             y++;

       8:     if (y>=10)

       9:         throw new Exception("bug");

      10: }

(I realize that this test doesn't do anything useful. It's just the essence of a testing problem.) You need to come up with a very particular input array to hit the "bug". The problem here for Pex is that the variable y, whose value decides whether we hit the bug, is related to the test inputs in a very indirect fashion. To better attack those kinds of problems, Pex now uses the notion of fitness to guide the search, here towards those test inputs which cause y to be closer and closer to 10.

Pex 0.6 Released

nikolait — Wed, 06 Aug 2008 20:30:19 GMT

We just released another milestone for Pex: version 0.6, still under the Microsoft Research License. Ideally you have Visual Studio 2008 Professional to get the full experience, but all you really need is .NET 2.0. Take a look at exercise 3 in the tutorial to get started in Visual Studio 2008 Professional, and exercise 4 shows you how to use the command-line.

There have been a bunch of changes that we documented in the release notes, most notably some UI polishing, and finally, the ability to install Pex under a 64-bit Windows operating system!

64-bit support: Pex 0.6 installs on 64-bit Windows (but cannot analyze 64-bit-only code)

We have taken some precautions that all processes that Pex launches are 32-bit processes. Since Pex actually runs your code during the analysis to generate test inputs, this will only work when the code that you want to analyze is marked as "Any CPU" or "x86", but not "X64" (or "Itanium"). This is normally not an issue.

And you can run the generated tests on any platform (assuming your test framework supports other platforms; note that MSTest only supports 32-bit as well).

Why Pex cares about 32-bit vs. 64-bit

Pex analyzes .NET code, and you would think that it shouldn't matter for such a tool whether it runs in a 32-bit or 64-bit environment. The only difference should be how many bytes a pointer/object reference occupies, and a .NET program cannot observe such implementation details, right?

Indeed, it is true that most safe .NET applications couldn't care less about the different pointer sizes.

But Pex can not only analyze safe .NET code, but also unsafe .NET code. (In fact, many of the methods of the .NET framework are implemented using unsafe code, such as string.Equals.) In unsafe .NET code, conversions between integers and pointers happen all the time. Some conversions are explicit, and many more are implicit in the code generated by the C#.

To keep things simple, let me show you an example with an explicit conversion. The following is a parameterized unit test that has quite different behaviors depending on whether it is running in 32-bit or 64-bit mode. Can you guess in which mode the "bug" can occur?

        1: [PexMethod]

       2: public unsafe void TestX86(long x)

       3: {

       4:     void* p = (void*)x;

       5:     if (p == null && x == (1L<<32))

       6:         throw new Exception("bug!");

       7: }

Answer: Only in 32-bit mode. There, the conversion from long to (void*) ignores the upper 32-bits, which can then be freely assigned in any way. Pex finds the following in 32-bit mode:

In words: When calling TestX86 with the value 4294967296, the exception is thrown. In 32-bit mode, that is. In 64-bit mode, the exception cannot be thrown for any value.

Does the difference in behavior between 32-bit and 64-bit really matter in practice?

For most purposes, probably not. (Unless you are writing unsafe code, and you are concerned about buffer-overflows and such.) In any case, you can just take the generated test suite and run it on any platform you wish.

Why don't you just recompile Pex for 64-bit then?

We were asked on our mailing list why we couldn't just recompile Pex for 64-bit. As you have seen above, it is not that simple, since Pex and its constraint solver needs to reason about different constraints depending on the bitness. All the explicit and implicit conversions in the .NET instruction have different meanings.

But a true 64-bit version of Pex is not that distant: My main development machine runs 64-bit Vista, and I implemented all relevant conversion for 64-bit as well. The main problem is that our automated testing process for Pex runs on a 32-bit Vista machine. Testing Pex itself for both 32-bit and 64-bit environments would double our testing matrix.

Fun with the ResourceReader

nikolait — Thu, 05 Jun 2008 02:49:00 GMT

After we released Pex recently, I came across a couple of interesting blog posts of people who tried out Pex, for example Ben Hall, Peter, and Stan. They all ran Pex on a small example. In this post, I want to run Pex on a more complicated piece of code: The .Net ResourceReader, and all the other classes which it uses under the hood.

The ResourceReader takes a stream of data, and splits it up into individual chunks that represent resources. In that sense, the ResourceReader is a big parser.

Writing a parameterized unit test

I started by creating a new C# test project in Microsoft Visual Studio, and adding a reference to Microsoft.Pex.Framework. (If you try this at home, you need to install Pex first, of course.)

Then I wrote a very simple parameterized unit test for the resource reader. It takes any (non-null) array of bytes, creates an UnmanagedMemoryStream with the bytes, and then decodes it with the ResourceReader. It's not a particular expressive test, since it doesn't contain any assertions. All this test is really saying is that, no matter what the test inputs are, the code shouldn't throw any exception. (No exception at all? I'll investigate this in more detail later.)

   1:      [PexClass, TestClass]

   2:      public partial class ResourceReaderTest

   3:      {

   4:          [PexMethod]

   5:          public unsafe void ReadEntries([PexAssumeNotNull]byte[] data)

   6:          {

   7:              fixed (byte* p = data)

   8:                  using (UnmanagedMemoryStream stream =

   9:                      new UnmanagedMemoryStream(p, data.Length))

  10:                  {

  11:                      ResourceReader reader = new ResourceReader(stream);

  12:                      foreach (var entry in reader) { /* just reading */ }

  13:                  }

  14:          }

  15:      }

Let's run Pex.

Run Pex Exploration

I can now let Pex analyze this parameterized unit test in-depth by right-clicking on the parameterized unit test, and selecting Run Pex Exploration.

(I am running our latest internal Pex bits, and we just renamed the "Pex It" menu item to "Run Pex Exploration". So if you just see a "Pex It" menu item, use that.)

Pex now generates test inputs. In a nutshell, Pex runs the code, monitors what it's doing, and uses a constraint solver to determine more relevant test inputs that will take the program along different execution paths.

That sounds great, but the result is a little bit disappointing: Pex reports only two test inputs: An empty array, and an array with one element that is zero.

(Again, I have to say that I am running our latest internal Pex bits. If you see a big black empty region in the lower right part of the Pex Results, titled with "dynamic coverage", then you can get rid of this waste of screen real-estate by pressing the button that I circled in red:

We won't show this "dynamic coverage" graph by default anymore. I might talk more about this feature in the future. And now, back to the ResourceReader.)

Pex tells us in bold letters on red ground that it encountered 3 Uninstrumented Methods.

What does that mean? Well, Pex performs a dynamic analysis, which means that it runs the code, and then Pex monitors what the code is actually doing. However, Pex doesn't monitor all code, but only code that Pex instruments. Pex doesn't instrument all code by default, since the instrumentation and the monitoring comes with a huge performance overhead. And often it is not necessary to monitor all the code that is running. For example, when you want to test your algorithm, and your algorithm happens to write its progress to the console, then you still only want to test your algorithm, but not all the supporting code that hides behind Console.WriteLine.

Configuring what Pex analyzes: Instrumentation settings

But back to the ResourceReader: Let's click on the 3 Uninstrumented Methods warning.

We see that Pex neither instrumented UnmanagedMemoryStream nor ResourceReader. No wonder that Pex' analysis didn't find any really interesting test inputs! Pex also didn't monitor what happened in a call to GC.SuppressFinalize, but this relates to the .NET garbage collector, and probably isn't so important. (In fact, Pex has a built-in list of classes which don't need to be monitored, and we might add the GC class to this list in the future.)

Let's select UnmanagedMemoryStream..ctor (".ctor" is the generic .NET name for constructors),

and click on Instrument type, and repeat for ResourceReader..ctor. For GC.SuppressFinalize, I click on Ignore uninstrumented method instead. Pex persists my choices as assembly-level attributes in a new file called PexAssembyInfo.cs in the Properties folder. It now looks like this:

   1:  using Microsoft.Pex.Framework.Instrumentation;

   2:  using System.IO;

   3:  using System.Resources;

   4:  using Microsoft.Pex.Framework.Suppression;

   5:  using System;

6:

   7:  [assembly: PexInstrumentType(typeof(UnmanagedMemoryStream))]

   8:  [assembly: PexInstrumentType(typeof(ResourceReader))]

   9:  [assembly: PexSuppressUninstrumentedMethodFromType(typeof(GC))]

I let Pex run again. You can re-run the last exploration by clicking on the Play-button:

Pex did a little bit more this time -- three test cases! And another 7 Uninstrumented Methods warnings...

What are the uninstrumented methods this time?

As you can see, Pex' analysis already proceeded much deeper into the code, and now Pex hits all kinds of other methods. Most of the methods sound relevant. So I repeat the game, and let Pex instrument all of the types. And then I let Pex run again. Boom! Pex still reports some uninstrumented methods (and lots of other things which I'll explain another time), but Pex also doesn't stop so quickly anymore. Instead, Pex is having a ball and it keeps producing more and more test cases:

(If you run this example yourself, the precise results you get will vary. There is some non-deterministic code involved in the analysis.)

Pex even managed to produce a valid resource file! (here, row 215 with the green checkmark)

When you double-click on any row, Pex shows the generated unit test code. Here is the valid resource file:

   1:          [TestMethod]

   2:          [PexGeneratedBy(typeof(ResourceReaderTest))]

   3:          public void ReadEntriesByte_20080604_134738_032()

   4:          {

   5:              byte[] bs0 = new byte[55];

   6:              bs0[0] = (byte)206;

   7:              bs0[1] = (byte)202;

   8:              bs0[2] = (byte)239;

   9:              bs0[3] = (byte)190;

  10:              bs0[7] = (byte)64;

  11:              bs0[12] = (byte)2;

  12:              bs0[20] = (byte)7;

  13:              bs0[24] = (byte)128;

  14:              bs0[25] = (byte)128;

  15:              bs0[32] = (byte)4;

  16:              this.ReadEntries(bs0);

  17:          }

The resource file is 55 bytes long, and most bytes are zero, the default value of the byte type. All values which are not zero were carefully chosen by Pex to pass all the file-validation code of the ResourceReader.

Configuring the test oracle: Allowed exceptions

All the other rows are tagged with an ugly red cross, indicating that the test failed. Did these tests really fail? Let's see. In all cases, the code either throws a ArgumentNullException, ArgumentException, BadImageFormatException, IOException, NotSupportedException, or a FormatException.

These exceptions don't sound so unreasonable, considering that Pex tried obviously ill-formed test inputs in the course of its exploration.

If you look around the documentation of the ResourceReader (here, here, and here), you will find that in fact most of these exceptions are documented somewhere (although the documentation could be more explicit, and it's missing NotSupportedException).

I can inform Pex that these exceptions are okay. The easiest way to do that is to select a failing test case in the table, and then click on Allow It. Again, my choice is persisted as attributes in PexAssemblyInfo.cs:

   1:  [assembly: PexAllowedExceptionFromAssembly(typeof(ArgumentNullException), "mscorlib")]

   2:  [assembly: PexAllowedExceptionFromAssembly(typeof(BadImageFormatException), "mscorlib")]

   3:  [assembly: PexAllowedExceptionFromAssembly(typeof(ArgumentException), "mscorlib")]

   4:  [assembly: PexAllowedExceptionFromAssembly(typeof(NotSupportedException), "mscorlib")]

   5:  [assembly: PexAllowedExceptionFromAssembly(typeof(IOException), "mscorlib")]

   6:  [assembly: PexAllowedExceptionFromAssembly(typeof(FormatException), "mscorlib")]

When you run Pex again, all the exceptions that are marked as allowed are now tagged with friendly green checkmarks, indicating that they passed.

But what is that? An OverflowException... (here, the red cross in row 98) Yet another undocumented exception...

What can we learn from this exercise?

You have seen how to configure code instrumentation settings, and how to tell Pex that certain exceptions are okay.

But I think there is a much more interesting insight: By using Pex, we discovered some unexpected ways how some complicated library code behaves. Library code that we build our applications on. Documentation will always be imperfect, and we have to test our applications to make sure that we everything works correctly together. By using a white-box testing tool like Pex, that tried many relevant test inputs and not just one exemplary test input, we discovered behaviors (here, exceptions) we were not aware of before!

Stay tuned for more

That's it for today. Another time I might talk about the constraints that Pex had to solve to create a valid resource file, how to get an idea about how much code analyzed, how to visualize that Pex makes progress, and how to write more expressive parameterized unit tests.

This posting is provided "AS IS" with no warranties, and confers no rights.

Pex 0.5 Released

nikolait — Thu, 22 May 2008 04:26:12 GMT

Today we released the first version of Pex under a Microsoft Research License. Ideally you have Visual Studio 2008 Professional to get the full experience, but all you really need is .NET 2.0.

We even wrote a tutorial. While it's quite long, exercise 3 helps you to get started with Visual Studio 2008 Professional, and exercise 4 shows you how to use the command-line.

What is Pex?

Pex generates test inputs that cover all, or at least many of the corner cases in your .NET code. These test inputs are plugged into parameterized unit test that you write. The result is a small unit test suite, where each unit test calls the parameterized unit test with particular test inputs. There is a great picture on our main Pex page that illustrates this process.

Pex supports other unit test frameworks since the unit tests that Pex generates can be executed by other unit test frameworks without Pex. Pex comes with support for MSTest, the unit test framework of Visual Studio, out of the box. For support for other unit test frameworks, please look at the Pex Extensions project.

Parameterized unit tests have been around for quite some time already, under several names -- row tests, data-driven tests, theories, etc.

What is really unique about Pex is that it analyzes your .NET code, instruction by instruction, to understand what your code is doing. Then, in a fully automatic way, Pex computes relevant test inputs that trigger the corner cases of the code. When you write assertions, Pex will try to come up with test inputs that cause an assertion to fail.

Is Pex a new Microsoft product?

No, it's a prototype from Microsoft Research, developed mainly by two people. I started Pex in 2005, and Peli de Halleux joined the Pex team in 2006. Pex leverages other cool Microsoft Research technology, in particular the constraint solver Z3. We know that Pex has many rough edges, its version number is 0.5 for a reason.

Pex illustrates our vision how testing can be more powerful (and fun), without the difficulties of proving programs correct.

At this point we are mainly looking for feedback, and we want to see what it can do for other people.

Final Thoughts

To ask questions, get help, or just give feedback, please take a look at our mailing lists.

While Peli has blogged about Pex before, this is my first post. Stay tuned for more in-depth information on Pex at this place. I'll also talk about its history, and the other more or less related research projects I worked on in the past (AsmL, Spec Explorer, XRT).

Hope you have some fun with Pex,
Nikolai

Disclaimer: This posting is provided "AS IS" with no warranties, and confers no rights.