Ivo Manolov's Blog

Model-View-ViewModel (MVVM) Applications: General Introduction

ivom1 — Sat, 17 Mar 2012 23:35:31 GMT

MVVM is a central concept in WPF, Silverlight, WindowsPhone and Windows 8 development, so I decided to take some time and bubble up some of the content in the WPF MVVM Toolkit, which we created some time ago.

Model-View separation is by no means a novel idea in the software engineering industry—the idea has been around for at least 25 years. In the past few years, there has been a lot of interest in model-view architectures, fuelled both by the growing complexity of modern software systems and by the necessity to display UI on various devices, while reusing the same underlying business logic.

This post outlines the benefits of model-view separation. It is essentially the same content as the content included in Part 1 of the documentation, which goes with the WPF MVVM Toolkit.

1. Introduction to the Model-View-X Paradigm

A well-designed application is an application that is easy to develop, test, maintain, and evolve. Creating such an application typically involves some form of separation and encapsulation of responsibilities. A common approach involves separation of the UI (the views) from the business logic (the model). In fact, that approach is so common that there are a number of stock solutions, collectively known as the model-view design patterns:

Model-View-Controller (MVC) is the granddaddy of them all, dating back to 1979. In MVC, the Controller creates the Views and the Model. The Views register with the Controller, which also has a reference to the Model. A View notifies the Controller of user interactions, and the Controller in turn queries the View for its state and updates the Model appropriately. The Controller notifies the Views when there are changes in the Model, and the Views query the controller to update their states.
Model-View-ViewModel (MVVM) is a derivative of MVC that takes advantage of particular strengths of the Windows Presentation Foundation (WPF) architecture to separate the Model and the View by introducing an abstract layer between them: a “Model of the View,” or ViewModel. The pattern was first introduced by John Gossman – the architect on the Blend/WPF/Silverlight team – and derives from the Smalltalk ApplicationModel pattern, as does the PresentationModel pattern described by Martin Fowler. It was first utilized by the Expression team (John was part of that team at the time) as they developed version 1 of Expression Blend. Without the WPF/Silverlight-specific aspects, the Model-View-ViewModel pattern is identical to PresentationModel.

In MVVM, the View and the ViewModel are typically instantiated by the container application. The View keeps a reference to the ViewModel. The ViewModel exposes commands and observable entities (“bindables”) that the View binds to (or to put it in programming terms, view.DataContext = viewModel;). User interactions with the View trigger commands on the ViewModel, and updates in the ViewModel are propagated to the View through data binding.

2. Justification

2.1. Simplistic Application Design

To demonstrate the use of the Model-View-X architecture, let’s consider a simple contacts application. The application has two windows (views): a table view, where you can see all entries in the contact book in a table, and a record view, where you can see individual records.

Fig. 1 UI of the contacts application

The Model-View separation for this particular problem feels natural, where the Model is the contact book and the Views are the two windows shown in Figure 1. So the trivial design (which is a two-tier Model-View-only design) would look as shown on Figure 2.

Fig. 2 Simplistic application design – the views are directly connected to the model

As shown, ContactBook keeps a list of views (which are added and removed with AttachView and DetachView). When a contact changes, ContactBook notifies all views by calling their Update methods, and all views update themselves by calling the model's GetContacts method. Upon instantiation, RecordView and TableView are given a reference to ContactBook, which they keep as a field (contactBook).

Notice how the two views are tightly coupled with the ContactBook class. Addition of business logic would only increase the level of coupling.

2.2. Simplistic Application Design – WPF/Silverlight Version

Before examining the design further, let’s convert the design to a WPF/Silverlight-friendly design. WPF/Silverlight provides the following native facilities that can be utilized directly:

Databinding is the ability to bind UI elements to any data.
Commands provide the ability to notify the underlying data of changes in the UI.

Using these facilities, the WPF/Silverlight-friendly design is changed as follows:

Fig. 3 WPF/Silverlight-friendly simplistic application design

In the modified design in Figure 3, the views derive from UserControl (there is no need for an abstract View class) and ContactBook no longer needs to maintain a list of views—ContactBook no longer needs to know about the views. Instead, the views point to ContactBook as their DataContext and use WPF data-binding to bind to the list of contacts.

You will also notice that List<Contact> has been replaced with ObservableCollection<Contact> to allow the views to bind to the ContactBook using WPF data-binding mechanisms.

Notice how WPF’s data-binding mechanism allows us to create a much more a loosely coupled design (at least until further addition of business logic).

2.3. Problems with the Simplistic Design

Things get more complicated if we introduce the following new functionality:

Track the selected item, so that we update RecordView whenever the selection in TableView changes, and vice versa.
Enable or disable parts of the UI of RecordView and TableView based on some rule (for example, highlight any entry that has an e-mail in the live.com domain).

Feature (a) can be implemented by adding a CurrentEntry property directly in ContactBook. This solution breaks down, however, if you have more than one instance of the UI connected to the same ContactBook. Feature (b) can be implemented in the views (RecordView and TableView). The problem if we do that, however, is that if we want to change the rule, then we would need to change both views.

Gradually it becomes apparent that we need a third layer in our application. We need a layer between the views and ContactBook that stores state shared by the views. We need a meta-view of sorts.

2.4. Enter Model-View-ViewModel (MVVM)

A ViewModel can serve as the third layer, providing an abstraction that acts as a meta-view (a model of a view), storing state and policy that are shared by a group of views. By introducing the concept of a ViewModel, our design changes to the following:

Fig. 4 Application design using a ViewModel

In this design, the views know of the ViewModel and bind to its data, to be able to reflect any changes in it. The ViewModel has no reference to the views—it holds only a reference to the model.

For the views, the ViewModel acts both as a façade to the model, but also as a way to share state between views (selectedContacts in the example). In addition, the ViewModel often exposes commands that the views can bind to and trigger.This use of commands is covered in the next section.

2.5. The use of commanding in MVVM Applications

WPF and Silverlight introduce a new input programming mechanism: commanding. Commands allow loose coupling between the origin and the handling of an action. Multiple sources can invoke the same command, and the same command can be handled differently depending on the target. All commands implement the ICommand interface.

For UI actions, WPF provides a specialized RoutedCommand class. A RoutedCommand is routed through the UI element tree, so the target of a RoutedCommand must be part of the UI element tree. In MVVM designs, however, command targets are often implemented in the ViewModels, which in turn are not typically part of the UI element tree. This requires the introduction of a different kind of command—the DelegateCommand—that implements ICommand and can have non-UI elements as its target.

Fig. 5 Application design using a ViewModel, exposing bindable commands

The DelegateCommand construct was originally introduced during the development of Microsoft Expression Blend version 1. A generic version of DelegateCommand was later released as part of the Composite Application Guidance for WPF on MSDN. Note that some MVVM sources refer to the same construct as RelayCommand.

A DelegateCommand is an ICommand that allows the views to bind to the ViewModel. Even though the diagram above does not show this explicitly, all ICommands exposed by the ViewModel are DelegateCommands. For example, the code for the AddContactCommand in the ViewModel may look as follows:

3. Benefits and Consequences of Using MVVM

The benefits of MVVM are the following:

A ViewModel provides a single store for presentation policy and state, thus improving the reusability of the Model (by decoupling it from the Views) and the replaceability of the Views (by removing specific presentation policy from them).
The Model-View-ViewModel design improves the overall testability of the application. One can easily create unit tests that target the Model and the ViewModel layers. The MVVM design also improves the “mockability” of the application, by allowing easy run-time simulation of the tiers, which is crucial to testing complex software products.
The Model-View-ViewModel design is a very loosely coupled design. The View holds a reference to the ViewModel, and the ViewModel holds a reference to the Model. The rest is done by the data-binding and commanding infrastructure of WPF.

The consequences of MVVM are the following:

The typical relationship between a ViewModel and the corresponding Views is one-to-many, but there are also situations where that is not necessarily true. In general, any business logic and input-handling business logic (selection tracking, and so on) is kept in the ViewModel.
There are situations when one ViewModel is aware of another ViewModel within the same application. Such situations arise when there is a master-subordinate relationship between two ViewModels or when a ViewModel represents a single item (for example, the visual representation state of a single contact). When this happens, one ViewModel can represent a collection of ViewModels. Such a case in demonstrated in the 2nd part of documentation that comes with the WPF MVVM Toolkit.

4. Next Steps and References

Hopefully this post provides a simple introduction to the pattern. As a next step, download the WPF MVVM Toolkit, review the walk-through and the provided code and go write some apps.

It is important to note that there are a number of other (more full-featured) MVVM toolkits out there. Perhaps the most popular one is Laurent Bugnion’s MVVM Light, but there are many others on Codeplex. You should be using those for your WPF / Silverlight / Windows Phone and Windows 8 applications. Here are a few additional references that may help too:

Presentation Model by Martin Fowler, July 2004.
Introduction to Model/View/ViewModel pattern for building WPF apps by John Gossman, October 2005.
Separated Presentation by Martin Fowler, June 2006.
WPF Patterns by Bryan Likes, September 2006.
WPF patterns: MVC, MVP or MVVM or…? by the Orbifold, December 2006.
Model-see, Model-do, and the Poo is Optional by Mike Hillberg, May 2008.
PRISM: Patterns for Building Composite Applications with WPF by Glenn Block, September 2008.
The ViewModel Pattern by David Hill, January 2009.
WPF Apps with the Model-View-ViewModel Design Pattern by Josh Smith, February 2009.

Principles 5: End-to-End Development Process (for Large Projects)

ivom1 — Fri, 22 Apr 2011 06:21:00 GMT

Series Index

+++

This fifth post discusses and end-to-end development process for large software projects.

The only difference between large and small projects is that large projects require more “supporting infrastructure” (more processes, etc.) than small projects, due to the fundamentally imperfect communication between the team members. The “supporting infrastructure” is a tax we have to pay when we develop software. We can minimize it, but we can never really avoid it completely.

Larger projects require larger teams, which (due to accumulated communication imperfections) require larger process tax.

On one hand, we like short product cycles, because they get the product out in the market quickly; on another hand, the shorter the cycle, the higher the ratio between process tax and actual productive time. This is a classic case of an optimization problem, which can be defined as…

“Find the shortest possible project cycle, that keeps the process tax at a reasonably low level“.

Clearly, there’s a variety of other factors that we need to consider when we talk about project and product cycles. To list a few, we need to factor in the general product business model, pre-existing contractual obligations, specific calendar events such as conferences or trade shows, the competitive landscape, etc. For the purpose of this article we can assume that all of these additional factors have already been taken into consideration.

Single Project Rhythm

I am a big believer in having a single, transparent, well-socialized and strictly enforced project rhythm. All products that an organization delivers in my opinion should be aligned to this single rhythm. The rhythm (or cycle) should be short enough, so that it can accommodate all products in development. In this model, different products can take different number of cycles. At the end of every cycle, one or more products get released.

Having a single cycle in my opinion dramatically decreases the complexity (and ultimately the process tax) that a team has to pay. It helps create well-aligned, synchronized and cohesive organizations, reduces dramatically the release management costs of a product, practically eliminates context switching for the individual engineers, drives ownership deep into the organization, improves communication with supporting disciplines (docs, localization, support, sustained engineering), and so on.

Agility

In this model, agility is achieved by having an appropriately-sized cycle. For example, for a desktop OS the right cycle may be 6 months or 12 months. In contrast, a web site would probably need to have a release valve every 2 to 4 weeks. Agility doesn’t mean lack of planning – agility means a reasonably short cycle capable of getting high quality products out in a predictable, controlled fashion.

Process Summary

The process of developing a large software project can be organized in 4 major phases (see Fig. 1 below):

MQ
Pre-planning and Planning
1 or more Development Cycles
Stabilization and Release

(A) and (B) happen simultaneously. (A) is “Milestone for Quality” targeted at constructing a high-quality “software production factory”. This milestone gives the engineering organization a chance to optimize and simplify the tools and processes that it uses to product the product. (B) follows a predefined planning process. Planning is a whole other topic in itself. (C) consists of one or more close-looped development cycles.

Figure 1 Development Process Outline

Obviously this is just a general outline. The size of the different phases has to be determined by the development organization, based on deep understanding of what needs to happen in these phases and on the current state of the business and of the development factory. Some phases (specifically A and B) may shrink if there is no need for them.

Below I show an actual implementation of this general outline. Each small rectangle represents one week. Weeks in green are “stabilization weeks” (stabilization has different meaning for the different milestones – see tables below). Note again that the size of the actual milestones and phases that I show below are just a specific implementation of the general outline. You can have development cycles that are as short as a week, if you think that the ratio of dev time to process tax associated with managing every cycle is reasonable. You can also have longer cycles, although I personally wouldn’t recommend cycles that are significantly longer that 18 weeks or so.

Figure 2 Development Process - Detailed View

With these sizes we have…

About 18 weeks (4 months) of pre-planning and planning activities.
One or more dev cycles of 15 weeks (3.5 months)
Final stabilization of 10 weeks (2.5 months)

I.e. a release with 1 dev cycle takes about 9.5 months; a release with 2 dev cycles takes about 13 months, etc.

The number of the actual development cycles can vary and is a purely business decision. At the end of every development cycle, the organization delivers shippable functionality. When management deems that the available payload makes sense as a next major version of the product, they switch the team to the RTM (RTW) phase, and drive to an actual release.

Before we delve in MQ, it’s important to note that all of these phases are divided in “red weeks” and “green weeks” (a block denotes a week). “Red weeks” are weeks of active development, “green weeks” are weeks of stabilization. Obviously the actual activities that happen in these weeks depend on the phase of the project.

MQ

MQ is the Quality Milestone. The role of MQ is ensuring that we have a working high-quality “production factory” before we embark on producing the next version of the product.

Figure 3 MQ and Planning

The breakdown of MQ is as follows:

Figure 4 Breakdown of MQ

MQ is all-inclusive collaborative effort between the 3 disciplines. The majority of PM, Dev and Test will be working on MQ activities during MQ. A few PMs and Devs will be working on planning / prototyping in parallel. MQ is planned and rigorously project-managed. MQ has a set of exit criteria that the org needs to hit before exiting the milestone.

The Dev Cycle

All features are developed during the dev cycle, which looks as follows:

Figure 5 Dev Cycle

There is a single prioritized feature backlog that captures all features that are being worked on in the current development cycle or can be worked on in future cycles. The list of features that comprise the current development cycle is stable throughout the cycle i.e. Management does not change the list beyond M0 – they have to wait for the next M0.

Any feature that cannot be completed is “cut” until the new dev cycle. Features that take more than one dev cycle to complete will be split into smaller manageable “featurettes” can be completed within the constraints of a dev cycle. Several featurettes may comprise the final feature. At the start of the development new cycle the feature is re-evaluated again against the other features in the backlog.

The start of every new dev cycle is a clean slate. All bugs and features are automatically “cut” and need to be reevaluated and brought back (can be evaluated and brought back during)

IMPORTANT: The development cycle is a contained unit of delivery of added value. We can have as many development cycles as necessary to build a compelling value for the “big release”, but every one of them ends with high quality features that are ready to ship.

The breakdown of the development cycle is as follows:

Figure 6 Breakdown of a Dev Cycle

The M0 to M4 cycle can be repeated multiple times within a release. Every time in M0 (but only in M0), executive management gets to redefine the feature backlog.

Stabilization and Release

The final phase in the development process (disregarding the various post-release activities for now) is the “Stabilization and Release” phase or “RTM”.

Figure 7 Stabilization and Release (“RTM”)

This phase is about fixing bugs, integration (if necessary) and managing the final quality of the release. The breakdown is as follows:

Figure 8 Breakdown of Stabilization and Release

QFEs (Hotfixes)

Every large software project needs to have a mechanism to deal with hotfixes (also known as QFEs). The nature of these fixes requires reactive delivery (as opposed to the proactive approach described above). Teams typically maintain a separate source code branch (or branches) for QFEs and that is typically the same as the branch that contains the currently shipping version of the software.

Due to their reactive nature QFEs are delivered on demand with a release schedule that looks as follows:

Figure 9 QFE Release and RI Schedule

Typically, fixes introduced in the QFE branch must make their way into the Main product branch, which is done with a series of regular reverse integrations (or RIs) as shows above. QFEs also get packaged as larger packages and released as service packs (or SPs).

Every QFE is subjected to a root-cause analysis, following a simple standard template, to help the team avoid similar situations in the future.

Templates and Artifacts

Last but not least I wanted to provide a quick list with some of the fundamental documents that are required to bootstrap a development process for a larger product. Note that this list is nothing more than a reasonable starting point – at least according to me.

Figure 10 Key Templates and Artifacts

Conclusion

There are many ways to ship software products. Large products require more pampering and more scaffolding, at least until a critical mass of the team adopts a coherent product development culture. If you want to ship high quality products, you need to invest in rationalizing your product development cycle. The best way to do that is to plan, using a series of thought experiments (e.g. “imagine that we have achieved code complete, what are the steps from CC to release ready and how long are they realistically going to take”), and a guiding principle of lowest possible overhead costs and highest possible simplicity.

Updating Your WP7 (and Dealing with Error 800705B4)

ivom1 — Thu, 21 Apr 2011 23:44:00 GMT

I have had a Samsung Focus WP7 device for several months now and I absolutely love it. I love the hardware (fantastic bright display, sleek look, etc.), I love the Metro UI, and I love the development tools. VS 2010 and Silverlight for WP7 are a ton of fun.

Like many of you I patiently waited for the February and March updates and just yesterday, at the end of April, I finally got prompted to install them. Unfortunately, as I hooked up the phone to my PC to update it, I hit error 800705B4.

Error code 800705B4 resolves to the following:

0:000> !error 800705B4 Error code: (HRESULT) 0x800705b4 (2147943860) - This operation returned because the timeout period expired.

Since it took me a bit of digging to recover from it, I wanted to share here the fast way to overcoming this error:

Go to http://support.microsoft.com/kb/2530409 and download the Windows Phone Support Tool for your PC platform.
Follow the instructions in the tool. You won’t lose any of your information.
After the tool is done recovering your phone, hook it up to Zune again and get the update successfully.

Hopefully, that’ll save folks out there some time reading through MSDN docs.

TestApi v0.6 Released!

ivom1 — Tue, 08 Feb 2011 00:19:12 GMT

I am happy to announce that we have just released version 0.6 of TestApi – the API library for testing – on Codeplex. Version 0.6 contains the following additions and modifications:

Starting with this release, we are moving all of the development of the library to Codeplex. We have also moved the solution to VS 2010;
Jared Moore made several important modifications to our Fault Injection API, including but not limited to support for in-process fault injection, support for MethodInfo-based method identification, various performance improvements and bug fixes. In addition, we are releasing the complete source code of the unmanaged FaultInjectionEngine.dll, which now builds cleanly at warning level 4;
Dennis Deng extended the Input Simulation API with support for injection of non-ASCII strings in Keyboard.Type(…). We have also added support for Mouse.DragTo(…);
We have added support for 64-bit in our Leak Detection API;
Andrey Arkharov extended the Object Comparison API with support for encoding and decoding object graphs from arbitrary streams, annotating object graph nodes with custom comparison strategies and type-specific object graph factories, together with several additional tests;
We extended our String Generation API with support for string generation from regular expressions. We have also added a CommonRegularExpressions class, to speed up test development;
We have added lots of additional tests, and have refactored the existing tests into namespaces for better discoverability.
Last, but not least we have made some important API documentation improvements – the API documentation now sports namespace docs.

Check out the TestApi v0.6 release notes for a more detailed version of the above. Then let us know what you think and vote for features.

Registering Your WP7 as a Developer Device

ivom1 — Wed, 08 Dec 2010 07:12:00 GMT

Update: Shortly after I posted this article, Peter Henry noted that he had a very similar article out already. Here is the link to it – it has good additional information – check it out: http://www.pchenry.com/Home/tabid/36/EntryID/389/Default.aspx

So you got your Windows Phone 7 device, you downloaded the free development tools and now you are itching to write your first “Hello World” app for WP7. You fire up VS 2010, create a quick phone app, connect your WP7 device to your development PC, select “Windows Phone 7 Device” as a target and attempt running your first app on your phone.

You will get the following message…

You have to make sure your device is ”developer unlocked” before you can deploy applications to it. Here’s how you unlock your device:

Connect your device to your PC through USB.
Start the Zune desktop software.
Start the “Windows Phone Developer Registration” software.

At this point, provided you have a developer account (which you get on http://developer.WindowsPhone.com) you can register your device. Once registered you’ll be able to use it for development purposes.

Easy! (not necessarily straightforward though :) )

Performing Code Reviews in TFS 2010

ivom1 — Mon, 29 Nov 2010 09:15:00 GMT

The past few days I moved the development of TestApi to the Codeplex TFS servers. Previously, we used to develop TestApi in the internal WPF branch at Microsoft, which had become a bit of a drag due to the need to support contributors from various teams across the company.

As part of the move, I needed to set up a code review process. Turns out that it’s a bit of a struggle to figure out how to set up a simple CR process using standard tools – mostly due to poor documentation – so I thought I’d share what I did in this post. Hopefully, that will save someone out there some time.

Code Review Process

The general CR process goes as follows:

The developer creates a new shelveset (see the TFS documentation for a definition of “shelveset”) with his/her changes and sends the shelveset for CR.
The code reviewer performs the CR, sending back feedback.
The developer revises his/her code based on the CR feedback and generates a new shelveset.
Steps 1-3 get repeated until the code reviewer approves the code change for checkin.
The developer checks in the code.

The tools we use for CR are VS 2010 itself and WINDIFF. The latter is a diffing tool used for step 2 of the CR process.

Initial Setup

To set up WINDIFF as your diff-ing tool, in VS 2010 go to Tools > Options…, scroll down to the Source Control item and configure your compare tool to be WINDIFF as shown below:

Shelving

Shelving is the act of storing pending changes on the TFS server (see this link for more). To create a shelveset, in Source Control Explorer, right-click on the folder, containing all pending changes, and choose Shelve Pending Changes… in the displayed context menu. This will bring up a dialog that you can use to shelve your changes.

Once you shelve your proposed changes, send an email to the code reviewer, with your username and the name of your shelveset.

Unshelving, Diffing and Code Review

When you get a CR request, go to File > Source Control > Unshelve Pending Changes… and in the displayed Unshelve dialog, discover the shelveset that you need to review and click Details…

In the displayed Shelveset Details dialog, right click the file you want to review and compare it to its unmodified version. This will trigger WINDIFF as a comparison tool. Record your comments in an email and send them back to the developer (or approve the checkin if you agree to it).

Conclusion

That’s pretty much it. Obviously, this is a poor man’s solution, but it’s certainly better than nothing. There are a bunch of more advanced solutions out there providing code annotation, review workflows, etc. that you may find valuable too. This post by JB Brown may be a good starting point for those who want to go further.

Principles 4: End-to-End Development Process

ivom1 — Fri, 10 Sep 2010 09:11:34 GMT

Series Index

+++

This fourth post discusses an end-to-end development process. Obviously, there are many ways to develop software. While a lot of the basic principles apply almost universally, every sufficiently autonomous organization needs to figure out what works for itself, and then progressively refine it.

It is important to note that this process is the same process we used to manage several projects and has resulted in delivery of high-quality products, in a controlled manner, on time (actually ahead of time in a few cases). In other words, this is a real, proven process, not a theoretical construct. I’ll actually show two versions of the process – a simple one - for smaller orgs, and a more involved one - for larger orgs.

Guiding Principles

Before going further, I want to once again reiterate the Principles of Software Development. These principles are the underlying “Constitution”. In summary, they are:

Everybody on the team fully understands the plan.
Deliver features in a gradual, close-looped manner.
Define release criteria at the start of the cycle. Track visibly.
Exercise defect prevention – whenever possible.
Have a stable, fully-transparent, committed schedule from the start.
No code while planning the “What”. No code without spec.
Fully working and timed setup on day 1 of M0.
Strict and reliable reverse- and forward-integration gates from day 1 of M0.
Start together, work together, finish together.
No work happens outside of the main development cycle.
Every major system has an owner.

Goals

At the high level, we want to be able to produce high-quality, compelling software, in a well-organized, intentional, cost-effective, and predictable manner. In order to pull that off, we need the following:

Commitment, at all levels of the organization, to “do the right thing”. Full management sponsorship.
A well-designed organization;
A well-designed “business cycle”;
A well-designed development process;
A well-designed and working “engineering factory”;
A high-powered and high-morale team;

This post talks about “D”, but it goes without saying that “A” to “F” represent a system of connected vessels and need to be designed, implemented, and refined together.

This post also makes the assumption that you have a working “Planning Process” that generates compelling product value propositions. Clearly, without a compelling product that addresses a customer need, it doesn’t matter how excellent you are in delivery. There are various ways to plan a product. At Microsoft, we use a bunch of different approaches, with a lot of folks now gravitating towards the so called “Sinofsky model”. At any rate, this post does not discuss the planning process (I may do this in a future post).

In order to deliver a high-quality product, you need to plan and design before you code. On the other hand, you don’t want to get stuck in planning for a prolonged period of time, because time may erode the value prop of your product and because you need fast and frequent customer feedback – you may be working in the wrong direction. Clearly, we need to “marry” these two competing priorities and clearly this is an optimization problem. What follows is our solution to the optimization problem, but solutions will vary based on specific product, business, industry and team considerations. Treat this as a starting point – take it and make it your own.

Process 1 – Simple Projects

We used this simple version of the process to develop TestApi – the API library for test and tool developers. TestApi is a relatively small and simple project. With this process, we released five versions of the library and we released all of them on time or ahead of time.

The development schedule looks as follows:

Figure 1. Simple development schedule

The schedule consists of 5 major milestones:

M0 – Planning – in this phase, we finalize the list of features we’d like to deliver in the release. Every feature has an owner. All owners understand the schedule. The necessary prototyping is done. At the end of M0 we have a backlog for the release, to which we have committed. Once we commit to the backlog. We do not allow feature creep after the end of M0.
M1 – Design – in this phase every owner delivers a “conceptual document” that explains the usage of their feature accompanied if necessary by a design document. In the case of an API library such as TestApi, this is a API usage document and API design document. The usage and design gets reviewed and approved by the Architecture review body (3 people in the case of TestApi). Once the design has been approved, the owner checks in the empty stubs (e.g. of the API) and the corresponding acceptance tests, together with an initial version of the XML documentation and gets stubs, tests and documentation to build cleanly. The documentation (both docs generated from XML comments as well as the conceptual documentation) gets sent to the Documentation team for editing. The Legal team (LCA) gets engaged.
M2 – Implementation – in this phase, the owner implements the tests, code, fixes the documentation (based on editorial feedback from the Docs team) and drives to ZBB (zero bugs)
M3 – LCA and Samples – in this phase the owner implements stand-alone samples (if applicable) and discussed the product with LCA (legal) if applicable. Legal signs off.
M4 – Stabilization and Release – final stabilization, release checklist (we have a predefined checklist) and final release.
Prep – At the end of every release cycle, we have about 4 weeks of “padding”. During that period, we do various post release activities (blogging, etc.), we invest in our engineering “factory” and we engage on preliminary planning for the next release (identifying possible participants, defining dates, etc.) This is a fairly relaxed period and that is by design.

All features associated with a release are tracked in a table similar to the TestApi Feature Backlog table presented in this post:

Figure 2. Feature readiness checklist

All source code is static analysis clean by default (FxCop, StyleCop, etc.) – checking in a violation results in a build break. There is a FAQ for commonly asked questions (e.g. “I can’t be done with a feature on time, what do I do?”, etc.) that evolves with every release.

As a general rule, the timelines above are rigorously adhered to. If you “fall off the train” (meaning – fail to meet a deadline), you don’t get back on the train until the next version (which is always no more than 3 months away, and of course you tend to deliver larger payload for it). We have a dedicated project manager who drives every release (for TestApi that was Alexis Roosa).

The tasks associated with every milestone are done in the order in which they are presented here. For example, we write the acceptance tests against the empty stubs before we write the actual code. Another thing that should be obvious is that we spend a lot of effort on planning and design. We are extremely rigorous in our design review, we look at the API from various different angles (we consider different options, we discuss naming, usage, simplicity, statelessness, future evolveability, etc.) employing both scenario-based design and producing various short-lived and long-lived documents. We have found that this saves us a lot of time during development and significantly increases the confidence of our dates.

Simple, Predictable, Agile

The simplicity and predictability of this process have a lot of positive effects – from team morale and cohesion to delivery on time with high quality. To date, we have always been done with M4 on time or ahead of time! The process does, however, have a waterfallish feel, so every once in a while I get questions about agility. Put simply, we believe that design and planning is important, but so is time to market and getting frequent customer feedback. We resolve that tension by having short release cycles (3 months or less). There is noting that is preventing us from making these cycles even shorter, but we have found that 3 months works well for us.

My experiences shows that you can’t achieve agility by cutting proven engineering practices such as solid design and code reviews – that only leads to fire-drills, death marches and low quality. You achieve agility by having a reasonably rapid development cycle, aligned to solid business considerations.

To Be Continued…

In my next post, I will introduce a process that is suitable for larger, more complex products and organizations.

The 3 Questions

ivom1 — Thu, 05 Aug 2010 05:13:00 GMT

A key goal for every manager is building a happy, productive team. One technique I use when building teams is asking folks on the team the following three questions:

Q1. What do you do, and like doing?
Q2. What do you do, and don’t like doing?
Q3. What do you not do, but would like to start doing?

I ask folks to give these questions some thought and ideally send me a bulletized list of items before we meet in person to discuss them. I explain that I will try my best to minimize the items that fall under Q2, and maximize opportunities for items that fall under Q3.

Even though this technique is extremely simple, I have found it to be a great tool to anchor and guide career conversations. It provides just enough structure to facilitate the conversation, for a very low overhead cost.

Clearly, this technique also gives me the ability to intelligently balance responsibilities within the organization. Very often the old adage that “one man’s garbage is another man’s treasure” proves to be true, which of course makes everybody in the org happier. And happy people are productive people.

Principles 3: Principles of Software Development (aka Principles of Shipping)

ivom1 — Mon, 02 Aug 2010 05:36:00 GMT

Series Index

+++

The third and fourth posts from my series on “Principles” outline a system of shipping of software, that takes into account the various lessons I have learned during my years as a manager at Microsoft, dating back to the first product I worked on – Windows XP. The post lists a set of guiding engineering principles that result in stable, high quality releases, delivered on time.

Before we jump into the principles, I want to call out a few things:

No exceptions to the rules: There is no such thing as partial implementation of these principles. Even though I take a lot of time socializing these principles in the organizations I manage and worked in, I and my managers tend to take a fairly assertive approach when implementing these principles and we apply them fully, 100% of the time. Yes, it takes a while to socialize these and yes, we get a few “you are not being agile” (see more on “agile” at the end of this post), “you don’t understand our reality, we are a special organization” and “can we make an exception for XYZ” comments. We still treat these principles as a Constitution and we (management) support them 100% of the time.
Real products get shipped with these principles: Everybody wants to ship a high-quality product, with compelling customer value, on schedule or ahead of schedule. As involved as these principles may sound, they are a very simple scaffolding that gets you to ship predictably with high quality.
Top management sponsorship, commitment and calibration is key: You cannot really achieve any significant change in an organization unless you have first-class sponsorship and commitment by the top of the organization. The top management team needs to be committed as well as well-calibrated i.e. different members of the top management team should come up with consistent answers when approached with the same problem.
Commitment eventually leads to organic adoption: Solid principles and processes inevitably lead to cultural changes in the organization. As managers, that’s exactly what we want to achieve – a cultural shift in the organization that results in organic rejection of bad practices, and an appetite for continuous improvement.

+++

Everybody on the team fully understands the plan.

For a team to be successful, everybody on the team needs to (at all times) understand “the plan” for the release and his/her role in its various cycles and milestones. The litmus test here is asking a random person on the team what “the plan” (schedule, owners, feature set) is.

Deliver features in a gradual, close-looped manner.

A common theme during software releases is a fire-drill towards the end of the release, due to the syndrome of “biting more than we can chew” and due to managing the deliverables in an open-ended, “agile” fashion. One way to overcome this situation is ensuring that we deliver features in smaller “sets”, on a more frequent cadence. Every “feature set” represents a meaningful collection, which gets developed and stabilized together before proceeding to the next set.

Thus, we end up with a number of “feature set development cycles”, producing stable sets that integrated together comprise our release. I like to use the metaphor of “putting features in the freezer” i.e. we start working on the new feature set only when we have gotten the current feature set to high shippable quality.

Define release criteria at the start of the release cycle. Track regularly.

This sounds like a common sense “must do”, but it is regularly not done sufficiently well. The release criteria generally fall into two groups: (a) RI release criteria, necessary to RI (reverse integrate) a feature into the main product branch and (b) shipping release criteria, necessary to release a feature or a set of features. We need to have the complete list of both before we check in a single line of code. That helps us understand the true cost of every feature.

Defect prevention – whenever possible.

A lot of our effort is spent on generating and fixing defects that could have been prevented in the first place. We need to make heavy use of all available tools in an attempt to prevent as many defects as possible from entering the product in the first place.

Examples of such defect prevention opportunities are enabling of tools such as FXCOP in the build environment, by default.

Have a stable, fully-transparent, committed schedule from the start.

Another common challenge in software releases is the lack of a stable, transparent, predictable schedule. Our product schedule mimics our features. It starts as a very “low-quality” feature, progressively solidifying throughout the release, as a result of “we are not on track” realizations.

I have heard claims that the schedule of the parent organization (division, company, etc.) is too “fluxy” which in turn affects the ability of the team to build a predictable, stable and transparent schedule. Even if we assume that this claim is true (it probably is, but it should not be – large companies and divisions prove that they can ship with a stable schedule), the local team still has the ability to be in control of its own fate by defining stable sub-schedules to combat randomization from the larger organization.

Having a stable, transparent, fully-internalized and fully-embraced schedule is a key ingredient to the success of any product – big and small.

No code while planning the “What”. No code without spec.

PM spends the first milestone in the product cycle (MQ – the quality milestone) on defining the “what” for the release. Dev and Test complete engineering debt work, preparing the engineering system for execution. While there may be limited prototype code generated during this milestone, none of this code is allowed to be reverse-integrated in any official branch. Dev and Test should spend the majority of their time stabilizing the engineering system (builds, static analysis tools, unit tests, functional tests and infra, stress tests and infra, performance tests and infra, meaningful reverse-integration tests to prevent other teams from breaking us, etc.) Having a stable factory will ensure producing a stable product.

Dev and Test start working together on the features in M0 (not earlier) once the “what” has been defined, scratching any previously generated code.

Fully working and timed setup on day 1 of M0.

Another common problem teams face is Setup instability that typically carries on, way into the release. Not having a stable Setup complicates significantly lab operations and prevents the team to self-host the build and discover issues early, so it is something that needs to be addressed.

Setup must always be stable and must visibly display the important metrics that we are optimizing against. Metrics such as speed (of setup), number of custom setup actions, etc. should be visible to the whole team from the start.

Strict and reliable reverse- and forward-integration gates from day 1 of M0.

If we believe that reverse- and forward-integration (RI and FI) gates have a valuable role in ensuring the health of the team branch, then we must ensure that the gates that we have in place are reasonable and that they are strictly adhered to. This simplifies immensely feature development, by defining a simple “rule” for RI-ing and FI-ing code that can be easily understood and embraced by the team.

No code changes in the team branch will be allowed without a fully operational test and validation systems (unit tests passing, test infra running, functional tests at 99.5% pass rate, Perf lab running, Stress lab running, Code Coverage generating results).

Start together, work together, finish together.

A company or a division ships a successful product by getting all team to adhere to the same fundamental schedule and gates. A development organization can and should be doing the same. Having a single schedule with a single set of expectations for all teams simplifies management of the release and creates synergy between the feature owners.

Some claim that “agility” is achieved by having different feature teams work against different schedules. My experience is that such pseudo-agility results in a fragmented organization, that is confused about its own deliverables and that delivers low-quality products, because products are not built with the same rhythm. Management of the product deliverables in such situations often degrades to “managing exceptions” as opposed to “managing by the rules”.

We can and will achieve agility through the power of contained, incremental milestones, not through randomization.

All teams in the development organization should start together, work together and finish together, adhering to a common schedule, RI gates and exit criteria. All disciplines (Dev, Test, PM) should do that too, working side by side as opposed to serializing work.

No work happens outside of the main development cycle.

“Out-of-band” releases and “off-the-books” investments are good and necessary, but they must be planned as all other work and managed within the constraints of the main development cycle.

This means that they never appear randomly in the middle of a development cycle (they can appear on the backlog to be considered for the next dev cycle). This includes dev work, test work, prototypes, forum engagement, conference prep work, etc.

Every major system has an owner.

Every major system on the team (“system” being a generic name for anything that needs an owner) has clearly defined and documented ownership. This includes, but is not limited to fundamentals, lab systems, build systems, test systems, infrastructures, etc.

+++

A final word on AGILITY.

AGILE is a magic word in our industry these days that gets thrown around somewhat lightly in my opinion. Some people hide behind the term AGILITY to justify the lack of basic structure, basic process, and basic planning adequacy in their organization. Invariably such pseudo-agility leads to constant fire-drills, regular death marches, and the eventual burn out of the team.

It is my strong belief that every project and team needs basic “scaffolding”. Larger projects and teams tend to need a bit more of that, at least until the scaffolding gets transformed into “team culture”.

Clearly, the team leaders have to be accountable for providing that basic scaffolding. Success is easy to define – if you have a happy, productive organization that delivers high-quality products on time, then you have provided the necessary basic structure.

I am of course aware of the goals of the agile movement, which I am very much supportive of. The best way to achieve agility in my opinion, is by sizing of your development cycle appropriately – a technique that I will demonstrate in my next post.

Introduction to TestApi – Part 8: Object Comparison API

ivom1 — Thu, 29 Jul 2010 19:48:00 GMT

Series Index

+++

Comparing two objects for equivalence is a relatively common task during test validation. For example, you might test whether a type follows the rules that are required by a particular serializer by saving and loading the object and comparing the two. In a deep object comparison, all the properties and their sub-properties are compared repeatedly until primitives are reached. The .NET Framework provides mechanisms to perform deep comparisons, but requires the types to implement part of the comparison logic (IComparable, .Equals). Obviously some types do not use these mechanisms. This API provides a mechanism to perform a deep comparison of two objects by using reflection, regardless of whether or not they implement IComparable.

TestApi Object Comparison

TestApi provides an API to create compare operations. The API separates the comparison operation into two parts:

ObjectGraphFactory: Builds an intermediate representation from the object that describes what needs to be compared.
ObjectComparer: Uses a reusable compare mechanism to compare the intermediate representations of the objects, generated by the factory.

Example #1

This example demonstrates how to use the object comparer to compare two objects. Assuming we have the following simple class...

class Person
{
    public Person(string name) 
    { 
        Name = name;
        Children = new Collection<Person>();
    }
    public string Name { get; set; }
    public Collection<Person> Children { get; private set;  }
}

… we can compare two objects using the following code:

Person p1 = new Person("John");
p1.Children.Add(new Person("Peter"));
p1.Children.Add(new Person("Mary"));
Person p2 = new Person("John");
p2.Children.Add(new Person("Peter"));
// Perform the compare operation
ObjectGraphFactory factory = new PublicPropertyObjectGraphFactory();
ObjectComparer comparer = new ObjectComparer(factory);
Console.WriteLine(
    "Objects p1 and p2 {0}", 
    comparer.Compare(p1, p2) ? "match!" : "do NOT match!");

Execution of this code produces the following output:

Objects p1 and p2 do NOT match!

Example #2

The following example demonstrates how to obtain the details of where the two objects do not match.

Person p1 = new Person("John");
p1.Children.Add(new Person("Peter"));
p1.Children.Add(new Person("Mary"));
Person p2 = new Person("John");
p2.Children.Add(new Person("Peter"));
// Perform the compare operation
ObjectGraphFactory factory = new PublicPropertyObjectGraphFactory();
ObjectComparer comparer = new ObjectComparer(factory);
IEnumerable<ObjectComparisonMismatch> m12 = new List<ObjectComparisonMismatch>();
Console.WriteLine(
    "Objects p1 and p2 {0}", 
    comparer.Compare(p1, p2, out m12) ? "match!" : "do NOT match!");
foreach (ObjectComparisonMismatch m in m12)
{
    Console.WriteLine(
        "Nodes '{0}' and '{1}' do not match. Mismatch message: '{2}'",
        m.LeftObjectNode != null ? m.LeftObjectNode.Name : "null",
        m.RightObjectNode != null ? m.LeftObjectNode.Name : "null",
        m.MismatchType); 
}

Execution of this code produces the following output:

Objects p1 and p2 do NOT match!
Nodes 'Children' and 'Children' do not match. Mismatch message: 'RightNodeHasFewerChildren'
Nodes 'IEnumerable1' and 'null' do not match. Mismatch message: 'MissingRightNode'
Nodes 'Children' and 'null' do not match. Mismatch message: 'MissingRightNode'
Nodes 'Count' and 'null' do not match. Mismatch message: 'MissingRightNode'
Nodes 'Name' and 'null' do not match. Mismatch message: 'MissingRightNode'
Nodes 'Count' and 'Count' do not match. Mismatch message: 'ObjectValuesDoNotMatch'

Example #3

The following example demonstrates how to create a custom object graph factory which determines which nodes in the object graph to compare.

For simplicity, we construct a factory that disregards any children.

class SimpleObjectGraphFactory : ObjectGraphFactory
{
    public override GraphNode CreateObjectGraph(object o)
    {
        // Build the object graph with nodes that need to be compared.
        // in this particular case, we only pick up the object itself
        GraphNode node = new GraphNode();
        node.Name = "PersonObject";
        node.ObjectValue = (o as Person).Name;
        return node;
    }
}

Then, we utilize the factory to generate graph representations of the two object we want to compare.

Person p1 = new Person("John");
p1.Children.Add(new Person("Peter"));
p1.Children.Add(new Person("Mary"));
Person p2 = new Person("John");
p2.Children.Add(new Person("Peter"));
ObjectGraphFactory factory = new SimpleObjectGraphFactory();
ObjectComparer comparer = new ObjectComparer(factory);
Console.WriteLine(
    "Objects p1 and p2 {0}",
    comparer.Compare(p1, p2) ? "match!" : "do NOT match!");

Execution of this code produces the following output…

Objects p1 and p2 match!

… which is the expected result, given the fact that our simple factory completely disregards the Children of Person.

Conclusion

Comparing of objects is a common problem in automated test development and TestApi provides a simple generalized and extensible solution.

Principles 2: Principles of Software Testing

ivom1 — Fri, 23 Jul 2010 22:06:00 GMT

Series Index

+++

This is part 2 of my series on “principles”. In this post, I talk about the principles I have used to guide all teams I have managed. Before we delve in – a couple of clarifications:

I have skipped some of the obvious basic principles such as “strive for high levels of automation”, “integrate testing into your development process”, etc. I know that a number of organizations are still struggling with the basics. If you happen to be part of such an organization, grab a book on software testing before reading this post.
The majority of the principles you will read below are common sense. Some of them, however, seem to contradict common sense and definitely contradict what you’ll read in books on testing. Don’t trust everything you read (including this little gem)... But consider the fact that we used these exact principles to ship several versions of WPF and Visual Studio.

The principles, which I cover, are:

High test stability (99.5%)
- Driving testing upstream
- No major investments in failure investigation tools
- Random configuration management
- Devs and Testers use the same test harness
High test performance
- Fast builds
- “No BVT” policy
Minimum dependencies
- All test metadata is in the source control system
- Test code, dev code and specs are checked in the same source depot
- Enable a “build-xcopy-run” testing workflow through zero server dependencies
Fundamentals, Fundamentals, Fundamentals
- Performance and Scalability
- Stress and Stability
Test Development Principles
- Focus on design and reuse
- Componentization and aggregation (“Lego blocks” mentality)
- Automated code verification as much as possible
- Focus on End-To-End Scenarios
Continuous Training

Enjoy!

+++

High test stability (99.5%)

High test stability is the cornerstone of every good test automation system. Ideally, a test should fail only when there is a product bug. In reality, tests tend to fail for other reasons too, but we should strive to minimize those failures by either fixing or removing (and rewriting) unstable tests.

The principle of high test stability (99.5% test stability) has several important implications. It allows us to spend less time chasing random test failures and more time discovering bugs in the product. It also allows us to increase test coverage faster than increasing support costs. It helps us land the product in a stable, predictable manner. But most importantly, high test stability allows us to drive testing upstream.

Driving testing upstream

Defect prevention is the “holy grail” of quality management. There are many studies that show that preventing a defect of entering the product is a much cheaper option that discovering and fixing the defect after it’s has been introduced.

Obviously there are many different ways to go about defect prevention. One of the simplest and most robust one is having a stable and fast suite of tests that can easily be run by developers prior to check-in. The suite has to be stable – because no one likes to debug through 100s of unrelated failures to confirm that the code works. The suite has to be fast, because no one likes to wait for hours for test results to come out.

Having a stable and fast test suite allows us to practice a continuous “develop – run tests – fix issues – repeat” cycle prior to check-in.

No major investments in failure investigation tools

About 5 years ago, we used to invest a lot in advanced failure investigation tools, which allowed us to quickly deal with and resolve large numbers of test failures. It turned out that having high test stability negated the need for having advanced failure investigation tools. For the most part, a NUnit-like report (typically XML with a XSLT transform to make the data readable) is all the team needed. At this point of time, I assert that having an advanced failure investigation tool may actually be a dangerous practice for test teams as it acts as a powerful crutch that pulls the team away from the root underlying issues that need to be fixed – i.e. tests with low stability or a very low quality product.

There is one important aspect of failure investigation tool development that we need to account for though. SDETs are fundamentally software engineers. As software engineers we like to develop software and a failure investigation tool often presents opportunities to hone one’s design and coding skills and to also experiment with new technologies (web services, databases, extensibility frameworks, AI, GUI frameworks, etc.) We clearly have to find ways to expose our SDETs to other opportunities to improve their transferable skills (design, algorithms, etc).

Case Study: 99.5% pass rate of the WPF tests

99.5% is obviously an arbitrary number. The right number to strive for is always 100%. A realistic stability goal depends on the size of the team and on the size of the test suite, but typically revolves around 99.5%. For example, in the WPF test team, we had a suite of about 105,000 tests which at 99.5% pass rate produced about 500 test failures for a team of about 30 engineers or about 18 failures per person, which seemed to be a reasonable number (so that every SDET spends no more than about 30 minutes investigating failure every day).

With time, 99.5 became a team mantra. SDETs, SDEs, PMs actively identified with it and fought for it.

Random configuration management

At Microsoft, we serve hundreds of millions of customers with great environment variability. A typical “test matrix” consists of a number of HW configurations, OS platforms, languages, locales, display configurations, etc. and typically contains millions of variations.

Teams typically do the following to deal with the “configuration matrix explosion” problem:

Use matrix reduction techniques (“pairwise variation generation”, etc.)
Prioritize configurations (e.g. “English, x86, FRE, 96 DPI” is higher priority than “Portuguese, x64, CHK, 140 DPI” – no offense to Portuguese folks, I myself am Bulgarian and I have to admit that in the general case English is higher pri than Bulgarian).
Create and manage a deterministic configuration testing schedule (today we cover “English”, tomorrow we cover “German”, etc.) in an attempt to cover all high priority configurations.

Technique (b) specifically often has the unfortunate effect of getting us to test mostly on “vanilla configurations”, which results in missing bugs until late in the testing cycle and results in “training” the tests to pass on vanilla configurations only. Technique (c) tends to result in high testing schedule management costs.

An easy way to combat the undesired effects of (b) and (c) is to switch to a weighted random configuration management. The weight of a certain config can be calculated dynamically based on historical data (pass rate of the last time the config was run on, and frequency of running).

One can even build predictive models that would allow the team to search for configurations that result in large number of bugs.

Case Study: WPF test execution matrix management

Prior to early 2007, the WPF used to invest a considerable amount of effort to plan day-to-day test execution. The team had a dedicated test lead who managed the test execution matrix and sequence. Introduction of a new OS to the matrix was a fairly disruptive event, necessitating a regeneration of the whole execution sequence. Test passes took a long time, because of completeness concerns.

At the beginning of 2007, the WPF test team switched to weighted random configuration management. We introduced a simple Excel spreadsheet, which was used by the lab engineers to generate a random testing config every day that was then installed in the lab and used for test execution. The team identified about 20 configurations as high priority configurations. 4 out of 5 days in the week, the team used a randomly selected high pri config. Every 5^th day or so, the team explored the rest of the configurations.

The switch to random test config generation removed the need for test execution scheduling, resulted in additional test stabilization (because the tests were not trained to pass on vanilla configurations) and enabled the team to find bugs off the beaten path earlier in the development cycle.

Devs and testers use the same test harness

Developer unit tests use exactly the same test harness as the functional tests developed by SDETs. This enables code reuse between the unit tests and the functional tests. It also enables developers to easily run and debug functional tests (because they know how to do it), thus enabling defect prevention.

High test performance

Automated tests are programs that get executed multiple times throughout the lifecycle of the product. Having slow tests results in the following problems:

Delays bug discovery (because test runs take longer, need to be incomplete due to their length, etc);
Precludes the team from effectively employing defect-prevention techniques such as TDD, test before check-in, etc;
Wastes machine time and electricity;
Creates test management complexity - splitting the test suite in different priorities (BVTs, “nightlies”, etc) and creating the need for managing these different sub-suites, etc.

Fast Builds

One area that often gets overlooked is the speed of building the tests compared to the speed of building the product. Tests often take much longer to build because they are not properly packaged (e.g. too many DLLs) and because no one really looks into improving the build times. In reality having a fast product and test build enables defect prevention.

“No BVT” Policy

In theory, the idea of BVTs (build verification tests : a sub-suite of high priority tests that get run more often than the rest of the tests) sounds good. In practice, BVTs tend to become just another “crutch” that typically prevents test teams from addressing the root underlying problem of test slowness and instability[1]. Introduction of the notion of BVTs also introduces various “management costs” (suite management and curation, execution schedule management, etc.) that are moving the focus away from more important activities directly related to the quality of the product.

So I highly discourage the use of BVTs.

Case Study: WPF BVTs

In the WPF test team, we experienced the full cycle. We started with a suite of tests that we used to run every day. As we expanded the suite of tests, we saw that the run times of the suite became longer and that the stability of tests became lower. Instead of investing in fixing test perf, stability and duplication of coverage, we decided to segment the suite into P0s, P1s, P2s, etc. We created various processes to handle bugs produced by P0 tests, etc. (the “hot bug” concept). Because BVTs were treated as special high-priority tests fortified by these additional processes, SDETs tended to add more and more tests to the BVT suite, which in turn increased the run times of the suite, reduced its stability, and necessitated introducing additional “BVT nomination” and “BVT auditing” processes. We had a BVT team (!!!) whose sole purpose of existence was handling the BVT suite. The “BVT auditing” process did not work, so we invested in further segmenting the BVT suite into “micro-BVTs” and “regular BVTs”, we introduced micro-BVT execution time budgets per area, etc, etc, etc. We lived with this crazy complicated system for years.

At the beginning of 2007, we decided to put an end to the madness, and focused on optimizing our run times and improving our stability. We improved our test build times 800% and we improved test run times 500%, which enabled us to do a complete test run for about 2 hours on 20 machines. We did not really get rid of priorities, but we found that when the tests are stable and fast, people tend to not care about test priority. Today, nobody cares about priorities.

The test result reports visibly display run times, so that SDETs, SDEs and leads can address any excessively long tests. SDEs are also asked to provide feedback whenever they feel that the test run times are too long.

Run times are reported by test (and aggregated into feature area run times) to allow drilling into areas that take too long to run.

Minimum Dependencies

All test metadata is in the source control system

Tests have both build-time dependencies (the test code, etc.) and run-time dependencies (test metadata such as owners, priorities, test support files, configuration details, etc.) Some teams tend to keep the run-time dependencies on a dedicated server or in databases. In theory that sounds great – after all databases are created for storing of data.

In practice, storing of test metadata in databases is problematic, because it introduces versioning issues. Handling code branch forks, RIs, FIs, etc. becomes very difficult and error prone because you have to mirror the branch structure on the server or in the database. Some teams have designed elaborate processes to maintain referential integrity between the source control system and the support servers / databases. Although these processes may work (or can be made to work), they come with a significant support cost and are typically not robust when left on their own.

A better approach is to keep all test-related data in the source control system. There are obviously exceptions (e.g. checking in large video files may not be a good idea), and we have to be smart to not bring the source control system to its knees, but in general this approach is superior to maintaining separate databases, servers, etc.

Test code, dev code and specs are checked in the same source depot

It’s a good idea to keep dev and test code in the same branch. That helps catching and preventing build breaks as a result of breaking changes. It also enables code reuse across the organization (SDEs can reuse test framework pieces for unit tests, SDETs can reuse dev code for tests).

A lot of teams tend to keep specs on SharePoint servers. While this makes specs readily accessible and easily editable (if you can get around some of SharePoint’s idiosyncrasies), it suffers from the following problems:

Spec versioning is very tricky – tracking revision history is even trickier;
Spec discovery can be tricky – you have to know the address of the SharePoint server in addition to the source code location;
Specs tend to get lost – as the team moves from SharePoint site to SharePoint site;
Migration of SharePoint content (e.g. to a SE team) is trickier;
Specs tend to become stale – folks typically don’t bother updating specs unless they are right next to the code;
SharePoint servers may have additional maintenance costs (upgrades to new versions of SharePoint, security patching, etc.);
Specs are not easily accessible offline (although Outlook does provide offline access facilities now).

As a best practice, specs should be checked in right next to the source code (or in a obvious location that makes sense, based on the source code directory organization).

Enable a “build-xcopy-run” testing workflow through zero server dependencies

In principle, it’s a good idea to minimize the server dependencies you have. Having server dependencies increases the maintenance costs, complicates the move to Sustained Engineering (because SE has to replicate all servers and record server management knowledge), prevents SDETs and SDEs to run tests when not connected to the network and in general complicates the lab setup.

So we should subject every server dependency to careful examination and if we have to have servers, move them to the cloud.

Case Study: WPF server and database dependencies

At the beginning of 2007, the WPF test team had about 30 servers and a database-based test case management (TCM) system (its name was Tactics). This resulted in significant maintenance costs that could no longer be afforded. So the team decided to switch to a source-control-based TCM.

The results were outstanding – the team reduced server dependencies to about 6 servers (1 file server, 1 web server, 1 SharePoint server and 3 backup servers) and enabled SDETs and SDEs to build and run tests without being connected to the network. It also practically removed the need for any referential integrity related maintenance.

The WPF team also checked in all specs in source depot. Specs are bin-placed as part of the build process. This ensured that the team can produce a build with matching specs for every release of the product and we can track and compare different versions of the specs.

Fundamentals, Fundamentals, Fundamentals

For every piece of software, there are a number of cross-cutting concerns which we call “fundamentals” or “basics”. These are:

Accessibility
Compatibility (forward, backward, AppCompat, etc.)
Compliance
Deployment, serviceability and servicing
Globalization and localization
Performance and scalability
Security
Stability
Etc.

All of these are extremely important and have historically represented a significant competitive advantage for Microsoft. Testing of fundamentals is expensive so we need to have the right supporting processes and automated systems in place to ensure that we produce solid software by default.

Some of these fundamentals can be integrated within the functional testing – others typically need dedicated systems and processes. Below, I am only presenting two of the fundamentals above.

Performance and Scalability

Performance and Scalability are two fundamentals that can and should be automated. Ideally, the team has an automated system that runs Perf / Scalability tests (both micro-benchmarks and end-to-end scenarios are equality important) on target HW and SW configurations on every daily build. The system runs a set of tests, which capture agreed-upon performance goals and metrics. The system provides facilities for easy visualization of key trends.

Having an automated Perf system enables early discovery and fixing of Perf regressions. Due to the domain-specific nature of the work, it’s typically necessary to have a dedicated Performance team that develops and supports the necessary processes, tools and systems.

Case Study: WPF performance

The WPF team has a dedicated highly automated Performance lab. The Performance infrastructure automatically picks up and installs daily builds on a stable set of machines and configurations, runs a set of performance tests, presents results (cold startup times, warm startup times, working set size and various other metrics) and trends, identifies regressions, and captures necessary traces for follow up investigation.

The lab infrastructure also allows testing and generating diffs for individual dev changes (this feature is not really broadly used although the lab does do dedicated perf runs of specific potentially disruptive changes.

Stress and Stability

Stress and Stability are another fundamental that requires a dedicated process and system. It can either be done on a dedicated pool of machines / devices or on the devices in engineer’s offices during off-work hours. Some teams tend to invest a lot in automated triage systems, but in my experience these tend to be expensive to maintain so should be avoided at first.

The Stress system can also be used for security testing by running fuzzers, fault injection, other penetration tools alongside with the

Case Study: WPF stress

The WPF team has a simple dedicated stress framework, which consumes both stress-specific test code and generalized testing blocks. The tests are “deterministically random” i.e. a stress test failure can in theory (and often in practice) be reproved on demand. The tests are distributed to about 100 machines every night (these machines are in the lab). Results get analyzed and presented by vendors in China.

The team used to have a fairly sophisticated stress system that was able to do preliminary triage of failure stacks, map them to existing bugs, etc. This turned out to be an over-automated system which had a significant support cost, so we switched to the current significantly simpler system where stress failures are triaged by a vendor team in China, who manage the pool of the stress machines remotely. The system is managed by 2 engineers in China with some support from local Redmond SDETs.

Test Development Principles

Focus on design and reuse

Tests are long-lived software. Tests actually tend to live longer than the product code they test, due to AppCompat and other reasons – in Windows we have tests that are 20 years old, dating back to the Win16 days. In order to build a stable test suite teams need to invest in design and code and asset reuse.

Proper software design is an acquired taste. It is one of those things that doesn’t just happen on its own. It requires focus on architecture, design reviews, code reviews. It also requires training – both on the job (e.g. through mentoring, DR, CR) and structured training (e.g. design pattern “brownbag” sessions, etc.)

Componentization and aggregation (“Lego blocks” mentality)

A test suite (as any other piece of software) needs to be able to evolve. A test suite also needs to be portable. The single best way to create a robust, evolvable, maintainable test suite is to construct it as a combination of small self-contained building blocks.

We employ the principles of componentization and aggregation as opposed to inheritance and integration when constructing test suites. In that sense, tests are like “lego models” constructed as an aggregation of reusable “lego blocks” weaved together by test execution and configuration policy.

Automated code verification as much as possible

There are two major techniques for automated code verification:

Static code analysis
Dynamic code analysis

Static code analysis should be performed prior to check-in to prevent the team to check-in buggy code. Ideally tools such as PreFast, PreFix, FXCOP, StyleCop, Polycheck should be integrated in the build environment and any violation of their rules should result in a build break.

Dynamic code analysis is another powerful technique done through in-situ verification (asserts, special OS instrumentation such as that enabled with AppVerifier and driver verifier, etc).

Focus on End-To-End Scenarios

Product features are created in order to enable end-to-end user scenarios. The quality of a product feature does not matter if the e2e scenario doesn’t work. So test teams should work closely with the PM and dev teams to understand the e2e scenarios and to build tests that capture these e2e scenarios early in the development cycle.

Continuous Training

The single fastest way to improve a team’s throughput is through training. Of course all of us experience continuous on-the-job training (which is one of the great things about the dynamic industry we are in), but I am a big believer in providing a bit of a training structure for the organization.

The fundamental training activities we engage in are:

Team-wide brown-bag sessions:
- Sessions on fundamentals such as design patterns, data-driven testing, stress testing, perf and scalability testing, etc;
- Sessions on product-specific architecture and tools;
- Blitz sessions on product features.
1:1 mentoring:
- Done by leads or designated mentors;
- Typically done as part of test spec reviews, design reviews, code reviews.
Official Microsoft training

Parting Thoughts

Hopefully the two words that jump in your mind after reading this post are “simplicity” and “rigor”. As a general rule, I’d always trade advanced functionality for simplicity and we keep doing that all the time in the team. It takes a lot of spine though to truly commit to simplicity – as engineers our natural tendency is to solve the most complicated (and interesting) problem.

Principles 1: The Essence of Driving – A Crash Course in Project Management

ivom1 — Sun, 18 Jul 2010 21:21:00 GMT

Series Index

+++

About a year ago, I got feedback from my team that I needed to clarify what I meant by “drive this effort” or “lead that effort”. So I decided to create a quick document explaining what I meant. Below is that document. I later converted the document to a slide-deck, which is also available at the end of the post.

There are *a lot* of books on project management. From that point of view, there is nothing special about the techniques below – all of them are pretty much common sense and all of them can be found in those books. What is special about the post that follows is the condensed presentation (my goal is to essentially save you from reading a PMP book) and the fact that we actually used every single one of these techniques to manage the WPF 4 and Visual Studio 2010 products, which we released in April 2010.

The techniques listed below are:

Over-communication
Scorecard
Trend (and glide-path)
Backlog / Burn-down List
“Branded” status emails
Schedule in Excel
Schedule in Visio

Enjoy!

+++

Project management is a skill that can (as any other skill) be acquired and improved. Really most of our work and a significant part of our personal lives boil down in one way or another to project management.

Every project has a life-cycle, consisting of several standard phases:

Initiating
Planning and kick-off
Executing
Monitoring & Controlling
Closing
Post-mortem learning

An effective PM understands and actively manages the life-cycle of a project.

When I ask somebody to “drive” this or that, what I really mean is “be an effective PM of this effort”, exhibiting the following:

Independence and accountability
Ability to construct, communicate and get approval for a clear and well-though-out plan for the project, including:
- Timelines
- Scope, goals / non-goals and success criteria
- Internal and external “unmovables” and requirements
- Stakeholders
- Costs and funding
- Risks and mitigations
Ability to set the project in motion, keep the project in motion and close down the project
- Proactive contribution of directive energy to the project, creating excitement, and identifying and removing road-blocks.
- Active monitoring of the progress of the project
- Proactive communication of status
Ability to reach the desired results

These are the standard milestones of a project. Simpler projects can certainly be managed without necessarily differentiating between some of the milestones.

It is important to note that there are project management techniques out there that try to simplify the model above. Such simplifications are often based on “hardcoded rules” e.g. (“each project cycle takes exactly 4 weeks”, etc.) It’s also important to note that not all projects require all of the “line items” above delivered explicitly. Most projects do require them at least in implicit form though.

Depending on the specific project, some of the activities may or may not be required. Below are two example projects that demonstrate how to use the template above.

Keeping good communication throughout the project is a common-sense technique. It is, however, surprisingly rare. So, as a good rule of thumb, always over-communicate your project status. Don’t worry -- you will inevitably hear from your manager and other major stakeholders if you truly are spamming everybody, but I guarantee that would never happen.

Scorecards provide a way to track progress against a set of success criteria. Scorecards are often presented in time, although that may not be always possible or meaningful. Here is a sample scorecard:

You can track both “qualitative assessments” (e.g. “gut-feel of main stakeholders”) as well as “quantitative observables” (e.g. “number of defects”). When rating the progress, it’s useful to use a 4-level scheme similar to the one below. Excel’s conditional formatting feature makes construction of scorecards fairly easy.

Glide-paths are a common way of graphically displaying change in time of one or more observables in an attempt to inform future progress. Here are a few examples:

Note a few important details:

Having a glide-path or some other indication of important milestones (e.g. B1, B2, RTM) always helps since it establishes a context for the trend and acts as a gentle “forcing function”;
It’s a good idea to provide a title for the trend chart and names (and dimensions) for the chart’s axes except when they are self-evident (as is the case with time above).

Backlogs were popularized by Scrum and other agile project management methodologies. A backlog has two purposes:

Keep all tasks / items associated with a particular project in a central, regularly updated location that is easy to point to, prioritize (hopefully just once in release);
Have a “parking lot” for future items, feature requests.

Here is a sample backlog:

The average engineer at Microsoft receives between 200 and 300 emails every day. One way to allow stakeholders and observers to easily monitor the progress of a project you are managing is to “brand” the emails related to that project. Examples of “branding” include:

Standardized subject line (e.g. “WPF 4 Pulse: Stress”) – allows people to easily filter and search for emails pertaining to that project;
Standardized “look and feel” of the email – either by maintaining a consistent structure or by including a common logo in the email;
Setting up a custom alias for this project only (e.g. “WPF – VS Tactics”, etc.)

Excel presents an easy way to construct schedules. These typically have a “From”, “To” and “Length” columns. Here is an example:

More complicated schedules typically require the introduction of a “Type” (of event) column, to allow filtering per type, thus making your schedules easier to grasp. Using “tables” in Excel makes the constructed schedules easily “filterable”.

What Excel schedules lack is a graphical representation of scaled time periods. This is where Visio schedules excel. Here is a sample.

It’s often very helpful to complement this with a “we are here” arrow:

The arrow accomplished several things:

Lightning fast way to communicate to your team and stakeholders where you are in relation to the whole project;
Increases the confidence that the schedule is real and recent (if you bothered placing an arrow, you probably have ensured that you are placing it on an up-to-date schedule).

Good project managers are hard to find. Partly, because good project managers grow through experience. Invariably though, through experience, they (or at least the ones I have observed) tend to gravitate to the same basic techniques – the ones you see above. Hopefully, you will find these helpful.

In case you haven’t noticed, I love Excel. I use it as my main project management tool. My wife is a construction project manager and she constantly bugs me about not using Microsoft Project. Project is a fine product and I have in fact used it for some of the more complicated projects I have managed. What I love about Excel is its simplicity, integration (charts) and high-availability (Excel files are viewable on any device).

Some manage their projects using databases and web sites. I still prefer Excel. The reason? Versioning. With Excel (or Project), all you need to do is check in the corresponding file in your SCM (source code management) system. Databases are harder to version.

Finally, here’s a pointer to the slide-deck accompanying this document as well as a sample Feature Backlog spreadsheet:

The Essence of DRIVING

View more presentations from ivom1234.

Inserting of file(Feature Backlog.xlsx) failed. Please try again.

PM in this context refers to any “project manager” (a lead or an individual contributor in any disciplines tasked with “driving” something), not just a Microsoft PM.

VMI (Vision-Mission-Identity) is something that is often neglected, mostly because project teams tend to eventually infer these themselves. VMI is, however, a major part of the branding of the project and the project team and tends to have a very positive impact on driving ownership into and empowering of the team members. In practical terms, VMI is answering the questions “what is the ideal outcome of the project (e.g. “deliver the best text on any platform”), who are we (e.g. “we are the team, enabling clear text in VS 2010”), and why are we the right team to do it.

Building Outdoor Pull-Up Bar and Parallel Bars

ivom1 — Thu, 15 Jul 2010 01:50:00 GMT

Pull-ups and dips are among the best exercises for the upper body. So, I have always wanted to build the necessary exercise equipment in my backyard. Our move to San Jose and this excellent article on HowToMatic provided the critical mass to get me started. This post captures all you need to know to build your own.

Necessary Tools and Materials

I got everything I needed at Home Depot, except for the bungee ropes, which I already had.

Fig. 1 Tools and materials

Building the Equipment

1. Have a plan. Here’s what mine looked like:

Fig. 2 Equipment diagram

Fig. 3 Final result

2. Dig the 6 holes. It took me about 4 hours to dig all of them without killing myself and I made the mistake of digging them a bit too narrow. Don’t be afraid to dig wide. The holes need to be at least 1.5 feet (50 cm) deep and need to be about 1.5 feel (50 cm) in diameter.

3. Drill holes for the pipes through the wooden beams. Drill about 4 inches (10 cm) away from the end of the beam. Don’t forget to wear glasses / goggles for eye protection. You can either drill al the way through the entire beam or drill just on one side of the beam. In retrospect, I should have done the latter for all holes.

4. Position the beams in the holes. Use small stones and the bungee ropes to ensure that the beams are properly positioned and straight. I decided to make one side of the dip bars slightly wider than the other side – for exercise versatility – so my dip bars are not exactly parallel. Position the pipes into the beam holes.

5. Prep the concrete, following the directions on the bag. I prepped the concrete one bag at a time. Mixing concrete is tough so I recommend the following workflow:

Pour all water you need for a full bag of concrete in the bucket
Pour half a bag of concrete in the bucket. Mix, until you get a homogenous mixture.
Pour the other half – mix again.
Work fast – you have less than 5 minutes to mix the whole bag and probably another 5 minutes to pour the concrete into the holes.
Use some help – having a second set of hands helps a lot.

6. Pour the concrete into the holes, compressing it.

7. Clean up. Let it cure for a day before using the equipment. Then drill the small holes for the screws to prevent the pipes from rotating.

Frankly, I loved the physical labor, and you will too… It is a surprisingly therapeutic and rewarding experience.

Exercises

The great thing about this setup is that you can do all sorts of different exercises that develop strength and agility, using the weight of your own body, for example…

Chin-ups
Pull-ups
Muscle-ups
Pull up and around the bar
Dips
Swinging dips

I recommend starting with chin-ups and dips (3 sets each, max repetitions in each set) and progressing from there.

Enjoy!

Move to California, Principles, etc.

ivom1 — Tue, 13 Jul 2010 20:44:09 GMT

A couple of quick announcements…

First, about a month ago we moved to California. It was a bit of a bitter-sweet move for us. On one hand, we left behind a lot of good friends and colleagues – it felt like the end of an era. On another, while Seattle itself is a nice city, we never got used to the rain – we’ve been aching for sunshine for 9+ years. The release of WPF 4 and VS 2010 presented the right time to make the jump, so we made the jump we had been thinking of. I took a position in the Mediaroom team, working at Microsoft’s Silicon Valley Campus in Mountain View. The past 3.5 years in WPF were fantastic – I worked with great people, worked for a great manager (Ian Ellison-Taylor) and shipped 3 great versions of WPF, culminating with WPF 4, which now powers the $1B Visual Studio business.

Second, a change is always a good opportunity for a retrospective brain-dump. So in the coming weeks I will be sharing some pearls of wisdom (“the principles”) that guided our execution in the WPF team. Hopefully, you’ll find them useful…

As for Cali, give me a shout if you are in the vicinity and want to hook up!

XAML Compliance Suite v1 Released!

ivom1 — Thu, 08 Jul 2010 14:57:00 GMT

Lester and his team (the XAML test team) just released v1 of the XAML compliance suite – a set of basic tests that can be used to assess the compliance of any customer XAML parser to the XAML spec.

To download the suite, check our Lester’s blog post.

A Lap Around TestApi v0.5

ivom1 — Wed, 23 Jun 2010 06:50:00 GMT

A few days ago we released the new version of TestApi – the API library for testing. My previous post from 6/18 summarized the additions and modifications to the library. This post goes a little bit deeper.

Documentation

In the TestApi package, we release both MSDN-style documentation (as CHM file and as a set of HTML files) as well as “conceptual documentation”. In TestApi 0.5, we spent a lot of effort cleaning up, updating and extending all of our documentation in an attempt to improve the usability of the library.

Theme Configuration API

TestApi v0.5 has a simple API for OS theme management. For example, you can easily list all available themes on the system and dump various attributes:

using Microsoft.Test.Theming;
...

Console.WriteLine("Current Theme:");
Console.WriteLine("\tName:    {0}\n", Theme.GetCurrent().Name);

Console.WriteLine("Available Themes:");
foreach (Theme t in Theme.GetAvailableSystemThemes())
{
    Console.WriteLine("\tName:    {0}", t.Name);
    Console.WriteLine("\tPath:    {0}", t.Path);
    Console.WriteLine("\tEnabled: {0}", t.IsEnabled);
    Console.WriteLine("\tStyle:   {0}\n", t.Style);
}

You can also switch to a desired theme programmatically, which is valuable when you need to verify appearance across themes:

Theme ot = Theme.GetCurrent();

try
{
    Theme[] availableThemes = Theme.GetAvailableSystemThemes();
    foreach (Theme t in availableThemes)
    {
        Console.WriteLine("Switching to theme \"{0}\"...", t.Name);
        Theme.SetCurrent(t);
    }
}
finally
{
    Theme.SetCurrent(ot);
}

Revamped Input Injection API

In previous releases of the library, our input injection API had the peculiarity of being dependent on WPF types (specifically the Key and MouseButton enums). Apart from introducing an unnecessary dependency for tests that target WinForms, this had the annoying side effect of making the use of the API too verbose, i.e. in TestApi 0.4 you had to explicitly qualify Mouse and Keyboard, because you had to have dependencies on WPF namespaces which happen to also have similarly named classed in them:

using System;
using System.Windows.Input;
using Microsoft.Test.Input;
...

Microsoft.Test.Input.Mouse.MoveTo(new System.Drawing.Point(...));
Microsoft.Test.Input.Mouse.Click(MouseButton.Left);
...
Microsoft.Test.Input.Keyboard.Type("Hello, world!");
...

TestApi 0.5 fixes all of that. Now you can simply do:

using System;
using System.Drawing;
using Microsoft.Test.Input;
...

Mouse.MoveTo(new Point(...));
Mouse.Click(MouseButton.Left);
...
Keyboard.Type("Hello, world!");
...

Visual Verification API Additions

By popular demand, we have added support for searching for sub-images to Snapshot, so now you can do things like:

using Microsoft.Test.VisualVerification;

Snapshot s1 = Snapshot.FromFile("BigImage.png");
Snapshot s2 = Snapshot.FromFile("SubImage.png");

foreach (Rectangle r in s1.Find(s2))
{
    Console.WriteLine(
        "Found s2... Coords: ({0}, {1}); Size: ({2}, {3})",
        r.Left, r.Top, r.Width, r.Height);
}

Snapshot.Find also has an overload which takes as a second parameter a SnapshotVerifer, so that you can do “fuzzy” searches.

Text String Generation API Extensions

TestApi 0.4 introduced the new text string generation API. However, what we had in TestApi 0.4 had an important limitation – you could only generate strings in one UnicodeRange at a time, i.e. you used to write the following code:

using Microsoft.Test.Text;
...

StringProperties sp = new StringProperties();

sp.MinNumberOfCodePoints = 10;
sp.MaxNumberOfCodePoints = 20;
sp.UnicodeRange = new UnicodeRange(UnicodeChart.BraillePatterns);

string s = StringFactory.GenerateRandomString(sp, 1234);

TestApi 0.5 lifts that restriction. Now you can generate strings that contain code points from multiple ranges:

using Microsoft.Test.Text;
...

StringProperties sp = new StringProperties();

sp.MinNumberOfCodePoints = 10;
sp.MaxNumberOfCodePoints = 20;
sp.UnicodeRanges.Add(new UnicodeRange(UnicodeChart.BraillePatterns));
sp.UnicodeRanges.Add(new UnicodeRange(UnicodeChart.Cyrillic));

string s = StringFactory.GenerateRandomString(sp, 1234);

Revamped Combinatorial Variation Generation API

The Combinatorial Variation Generation API underwent a major revamp in TestApi 0.5. We introduced support for generics, support for declarative models and moved to a LINQ-based syntax for constraints.

The example below demonstrates the basic usage:

using System;
using System.Collections.Generic;
using Microsoft.Test.VariationGeneration;

...

// Specify the parameters and parameter values
var os = new Parameter<string>("OS") { "WinXP", "Win2k3", "Vista", "Win7" };
var memory = new Parameter<int>("Memory") { 512, 1024, 2048, 4096 };
var graphics = new Parameter<string>("Graphics") { "Nvidia", "ATI", "Intel" };
var lang = new Parameter<string>("Lang") { "enu", "jpn", "deu", "chs", "ptb" };

var parameters = new List<ParameterBase> { os, memory, graphics, lang };

var model = new Model(parameters);

// The model is complete; now generate the configurations and print out
foreach (var config in model.GenerateVariations(2))
{
    Console.WriteLine("{0} {1} {2} {3}",
        config["OS"],
        config["Memory"],
        config["Graphics"],
        config["Lang"]);
}

Often, one needs to apply constraints when generating variations. TestApi 0.5 constraints are now LINQ-based as shown below:

var de = new Parameter<string>("Destination") { "Whistler", "Hawaii", "Las Vegas" };
var ho = new Parameter<int>("Hotel Quality") { 5, 4, 3, 2, 1 };
var ac = new Parameter<string>("Activity") { "gambling", "swimming", "shopping", "skiing" };

var parameters = new List<ParameterBase> { de, ho, ac };
var constraints = new List<Constraint<Variation>>
{
    // If going to Whistler or Hawaii, then no gambling
    Constraint<Variation>
        .If(v => de.GetValue(v) == "Whistler" || de.GetValue(v) == "Hawaii")
        .Then(v => ac.GetValue(v) != "gambling"),

    // If going to Las Vegas or Hawaii, then no skiing
    Constraint<Variation>
        .If(v => de.GetValue(v) == "Las Vegas" || de.GetValue(v) == "Hawaii")
        .Then(v => ac.GetValue(v) != "skiing"),

    // If going to Whistler, then no swimming
    Constraint<Variation>
        .If(v => de.GetValue(v) == "Whistler")
        .Then(v => ac.GetValue(v) != "swimming"),
};
var model = new Model(parameters, constraints);


foreach (var vacationOption in model.GenerateVariations(2))
{
    Console.WriteLine("{0}, {1} stars -- {2}",
        vacationOption["Destination"],
        vacationOption["Hotel Quality"],
        vacationOption["Activity"]);
}

Another major addition in TestApi 0.5 is the ability to use declarative models, which simplifies the model definition syntax significantly. Users of the API apply attributes to a regular class to declare parameters and possible parameter values:

class OsConfiguration
{
    [Parameter(512, 1024, 2048)]
    public int Memory { get; set; }

    [Parameter("WinXP", "Vista", "Win2K8", "Win7")]
    public string OS { get; set; }

    [Parameter("enu", "jpn", "deu", "chs", "ptb")]
    public string Language { get; set; }

    [Parameter("NVidia", "ATI", "Intel")]
    public string Graphics { get; set; }
}

Then the model is formed and used as follows:

var model = new Model<OsConfiguration>();
foreach (OsConfiguration c in model.GenerateVariations(2))
{
    Console.WriteLine("{0} {1} {2} {3}", 
        c.Memory, 
        c.OS,
        c.Language,
        c.Graphics);
}

The declarative syntax supports also equivalence classes, as well as parameter value weights. In the example below for example”Vista” and “Win7” are treated as values in same equivalence class. “Vista” or “Win7” will collectively appear more often than “WinXP” because of their larger weight value:

class OsConfiguration
{
    [Parameter("WinXP", Weight = .3F)]
    [Parameter("Vista", "Win7", Weight = .5F)]
    public string OS { get; set; }

    [Parameter("EN-US")]
    [Parameter("JP-JP", "CN-CH")]
    [Parameter("HE-IL", "AR-SA")]
    public string Language { get; set; }
}

Fault Injection API Enabled for .NET 4 Binaries

The fault injection facilities in TestApi 0.4 did not work for .NET 4 binaries. TestApi 0.5 fixes that problem. TestApi 0.5 also ships all binaries necessary for fault injection as part of the package, to reduce dependency on the .NET installation on the target machine.

We have also made a minor naming change, renaming BuiltInConditions.TriggerEveryOnNthCall to BuiltInConditions.TriggerOnEveryNthCall.

Fixes and Additions to the Command-Line Parsing API

We have made significant modifications and fixes to the documentation of the command-line parsing API, expanding and fixing the MSDN-style documentation and the conceptual documentation to achieve consistency and correctness and to streamline the use of the API.

xUnit 1.5

As part of TestApi 0.5 we have switched to using xUnit 1.5. xUnit continues to be a fantastic unit-testing framework – both simpler and more full features than other comparable frameworks. xUnit 1.5 in particular provides a significantly improved GUI tool, which makes execution of the TestApi acceptance tests significantly easier.

+++

Most of the updates in TestApi 0.5 are under the cover, but in their entirety represent a significant update compared to TestApi 0.4. Let us know what you’d like to see in the future revisions of the library.

Attached is a solution with the sample code above.

TestApi v0.5 Released!

ivom1 — Fri, 18 Jun 2010 07:13:00 GMT

I am happy to announce that we have just released version 0.5 of TestApi – the testing API library – on Codeplex. Version 0.5 contains a number of important additions and modifications:

We made significant updates to our documentation, revising practically all conceptual and MSDN-style API documents, to make the library easier to use. I have also been updating my blog posts on the library because some of the content there has gotten somewhat stale. Big thanks to Christine Warren – our technical documentation editor.
Vincent Sibal created a new Theme Configuration API allowing you to easily switch Windows OS themes.
Vince also revamped the Input Injection API, removing the dependency on WPF, and adding new acceptance tests.
Tim Cowley added sub-image search support to the Visual Verification API, based on a request by NLord.
Dennis Deng extended the Text String Generation API with the ability to generate strings from multiple Unicode ranges. He also added additional acceptance tests.
Nathan Anderson and Eugene Zakhareyev revamped the Combinatorial Variation Generation API, converting constraints to a LINQ-based syntax, adding support for generics, adding support for equivalence classes, adding weights and tags and enabling declarative models.
Bill Liu enabled the Fault Injection API for .NET 4 binaries.
Dan Marley made various fixes and test additions to the Command-line Parsing API.
We also switched to using xUnit 1.5, which is included in the TestApi 0.5 package.

Last but not least, a big thank you to Alexis Roosa (the PM of the effort) and Peter Antal (who together with Dan Marley did a bunch of design reviews).

Let us know what you think! And vote for features!

TestApi Voting Site

ivom1 — Tue, 15 Jun 2010 03:18:00 GMT

I wanted to point all TestApi users to our new feature-suggestion-slash-voting site for TestApi on http://testapi.uservoice.com. Earlier today we seeded the site with a few ideas for features.

Help us prioritize future TestApi features by submitting your feature requests and voting on existing submissions.

Introduction to TestApi – Part 7: Memory Leak Detection API

ivom1 — Thu, 15 Apr 2010 17:16:00 GMT

Series Index

+++

General Notes

A memory leak is the inability to release reserved memory. Memory leaks often lead to continuously increasing memory usage in applications, which in turn degrades the application and system performance.

Most modern operating systems free all the memory that is reserved by a process when the process terminates, which provides some protection against cumulative memory leaks that result from repeated execution of the same “leaky” application. However, this solution is at best only partially successful because all kinds of memory leaks are undesirable, and you need to catch and eliminate them.

TestApi provides a simple memory-leak-detection mechanism, which takes memory snapshots of an executing process and then processes the snapshots with arbitrarily complex leak-detection algorithms, as the following figure shows. The leak detection APIs were designed and implemented by our engineer Shozub Qureshi in collaboration with other members of our team.

Fig.1 Leak detection workflow

TestApi currently only provides mechanisms for crude detection of leaks. If a leak is suspected, the TestApi user should use a profiler to narrow down possible memory leaks, and then confirm or disprove them.

Terminology

Different tools often use different names for the same memory metrics. The following table maps the naming across three popular tools – the OS-provided Performance Counters, Process Explorer and Task Manager (on Windows 7).

Except when necessary to resolve inconsistencies, TestApi typically follows the naming convention that is used by the Performance Counters. For an in-depth description of all metrics, refer to the TestApi reference documentation.

Usage

Now let’s demonstrate the usage of the API through a few examples...

Example #1

TestApi provides the MemorySnapshot class, which helps you capture, serialize, deserialize, and make initial comparisons of the memory state of a process. The following example demonstrates how to use the MemorySnapshot class to display memory usage differences for a given process.

MemorySnapshot gives you a very easy way to capture crude memory information for your process. In this first version of the API we don’t provide facilities to track individual memory allocations – we will provide these in a future revision of the API.

//
// Taking two memory snapshots of Notepad and comparing them for leaks
//

// Start an instance of Notepad.exe, get its PID, and take a memory snapshot
Process p = Process.Start("notepad.exe");
p.WaitForInputIdle(5000);
int pid = p.Id;
MemorySnapshot s1 = MemorySnapshot.FromProcess(pid);

//
// Perform operations that may cause a leak in Notepad...
//

// Capture a second snapshot
MemorySnapshot s2 = MemorySnapshot.FromProcess(pid);

// Compare the two memory snapshots and generate a diff.
// Then display the diff to the console.
MemorySnapshot diff = s2.CompareTo(s1);

Console.WriteLine("Memory diff for process {0}:", pid);
Console.WriteLine("Start time: {0}", s1.TimeStamp);
Console.WriteLine("End time: {0}", s2.TimeStamp);
Console.WriteLine("\tGDI Object Count: {0}", diff.GdiObjectCount);
Console.WriteLine("\tHandle Count: {0}", diff.HandleCount);
Console.WriteLine("\tPageFile Bytes: {0}", diff.PageFileBytes);
Console.WriteLine("\tPageFile Peak Bytes: {0}", diff.PageFilePeakBytes);
Console.WriteLine("\tPool Nonpaged Bytes: {0}", diff.PoolNonpagedBytes);
Console.WriteLine("\tPool Paged Bytes: {0}", diff.PoolPagedBytes);
Console.WriteLine("\tThread Count: {0}", diff.ThreadCount);
Console.WriteLine("\tUser Object Count: {0}", diff.UserObjectCount);
Console.WriteLine("\tVM Bytes: {0}", diff.VirtualMemoryBytes);
Console.WriteLine("\tVM Private Bytes: {0}", diff.VirtualMemoryPrivateBytes);
Console.WriteLine("\tWS Bytes: {0}", diff.WorkingSetBytes);
Console.WriteLine("\tWS Peak Bytes: {0}", diff.WorkingSetPeakBytes);
Console.WriteLine("\tWS Private Bytes: {0}", diff.WorkingSetPrivateBytes);

// Close the process.
p.CloseMainWindow();
p.Close();

Example #2

In addition to MemorySnapshot, TestApi provides the MemorySnapshotCollection class, which allows serialization / deserialization of a series of memory snapshots. The use of this class is demonstrated below:

//
// Taking multiple memory snapshots of Notepad and storing them for
// later analysis
//

// Create a memory snapshot collection and add the snapshots to it.
MemorySnapshotCollection c = new MemorySnapshotCollection();

// Start an instance of the target process and wait for it to reach steady state.
// Then take a memory snapshot.

Process p = Process.Start("notepad.exe");
p.WaitForInputIdle(5000);
MemorySnapshot s1 = MemorySnapshot.FromProcess(p.Id);
c.Add(s1);

// Perform operations that may cause a leak in the target process...
// Then take a second memory snapshot
MemorySnapshot s2 = MemorySnapshot.FromProcess(p.Id);
c.Add(s2);

// Save the collection to a XML file for later analysis.
c.ToFile(@"C:\mySnapshots.xml");

// Close the process.
p.CloseMainWindow();
p.Close();

Conclusion

Although admittedly crude in its first version, the memory leak detection API provided by TestApi enable easy creation of memory leak detection tests and tools. The documentation provides a lot of information about how the different metrics are calculated with pointers to the relevant articles and OS APIs.

We will be expanding this API in the future to include allocation tracking as well as advanced leak detection analysis (currently, you have the perform the analysis yourself based on the collected data or fire up a debugger or profiler to analyze a leaky process), so let us know of specific needs, feature requests and ideas, etc.

Introduction to TestApi – Part 6: Text String Generation API

ivom1 — Wed, 14 Apr 2010 13:53:00 GMT

Series Index

+++

General Notes

Testing often necessitates constructing strings to deliberately exercise specific features of an application. TestApi provides a text string generation API to facilitate generating strings that are random within a set of fixed parameters, such as the number of characters, target Unicode range, whether or not the string should contain numbers, whether or not the string should contain line breaks, etc.

The API was created by our engineer Dennis Deng in collaboration with other engineers around the company.

Unicode Basics

The TestApi text string generation API targets both general testers and professional text testers and exposes functionality for constructing strings given a novice or advanced understanding of Unicode. Two common Unicode concepts which are used throughout the library are defined below. For a comprehensive introduction to Unicode and a complete glossary, visit http://unicode.org/glossary/.

Code Points: In Unicode terminology any particular character set is referred to as an Encoding or Code. A code point is one specific point within a code, i.e. one specific character within a character set.
Unicode Character Code Chart: The Unicode standard defines a number of different character sub-sets, which are called Unicode character code charts (or Unicode charts for short). These charts are available on http://unicode.org/charts. Every chart (represented by the UnicodeChart class) is uniquely identified by three properties: group name, name and sub-name. For example, the “Armenian Ligatures” chart has a group name of “European Scripts”, name of “Armenian” and sub-name of “Armenian Ligatures”. It is important to note that the Unicode charts are versioned. TestApi supports the Unicode chart which is built in the referenced version of the .NET framework.

String Generation Technology

StringFactory: A StringFactory is an object used for generating random strings. Rather than constructing purely random strings, the factory will generate strings within a certain set of parameters, determined by a provided StringProperties object.
StringProperties: A StringProperties object specifies the properties of a string to be randomly generated by a StringFactory, such as the number of unique characters or the Unicode character set (or subset) to use. A StringProperties object is passed to StringFactory.GenerateRandomString(…).
UnicodeRange: A UnicodeRange is a range of Unicode code points. A StringProperties object contains a UnicodeRange. When the UnicodeRange of a particular StringProperties object is set to a particular range of values, any strings generated using that StringProperties object will be confined to the given UnicodeRange.

Example #1

The following example demonstrates how to generate a Cyrillic string that contains between 10 and 30 characters.

//
// Generate a Cyrillic string with a length between 10 and 30 characters.
//

StringProperties properties = new StringProperties();
properties.MinNumberOfCodePoints = 10;
properties.MaxNumberOfCodePoints = 30;
properties.UnicodeRanges.Add(new UnicodeRange(UnicodeChart.Cyrillic));

string s = StringFactory.GenerateRandomString(properties, 1234);

The generated string may look as follows:


s: Ӥёӱіӱӎ҄ҤяѪӝӱѶҾүҕГ

Notice how the string contains 17 characters that are all part of the Cyrillic character code chart.

Example #2

The following example demonstrates how to generate a string of a fixed length with numbers.

// 
// Generate a string of 20 random code points containing numbers.
//

StringProperties properties = new StringProperties();
properties.MinNumberOfCodePoints = 20;
properties.MaxNumberOfCodePoints = 20;
properties.HasNumbers = true;
properties.UnicodeRanges.Add(new UnicodeRange(0, 0xFFFF));

string s1 = StringFactory.GenerateRandomString(properties, 5678);
string s2 = StringFactory.GenerateRandomString(properties, 5678);

The generated strings may look as follows:


s1: 鼯粳뷷賵Ɨ烣૝犥0崪𣏕窚氢ཝꥁ姽戁䅽7
s2: 鼯粳뷷賵Ɨ烣૝犥0崪𣏕窚氢ཝꥁ姽戁䅽7

Notice how the strings contain numbers and have exactly 20 code points (characters). Notice also how using the same seed (5678) results in generation of exactly the same strings.

Conclusion

Text string generation is a tricky. TestApi exposes a simple straightforward API for generation of text strings from a set of desired string properties.

A Few Good Posts on WPF 4

ivom1 — Tue, 13 Apr 2010 14:59:24 GMT

Yesterday, we celebrated the official release of Visual Studio 2010 and .NET 4. Check out Bob Muglia’s keynote here.

One of the major changes in VS 2010 is the fact that its UI was built using WPF. We in the WPF team had to do a fair amount of work to support the migration of the VS UI to WPF. Paul Harrington’s series on WPF in Visual Studio 2010 has all the details.

There is, however, a lot more that is new and fresh in WPF 4 in addition to VS-related improvements.

Here is a list of a few blog posts that walk through the new features:

Pete Brown, a Community PM of ours, has an excellent post called WPF 4 Release: A guide to the new features. See Scott Guthrie’s post on WPF 4 too.

Jossef Goldberg, our Deployment and Performance PM, has a couple of posts on deployment (What’s new in .NET Framework 4 Client Profile RTM) and performance (What’s New for Performance in WPF in .NET 4).

Chipalo Street, our Text PM, has several blog entries on text clarity – a big area of investment for WPF 4. I recommend starting with the Additional WPF Text Clarity Improvements post.

Lester Lobo, the XAML SDET Lead for .NET 4, has a great series of posts on new WPF 4 features.

Enjoy WPF 4 and VS 2010!

The 22 Minute Meeting

ivom1 — Tue, 09 Mar 2010 21:00:00 GMT

A good back-to-basics article by Scott Berkun on how to do efficient meetings, based on the original idea by Nicole Steinbok.

And here is the poster that says it all:

February 2010 WPF Toolkit Release

ivom1 — Wed, 24 Feb 2010 05:59:44 GMT

I am happy to announce that we have just released the February 2010 release of the WPF Toolkit! The toolkit contains updated versions of previously released components (DataGrid, DatePicker, Calendar and VSM) as well as WPF ports of 3 popular controls from the Silverlight toolkit -- the AutoCompleteBox, Accordion and Rating controls.

Check out this new release on http://wpf.codeplex.com/releases/view/40535 and of course – share your feedback.

Introduction to TestApi – Part 5: Managed Code Fault Injection APIs

ivom1 — Wed, 25 Nov 2009 08:47:00 GMT

Series Index

+++

Fault Injection is the act of artificially changing the behavior of an existing executable code to simulate various faults. FI is very useful for validation of error handling code paths and for improving code coverage.

There are several types of fault injection. In runtime fault injection, the fault injecting test modifies the execution logic of the application under test (AUT), by injecting faults, triggered by specific runtime conditions. One could for example implement a FI test with the following semantic:

Throw an out-of-memory (OOM) exception, whenever the application calls method CreateWidget of class WidgetManager.

The FI terminology is as follows:

AUT (application under test) – this is the tested application, in which faults are being injected;
Fault Rule – The fault rule is a central construct in an FI test that determines WHEN faults get triggered and WHAT TYPES of faults get triggered. A fault rule consists of:
- a Method Signature, determining the method where the fault will be injected;
- a Fault Condition, determining when the specific fault should be triggered (e.g. every N^th call)
- a Fault -- Determines the type of fault (e.g. throwing an exception, returning a specific value, etc.) that occurs when the fault condition is met.
Fault Session – The fault session is a collection of fault rules that are applied to a given AUT.

TestApi provides a simple, but powerful runtime fault injection API for injecting faults in managed code. The API was originally designed and implemented by Bill Liu et al from the “Essential Business Server” team, and adapted to TestApi by Sam Terilli from our WPF XAML team. The following content provides a quick introduction to the API.

Sample AUT

Following is the code of a trivial AUT that we will use for demonstration purposes:

//
// This is a sample application used for demonstration purposes.
//
using System;

class MyApplication
{
    static void Main(string[] args)
    {
        int a = 2;
        int b = 3;

        for (int i = 0; i < 10; i++)
        {
            Console.WriteLine("{0}) {1} + {2} = {3}", i, a, b, Sum(a, b));
        }
    }

    private static int Sum(int a, int b)
    {
        return a + b;
    }
}

The result of running this application is of course:

> MyApplication.exe 

    
0) 2 + 3 = 5 

    
1) 2 + 3 = 5 

    
2) 2 + 3 = 5 

    
3) 2 + 3 = 5 

    
4) 2 + 3 = 5 

    
5) 2 + 3 = 5 

    
6) 2 + 3 = 5 

    
7) 2 + 3 = 5 

    
8) 2 + 3 = 5 

    
9) 2 + 3 = 5

A Simple Fault Injection Test

Now, let’s try to inject a fault in the AUT. Let’s assume that we want to modify the return value of Sum. Here’s how we can accomplish that:

// 
// Simple fault injection test
//
using System;
using System.Diagnostics;
using Microsoft.Test.FaultInjection;

public class FaultInjectionTest 
{ 
    public static void Main() 
    { 
        // 
        // Set up a fault rule to return –1000 the second time Sum is called.
        //
        string method = "MyApplication.Sum(int,int)";
        ICondition condition = BuiltInConditions.TriggerOnNthCall(2);
        IFault fault = BuiltInFaults.ReturnValueFault(-1000);
        FaultRule rule = new FaultRule(method, condition, fault);            

        //
        // Establish a session, injecting the faults defined by the fault rule(s)
        //
        FaultSession session = new FaultSession(rule);
        ProcessStartInfo psi = session.GetProcessStartInfo(@".\MyApplication.exe"); 

        //
        // Launch the target process and observe faults
        //
        Process p = Process.Start(psi);        
        p.WaitForExit();
    }
}

Fairly straightforward. Upon running the test (which will itself spawn the AUT) we observe the following output:

> FaultInjectionTest.exe 0) 2 + 3 = 5 1) 2 + 3 = -1000 2) 2 + 3 = 5 3) 2 + 3 = 5 4) 2 + 3 = 5 5) 2 + 3 = 5 6) 2 + 3 = 5 7) 2 + 3 = 5 8) 2 + 3 = 5 9) 2 + 3 = 5

As intended, we injected a runtime fault in MyApplication.exe, which resulted in Sum returning –1000 the second time it got called.

Under The Covers

Under the covers, the managed code Fault Injection API uses the CLR profiling API to modify the prologue of the intercepted method at runtime in order to inject the desired fault. The injected prologue instructions essentially call a method in the library, which then dispatches the call to the specified fault.

Because faults are injected at runtime, the code of the original application is not modified in any way. There is a certain performance degradation, which depends on the number of the injected faults.

The fault injection API provides a variety of built-in conditions and faults (in the BuiltInConditions and BuiltInFaults classes respectively). Users of the API can also create custom conditions and faults (by implementing the ICondition and IFault interfaces respectively). The API also provides a set of classes that expose the ability to fine tune and monitor the injected faults.

In addition, the API provides a facility to set “global faults”, which is useful for server application testing, where one application typically consists of and recycles many different processes.

I have attached the sample above, which should get you up and running with fault injection.

Application Accessibility Testing

ivom1 — Wed, 25 Nov 2009 00:31:00 GMT

A lot of our customers and partners have asked us to provide guidance on how to make their WPF and Silverlight applications accessible, so I decided to publish a post folks can refer to. Note most of the content below is directly applicable to any other Windows application.

Introduction

In general “Accessibility” means two things:

Nominal accessibility: An application or an application programming stack exposes the necessary information so that it makes its UI elements accessible;
Accessibility with an accessibility tool: An accessibility tool (e.g. a screen reader) can access the UI of a given application and expose the UI to its users.

For an application to be “accessible”, ideally it would satisfy both (1) and (2) i.e. be both nominally accessible and also accessible with popular accessibility tools. Satisfying (1) guarantees that the level of accessibility of an application does not depend significantly on the accessibility tool. Satisfying (2) guarantees that a disabled user can interact with an application, when using a particular accessibility tool. Satisfying (2) is clearly more important, and as a matter of fact the various accessibility tools go to extraordinary lengths to make applications that are not nominally accessible be accessible through the tool.

+++

Accessibility tools use several techniques to build an accurate in-memory representation of what is presented on the screen:

Public accessibility API
Public platform API
Kernel interfaces
In-process code injection

Microsoft provides two accessibility API – MSAA and UIA. MSAA is a legacy API, introduced circa 1996, which is not extensible and is no longer sufficient for modern UIs. UIA is the new much richer and fully extensible Accessibility API from Microsoft, originally introduced in WPF / Windows Vista in 2007.

It’s important to note that many existing Win32 applications are not nominally accessible. I.e. they do not expose the right accessibility information, using public accessibility API. They are only made accessible through the use (by the accessibility tools) of techniques such as (ii), (iii) and (iv) above.

+++

Accessibility API typically expose:

Provider API : these are the API used by an application stack to expose its UI elements;
Client API: these are the API used by an accessibility tool (e.g. a screen reader) to query the UI elements of an application.

MSAA and UIA both conform to that client-provider model. There is also a bridge between MSAA and UIA, which allows a client using the MSAA client API to access an application exposed through UIA.

General Accessibility Testing Workflow

Microsoft provides a number of accessibility testing tools that can help with identifying of nominal accessibility problems. Combining these with the target accessibility tool is the winning recipe for creating of fully accessible applications.

Below is a suggested accessibility testing work-flow:

Step	If “Pass”	if “Fail”
1. Run “AccChecker” (download from here)	Go to step 2	Fix reported issues Go to step 2
2. Run “UIA Verify” (download from here)	Go to step 3	Fix reported issues Go to step 3
3. Run “Inspect” (download from here) - only for focus and caret tracking testing	Go to step 4	Fix reported issues Go to step 4
4. Verify contrast, DPI, color, magnification (see next table)	Go to step 5	Fix issues Go to step 5
5. Run target accessibility tool	DONE	Contact tool vendor. Report issue to Microsoft.

For contrast, DPI, color and magnification verification (step 4 above), use the following tools and steps:

Test Criteria	Verification Tools and Steps
High-contrast	Control Panel - Ease of Access Switch to each of 6 High Contrast modes
High-DPI	Control Panel – Display Switch to 150% (1200x1000 min resolution) Switch to 200% (1600x1200 min resolution)
Redundant Color Information	Use a tool such as Fujitsu’s Color Doctor Try each of the 4 conversion filters
Magnifier tracks all keyboard navigation	Control Panel – Ease of Access – Magnifier Enable follow mouse/keyboard/insertion point Use WinKey+Plus / WinKey+Minus to change magnification
All UIs respect the accessibility system settings	Control Panel – Ease of Access
All documentation is accessible
All UIs have 5:1 or higher contrast ratio of text to background	Use a tool such as the Color Contrast Analyzer Install the luminosity and brightness version
All UIs can be used fully with On-Screen Keyboard	Control Panel – On-Screen-Keyboard (OSK.EXE)
UIs do not flash with rate between 2 Hz and 55 Hz	Use a tool such as PEAT (Photosensitive Epilepsy Analyzer Tool)

WPF Accessibility

WPF is a fully accessible stack. WPF uses UIA (in fact the original UIA team used to be part of the WPF team) to provide access to the various UI elements. All standard WPF elements (packaged controls, etc.) implement the necessary client-side UIA interfaces to expose themselves to accessibility readers.

Development and testing of nominally accessible WPF applications is fairly straightforward. For more information, refer to the “Accessibility Testing” section in the “WPF Application Quality Guide”.

While following these guidelines will result in nominally accessible applications, it is important to once again stress that all applications should also be tested with the target accessibility tool to confirm that they are truly accessible from the point of view of a disabled user.