Charles Eliot's Take On The World

The Technical to Customer-Focused Axis

2006-07-12T06:51:00Z

There's a drawing I've been making for a while now that argues for a beneficial overlap in the skills for program managers and developers.

Imagine an axis that goes from "Technical" at one end to "Customer-Focused" at the other end. Let's also imagine that we can agree on what these terms mean, at least at a rough level, without going into a labored attempt to come up with definitions. Let's also imagine that we can accept for the purposes of constructive argument the simplistic notion that this is a one-dimensional, reifiable continuum, and not the rich combination of intelligence, skills, attitudes, behaviors, and aptitudes that it surely is in fact.

Technical Customer-Focused
-------------------------------------------------------------------

Case 1 - Entrepreneurs, start-ups
+-----------------------------------------------------------------+
| |
+-----------------------------------------------------------------+

Case 2 - Good overlap
+----------------------------------+
|              Dev              xxx|
+----------------------------------+
                               +----------------------------------+
                               |xxx              PM               |
                               +----------------------------------+

Case 3 - Bad overlap
+----------------------------------+
| Dev |
+----------------------------------+
+----------------------------------+
| PM |
+----------------------------------+

Case 4 - Bad lack of overlap
+-----------------------------+
|            Dev              |
+-----------------------------+
                                    +-----------------------------+
                                    |             PM              |
                                    +-----------------------------+

Case 1 is what you get in technical entrepreneurs and small startups. Technical entrepreneurs have a rare ability to understand customer needs deeply and translate those needs into code. In small companies, on the other hand, there's no room for specialization so every employee has to run the full gamut as far as they are able. A company doesn't have to grow very big before it is happy to start moving away from this model. (Extreme Programming advocates will assert that every developer can be customer-focused, as long as the developer has ready and direct access to customers. I have seen too many counter-examples to believe that this can really be universally true.)

Case 2 is, in my opinion, ideal, even with the areas of potential overlap (marked with x's). Potential overlap invites potential friction. I believe, however, that a team that learns how to collaborate effectively and to manage the occasional overlap-related problems will function better in the long run than a team that tries to avoid turf wars by precisely delineating the Dev/PM boundary.

Case 3 is an extreme example of what happens when specialist PMs try too hard to be technical. Not only is there maximal possibility for corrosive turf wars, but there is a big gap left on the right, at the customer-focused part of the axis. The net effect is technically brilliant products that don't meet customer needs.

Case 4 is what happens when a turf war has erupted and the Dev and PM communities have contracted into their mutual domains. The net result is that PMs write requirements specs and "throw them over the wall" (or in this case, over the gap) to the Devs. Devs will increasingly argue that big parts of the product don't need to have functional requirements specs at all because everything that matters is in the code design and implementation. The functional requirements specs will also suffer from not being written with insight into whether the feature can actually be coded efficiently.

Heuristics of Software Testability

2006-05-23T04:42:00Z

These heuristics are from James Bach of Satisfice, Inc. I highly recommend a trip to the company’s web site at http://www.satisfice.com.

Bach addresses the fundamental testability concepts of controllability and observability right at the top, but also adds some useful process insights.

Controllability

The better we can control it, the more the testing can be automated and optimized.

• A scriptable interface or test harness is available.

• Software and hardware states and variables can be controlled directly by the test engineer.

• Software modules, objects, or functional layers can be tested independently.

Observability

What you see is what can be tested.

• Past system states and variables are visible or queriable (e.g., transaction logs).

• Distinct output is generated for each input.

• System states and variables are visible or queriable during execution.

• All factors affecting the output are visible.

• Incorrect output is easily identified.

• Internal errors are automatically detected and reported through self-testing mechanisms.

Availability

To test it, we have to get at it.

• The system has few bugs (bugs add analysis and reporting overhead to the test process).

• No bugs block the execution of tests.

• Product evolves in functional stages (allows simultaneous development and testing).

• Source code is accessible.

Simplicity

The simpler it is, the less there is to test.

• The design is self-consistent.

• Functional simplicity (e.g., the feature set is the minimum necessary to meet requirements)

• Structural simplicity (e.g., modules are cohesive and loosely coupled)

• Code simplicity (e.g. the code is not so convoluted that an outside inspector can’t effectively review it)

Stability

The fewer the changes, the fewer the disruptions to testing.

• Changes to the software are infrequent.

• Changes to the software are controlled and communicated.

• Changes to the software do not invalidate automated tests.

Information

The more information we have, the smarter we will test.

• The design is similar to other products we already know.

• The technology on which the product is based is well understood.

• Dependencies between internal, external and shared components are well understood.

• The purpose of the software is well understood.

• The users of the software are well understood.

• The environment in which the software will be used is well understood.

• Technical documentation is accessible, accurate, well organized, specific and detailed.

• Software requirements are well understood.

A Framework for Thinking about Testability

2006-05-23T04:34:00Z

Introduction

Testability Is About Cost

There are lots of definitions of “testability”, but this is my favorite:

“[The] effort required to test a program to insure it performs its intended function”

I like this definition because with its first word it puts the emphasis squarely where it belongs: on the cost of testing software. If you are testing to a defined quality bar, testability means that you get to that quality bar faster. If you are working to fixed time and resources, testability means that you will ship higher quality software.

Testability Is About Automation

To a large extent, imparting testability to a system means making the system more amenable to automated testing, so you can think of testability as “the ability to build test automation that can do more, do it faster, do it more reliably, and do it at lower cost”.

Testability Involves All Disciplines

It is easy to come to the mistaken belief that testability is just something that developers implement for the benefit of testers: more hooks, more interfaces, different architectures, etc. Testability also involves other disciplines. For example, for software to be testable we have to know what it is supposed to do when it is working correctly. This is called the “oracle” requirement, and it is usually PM specs which fulfill it.

Testability Is Also About Process

Testability is more than just an attribute of the system being tested. It is strongly affected by the processes used to build the system. For example, testability is reduced if software changes can abruptly break all the UI test automation.

The Core Concepts of Testability

From a tester’s perspective, there are two core concepts that together make a system more testable: controllability and observability.

Controllability refers to the ability to place the system into any desired state, and to control the inputs to the system. Controllability is closely related to the ability to isolate a system component by building software shims “below” the component and drivers “above” it. To quote from Binder (see below for ref): “A component that can act independently of others is more easily controllable.”

Observability refers to the ability to know the exact state of the system at any time, and to observe the outputs of the system. Observability covers the entire domain of the system, not just the limited information available through public interfaces (client UI, public APIs, etc). For example, the full state of an object in the CRM system includes the data held in the database, instance-specific local data maintained by the Platform, cached data, XML data transferred to the client, and data as rendered in the client UI. Many tools can be used to increase observability: asserts, traces, dumps, logs, etc, as well as the shims and drivers mentioned above.

The other core testability concept that deserves to be mentioned at this time is sensitivity.

Sensitivity refers to the ease with which a fault-causing change in the system (“mutations”) can be detected. For example, when a Platform API change halts the entire CRM system, we would say that the system is showing high sensitivity to the mutation. Despite the dramatic effect and the bad things this scenario might say about change control processes, we all understand that this high degree of sensitivity to mutation is actually a good thing. One of the biggest benefits of test automation is that it increases the sensitivity of the testing system by decreasing the cost of regression testing.

Testability Factors

In his article “Design for Testability in Object-Oriented Systems” (Communications of the ACM, September 1994, Vol 17, No 9, p 87), Robert Binder discusses in detail six factors that influence testability: representation, implementation, built-in testing, test suite, test support environments, and process capability.

Representation

Representation refers to the availability of an accurate and precise model of correct system functioning. If testers do not know how the system is expected to function, they can play with it, but they can not actually test it.

Representation refers in part to the quality of the system architecture documentation; the functional requirements specifications; and the software design specifications. For maximum testability, the representations of the system from which testers work must be complete, correct, and current (meaning that the representation reflects what is actually implemented).

Another useful attribute of the system representation is its traceability, which is the ability to determine which software component implements a given feature, and which features implement a given requirement. Traceability is a book-keeping function, and while it is conceptually straightforward and often requested, it is not easy to introduce.

Implementation

Many aspects of the way a component or system is implemented will affect its testability. Most of these factors are well-known and fall under the general heading of “good software design”, but three general dimensions are note-worthy:

- Complexity. The more complex the implementation, the harder it will be to test.

- Scope. The more the implemented component does, the more tests will need to be run to cover the component’s scope.

- Determinism. If the implemented component requires asynchronous cooperation with other tasks, it will be harder to test. When a system is inherently designed around cooperating asynchronous components (or services), it pays to think hard about how to make the overall system more testable by increasing the determinism of each cooperating component.

Built-In Testing

Highly testable systems have lots of test functions built into the system itself. For example, the ANSI/IEEE 1149.x standards define interfaces that can set the state of any of the millions of gates on a VLSI chip, report on the state of any gate, send specific inputs to any gate and report the outputs. The testability of VLSI chips is improved enormously by the existence of these interfaces.

In a similar vein, an object-oriented system should have interfaces on every object that can set/reset the object into any desired state (possibly including illegal states, and states involving implementation-specific instance variables), and report back all the state data (internal and external) for the object.

Note that set/reset and reporter methods, to be useful, break object encapsulation and therefore have security requirements. A common approach to this problem is to limit these functions to debug builds. While simple, this is not always the best approach, especially for debugging live systems that cannot be taken down to introduce an instrumented version. In this case it is a better approach to build a run-time security scheme into the set/reset and reporter methods.

A well-known way to introduce built-in testing capability is through code assertions (“assert statements”). A lot has been written about the best ways to use assertions (to ensure that the internal logic of an implementation is maintained, to check that invariants are truly invariant, to implement observable tracing or reporting functionality, etc.), so I will not go into details here. I will, however, note that one of the key uses of assertions to increase testability is to police the extent of unit testing or class testing once a new component is integrated into the full system. In other words, assertions can be placed into the component code during unit testing to say “this component is not known to work outside these ranges”.

Test Suite, Test Support Environments, and Test Process Capability

These three factors address the way tests are authored, recorded, etc; the environment and tools used for testing; and the surrounding team processes (change control, build verification tests, spec inspections, etc). Detailed discussion of these factors is beyond the scope of this article.

CRM Team blog open for business

2006-05-02T02:04:00Z

The Microsoft Dynamics CRM development team just started a team blog. It's at http://blogs.msdn.com/crm/, and it's well worth a visit, if only because the first full article was written by me!

TechEd 2006

2006-04-29T02:19:00Z

I'll be giving an overview presentation on Microsoft CRM 3.0 at the TechEd conference in Boston. The session number is BAP202, and it will be on Monday 6/12 at 10:45am.

It will be a fun session. CRM 3.0 was a huge step forward for us, and I always love showing off the system. Plus I'll be able to talk about how CRM 3.0 fits into the bigger picture of Microsoft's business and productivity applications.

Welcome!

2006-04-29T00:45:00Z

Hi there!

My name is Charles Eliot, and I'm the Group Program Manager for Release Management for the Microsoft Dynamics CRM team, which is just a fancy way of saying that I'm the lead project manager.

I joined the CRM team in late 2001. I started at Microsoft in 1992 as a contract tester, and was hired on full-time about 6 months later as a test lead. In 1994 I jumped over to program management, and have been a program manager (PM) ever since. From 1992 until I joined the CRM team I worked entirely on email products: MS-Mail 3.2, Exchange 4.0, Exchange 5.0, Exchange 5.5, and finally Exchange 2000.

I joined the CRM team as the lead PM for system management tools (including workflow), but I started taking on release management responsibilities almost immediately. When I joined we hadn't shipped anything yet, so it's been a blast to have been involved in all the CRM releases to date: v1.0 in January 2003, v1.2 in October 2003, and most recently v3.0 in October 2005.

I'll be blogging about whatever strikes my fancy. You can expect lots of stuff about the challenges of managing large software development projects, but I'll also be writing about the CRM field in general, and of course Microsoft Dynamics CRM in particular.