Grigori Melnik: Thoughts on Agile Software Engineering and Beyond : software testing

News: Acceptance Test Engineering Guidance BETA1

gmelnik — Sat, 01 Nov 2008 03:46:00 GMT

I am pleased to announce the release of the Acceptance Test Engineering guide BETA1. You can download it from http://codeplex.com/TestingGuidance

Common Scenarios Addressed:

How to Plan for Acceptance Testing
What Kinds of Acceptance Tests to Run
How to Create and Run Acceptance Tests
Defining What “Done” Means
How to Justify Your Approach

Contents at a Glance:

PART I – THINKING MODELS: an overview of acceptance testing and explains several models that are useful in conceptual thinking about acceptance testing. It also includes items that are necessary for planning acceptance testing. This part is intended to be read from beginning to end.

PART II – THUMBNAILS: a thumbnail is a short overview of a practice that explains what it is, when you may want to use it, the risks that it mitigates, and an overview of how to perform the practice. Thumbnails also include a list of references to papers, books, and other resources that provide more complete descriptions of the practice in question. The main purpose of a thumbnail is to describe a topic well enough to provide an overview, serve as a mental reminder for someone who has used the practice on how to do it, and give someone unfamiliar with the practice enough information about the practice and its applicability to determine if they want to learn more about it. This part is intended to be used as a reference; most readers will not read it from beginning to end.

PART III – SAMPLES: a collection of sample artifacts generated by applying different practices in a fictional real-world situation for Global Bank. These artifacts are embedded in a series of case studies of what the Global Bank team may have produced while building the application. The case studies provide some context to the individual artifacts. They also provide cross-references to the practices described in Part II. The artifacts are intended to be used as way to learn more about how to perform a practice; they can also be used as templates for your own artifacts.

Webcast

Gerard, Jon and I did a webcast talking about the concepts and the way we are writing this guidance.

Feedback

Your feedback is invited. Please post it under Discussions on Codeplex. Also, if you have an interesting experience with acceptance testing that you’d like to share and perhaps be profiled in our guide as a case study, we’d like to hear about it.

Acceptance Test Engineering guide - How to Decide if Software is Ready for You or Your Customer - Community Preview 2 released

gmelnik — Wed, 06 Aug 2008 17:32:00 GMT

On Monday, we released our CTP2 on Codeplex: http://codeplex.com/TestingGuidance.

You’ll see it now includes the consumption model and also some recommendations on how to read the guide. There are three parts. Part I. Narratives and Models - this is a short section that is meant to be read end-to-end. This is where we define readiness assessment and acceptance testing and introduce several models that help conceptualize software acceptance. Among other things we discuss decision making process, risk evaluation, the doneness model and its variants for agile and Tayloristic processes. Part II consists of what we refer to as “Thumbnails”. Those are the short descriptions of various useful practices that also provide information on their applicability, risks addressed, implementation options and additional resources. The Thumbnails are complemented by the actual samples (Part III) that we’ve produced on a sample system (Global Bank). Both thumbnails and samples are meant to be used as references and in any order.

You’ll notice there are still a number of empty sections. If you see an empty section, assume we will fill in the section before we complete the book.

Please provide it by posting a reply to the corresponding thread under Discussions on codeplex or as a reply to this blog post.

Happy reading! We look forward to your feedback!

Do you have testing problems?

gmelnik — Fri, 27 Jun 2008 02:58:00 GMT

In our quest to produce actionable acceptance testing guidance, we are looking for hard-to-test scenarios from real world projects so that we could test our guide against them. We would appreciate if you shared your testing problems with us.

For example, you may find testing the logics that generates binary data (like image generation, voice synthesis) hard.

Also, feel free to tell us how you solved them if you did.

Link to survey.

Upcoming conferences - where I'll be (May - Oct 2008)

gmelnik — Sat, 31 May 2008 17:45:00 GMT

Tech-Ed 2008 Developers, June 2-6, Orlando, FL. I’ll be giving a talk on the Enterprise Library (DVP02-TLC). I’ll be also hanging out in the exhibition areas (mainly, Process & Project Management (DVP- VSTS 2008)).
CAST 2008 , July 14-17, Toronto, ON. I look forward to good interaction with testing experts and thought leaders from around the world.
Microsoft Product Fair, July 22-23, Redmond, WA.
TechReady7, July 28-Aug 1, Seattle, WA. I’ll do a couple of talks on the Enterprise Library and agility.
Agile 2008, Aug 4-8, Toronto, ON – the major world conference on agile software engineering. Mary Poppendieck and I are program-charing the conference.
STARWest 2008, Sep 29- Oct 3, Anaheim, CA. Jon Bach and I will give a talk on Acceptable Acceptance Testing, telling the tale of our team producing the guide on acceptance testing and dogfooding our own recommendations.
PNSQC2008, Oct 13-15, Portland, OR. We have a paper "Acceptance Testing: a love story in two acts". No Shakesperian-level drama, but some exciting stuff for sure.
PDC 2008, Oct 26-30, Los Angeles, CA. Together with Mary Poppendieck and other colleagues from p&p, we’ll be presenting the Agile Perspectives pre-conference event.

If you are planning on attending any of these conferences, please come by and say hello. I would love to hear your stories of using or reasons for not using the Enterprise Library, Unity, other p&p assets. I would also be glad to chat with you about our upcoming guidance on Acceptance Test Engineering and hear your experiences, lessons learnt, and suggestions.

Also, I talk to user groups regularly. If you are from the city I am visiting (or nearby) and would like me to give a talk to your user group (agile, .NET, software testing), feel free to ping me.

Acceptance Test Engineering Guidance - project kick-off

gmelnik — Tue, 08 Apr 2008 20:07:00 GMT

Traditionally, p&p primary audience included developers and architects. Even though we have some coverage of the testing discpline (in particularly Testing .NET Application Blocks - Version 1.0 guide and Performance Testing Guidance for Web Applications), it is a small portion of the p&p assets [link to catalog]. In our interactions with customers, we hear requests for good guidance on testing – all kinds of testing.

So, back in the fall 2007, I’ve put a number of projects related to test engineering and test automation on the patterns & practices backlog. Several other important projects took precedence (GAT/GAX, Unity, Enterprise Library). And now, after we have shipped GAT/GAX 1.4 and Unity 1.0, I am happy to be able to kick off this project with the focus on acceptance test engineering.

The core team consists of Michael Puleio, Jon Bach and I, Grigori Melnik. Michael is not a tester but a great developer with passion for testing, test automation and test tools. Jon is a professional tester, he is the manager of corporate intellect at Quardev Labs and a co-inventor of session-based test management for managing and measuring exploratory testing. I have devoted a number of years researching executable acceptance test-driven development (with FIT) and the relationships between software requirements and acceptance tests (see this article with my stance on this).

The topics we plan to focus on in this project include:

- test objectives and strategy,

- product readiness/acceptance,

- defining and reconciling good-enough criteria in various industrial contexts,

- working with customers and customer-proxies,

- supporting stories/requirements with acceptance tests.

We intend to support our guidance with case studies and exercises from the real world.

We are running this project as an agile project, with weekly iterations, standups, collocated team, etc. This deserves a separate blog post, which I’ll probably do later this month.

In the meantime, feel free to post your comments and thoughts on any specific (painful) aspects of acceptance testing that you would like help with. Also, if you have an interesting experience with acceptance testing that you'd like to share and perhaps be profiled in our guide as a case study, we'd like to hear about it.