Some Agile and/or ALM adoption efforts are canceled midstream because of lack of understanding of the basics of finding a suitable candidate development project. I have seen in more than a single situation that the chosen project is cutting edge in all three aspects of technology, process and people:

• The technology is brand new, or new to the team, sometimes in even more than a single tier (for instance, new database software coupled with new UI development tools and a new programming language)
• The development process is being changed (say from waterfall to Agile)
• New people are being added to the team just after receiving their first training in the new technology

But the biggest mistake with Pilot efforts is to to use a strategic, high profile brand new project as the proofing ground for all these aspects, all at the same time. This is more common than expected. It starts as something like this:

1. Business has some urgent need for strategic functionality that allows them to challenge the existing technical architecture
2. However, the effort still has to abide by the usual existing waterfall processes that dictate that all must be done in a single pass
3. So the project is approved, but no cycle is allowed to try out the new tools and processes in a smaller context , and multiple changes to the environment are bundled together in an insurmountable ticking bomb that will later explode as a "death march" project.

To add insult to injury, sometimes on top of all this no proof-of-concept is ever tried with the new technology and processes (Proof-of-concept differs from pilot in that it is done in a lab environment, with no impact on business). Pilot projects do have business justification, but usually one chooses a minor project instead of betting the "jewels of the crown" on risk upon risk.

The mistake on all these lies usually in the governance management tier(PMO, office of the CIO, etc). It is usually associated with just enforcing the status quo, and it takes some brand new business need to act as a catalyst to challenge it. This governance tier should be the one to understand how to evolve their environment through carefully taken steps, and to know how to spread the risk underlying the business need into preparatory small projects (using proof of concepts and pilots) that will pave the ground for more ambitious ones.

If a governance tier is not active in doing this, the new project decays into a rogue that just reinforces the "didn't tell you so" attitude of those who see governance only as keeping IT madness in straightjackets.

Allowing this to happen is like building on moving sand: the construction might be impeccable but will collapse upon itself if it doesn't have firm ground to support the pressure of adding new layers.

The best practices for selection of a Pilot project are widely known, and for quite a long time. Here is an excerpt from a Microsoft Official Curriculum course from 1993. It is part of Course 124 - Managing the Migration to Client-Server Architectures. I modified the text to fit ALM adoption (the text in brackets [] replaces "client-server" and updates the context of other points):

"Start small - with a Pilot Project

We suggest you start your exploration of [new ALM processes and tools] with a pilot project.

• Maintain excitement:
  • Sponsors will lose enthusiasm
  • Team members will lose enthusiasm
  • Reduce risk of turnover
• Need strategic answers quickly to be of value.
• Avoid management problems of large projects:
  • Large projects require more layers of management
  • Coordination of client developers and server developers is critical
  • Coordination will be much easier in a small group that talks to each other

Selection Criteria of Pilot Projects

• Well defined data requirements
  • Don't want to get bogged down in data analysis
  • Could be existing application
  • Could be part of a new application, where data analysis has been completed
• Benchmark available
  • If don't have, need to build in-house benchmarking capability
  • Performance criteria
  • Quantify savings and benefits
  • Define ball-park expectation
  • Use to validate tool selection
  • Use for quality control in future projects
• Decision support application [Business Intelligence in today's jargon] as opposed to data entry application
  • More showy, if that's what's needed
  • Safe place to start - it won't disrupt business operations
  • Usually a simpler system
  • Deliverable flexibility - keep concentration on testing the [ALM processes and tools]
• Committed and supportive users
  • Might be #1 critical success factor [that includes not only end users of the application in the role of product managers, but also developers, project managers and upper management]
  • Willing to work with the team
  • Willing to allocate the time required for the project
  • Could use internal IT system so "end users" are IT
• Low Cost
  • Use equipment you already have [for instance, VPCs]
  • Look for idle equipment [for instance, a PC with Windows XP can be a build server for a small project]
• Low Risk
  • It's better if this might be considered a throw-away project
    • Objective is to evaluate [new ALM processes and tools], not build an application. Concentrate on tools and platform rather than application development"
  • If you need to choose a project that is mission critical, at least let it not be time-critical

As you can see, those best practices are nothing more than codified common sense, and I highly recommend you have those in mind when scoping your next ALM project.