Last week I spent some time discussing operational pilots with customers.
An operational pilot has great broad appeal. Everyone agrees that it’s a good, common-sense idea to “prove” the solution in a safe and containted environment.
Operational pilots are often used for some or all of the following reasons:
- Win over sceptical management
- Stimulate change management ahead of a broader roll-out
- Add some meat to a proposed financial benefits calculation, e.g. ROI, EVA, etc.
- Test some functional/technical aspect, such as business process verification, user interface, integration, performance, etc.
So there are political, user-centric business, and functional/technical reasons for launching an operational pilot.
It’s also sometimes good for team morale to ship something early in the project lifecycle.
All of this is good, and I agree – a small scale, controlled roll-out, properly managed and measured, can be really effective.
However, too often pilots rarely deliver as expected.
It seems to me that operational pilots are often proposed in the spirit of scientific rigour – “Let’s test our concepts in the real world, and see if our projections are realistic.” However, remember that around 99% of scientific experiements fail to achieve the anticipated result. Failure is good in science, failure is embraced in science. It narrows down the possibilities. However, I’ve never seen an operational pilot that began with a clearly defined set of failure metrics and consequences. I’ve never seen a meaningful feedback mechanism. And if something does fail, the business manager running the project invariably embarks on a spin campaign to keep the project alive.
Before you do a pilot, take a step back and think about what in fact you’re trying to prove. Too often I’ve heard vague, unquantifiable reasons such as “that it works” or “we all know the portal concept is right for us, this will just create lots of positive momentum.”
A great first step is a clear, unambiguous list of targets. This could be something as simple as “That the new system will achieve a 12% decrease in mistyped orders, and, through integration with our ERP, enable our mobile salesforce to complete orders within 2 minutes, instead of the current 45 minutes.”
Look ahead to the meeting where you present the results of the pilot to senior management.
Senior manager: So how did the pilot go?
Pie-in-the-sky pilot guy, fumbling in his briefcase: It went great. The sales team loved the new system. Why, just this morning I was chatting to a guy in the cafeteria, I forget his name. I think he’s some kind of sales-support person. Anyway, he was raving about it. So a huge success.
Normally, this is how pilot effectiveness is reported back to senior management. Contrast this with:
Senior manager: So how did the pilot go?
Metric-focused pilot guy, clicking on his notebook: Well, here’s the list of targets we agreed at the start of the project. As you can see we’ve exceeded all our targets, except the usability target, which we missed by 2%. We’ve subsequently carried out some user-testing of the interface that was causing this, and we’ve dealt with the problem. I’m especially happy about how the system enables the mobile workforce to get orders into the systems in under 1:40 minutes. And I have a report here from IT about the integration with the ERP system. They’re happy with how the system performed, and have given a greenlight for a broader roll-out. By the way, I’ve shared all these results with the sales team, and they’re all really excited about the system.
And of course metrics of this depth enable you to create a really convincing business using probabalistic benefits projection about broader roll-out.
So here are are some rules of thumb for running effective pilots.
-
Establish success metrics and the means to track them at the start of the project. Ensure you consider the political, business, functional/technical and user-centric aspects.
-
Within this set of metrics, establish priorities. Allow these priorities to establish the nature of the pilot. For example, if you feel that integration with a back office system is the biggest unknown, consider focusing an aspect of your pilot on this in order to quantify the risk. Don’t shy away from potential problems. If you avoid these now, they will come back to haunt you later.
-
Don’t confuse a pilot with the first-phase of the project. Once the pilot is complete, you must allow time to incorporate what you’ve learned back into the project. If there is no feedback, no back-tracking, then you’re in phase 1, not a pilot.
-
Actively manage the pilot. Don’t just launch it and hope it’ll work. You need to track usage, engage with users, measure results, and generally hold the things hand
-
Communicate positive results. Demos can initiate a surge of positive momentum. Don’t neglect this – it improves project team morale, it primes future user-groups, and it sends a strong message to management.