After pondering the recent drives for "personas" testing a bit this weekend, I believe more strongly than ever that, while useful and definitely a step or more above purely functional testing, it is NOT a panacea for customer experience testing.

For those who are not familiar with the term, a "persona" is a sort of user profile designed to describe a particular type of user of a product or service.  Typically there are mulitple personas designed to be able to describe a range of "typical" users of the software.  One example might be a 16 year old teenage boy who likes hip hop music and cars and uses a piece of software to share music videos with his large group of friends.

These personas are often given names and you may see them used as an example in a bug report - "Jim" would find it hard to upload the files to share with his friends -- or similar (as an example). They are very specific to the software under test and may be created with the input of marketing and sales as well as program managers/designers.

There are some basic flaws with Persona testing that, if they are not addressed and mitigated, can really hurt the efforts toward better customer experience testing when personas are introduced or utilized.

One of these flaws is that the personas do not typically cover all classes of customers that testing should be mindful of. 

When I am working to design test cases, there are really three major classes of customers that I have try to keep in mind and "see through the eyes of":

  1. Customers of the product who use it as intended and for whom it is targeted.
  2. People who are not current customers of the product but for whom it is targeted and who will use it as intended.
  3. People who are not current customers of the product but for whom it is NOT targeted and who may not use it as intended.

Almost all personas tend to be focused on that first category with, perhaps, a few forays into the second category. I have never seen a published persona that covers the third category.  While this makes sense in some ways because it focuses testing efforts on the target audience, I think it's also artificially limiting, especially to less experienced testers.

When a person is told to test against a set of personas, the tendency is to no longer think outside the personas you are presented with which can cause some interesting bugs to remain undiscovered (including security bugs) and even cause a product to miss fulfilling a market that was unrecognized because of a basic flaw that could have been corrected.

Pre-defined personas also have a tendency to cause testers to accept the "wrappings" of the product and focus on functional issues for the personas.  If a tester is testing against the persona example I gave earlier for a music sharing service, would they pay sufficient attention to other things that might be important like look & feel or compatibility?  True, these are not necessarily persona issues - these may be global issues - but if they are missed because of a focus on only the defined personas, that would be a disservice.

One thing I have seen happen is that a set of personas is presented to testers who have no real familiarity with the point of view of the personas.  This has happened in particular in cross-cultural situations where the persona and the tester have very difference basic points of view and it can be very hard to get good test cases/test results from this. 

The very painful part of this is that these same testers had HUGE benefits to offer in broadening the perspective of the testers who designed the personas but didn't feel free to offer that input because it was outside the box of the defined personas.  In the modern software world, this is a big hit to take.

Unfortunately, I have never personally seen cross-cultural personas offered.  It may be simply because I am locked into the United State/English market and do not tend to do localization/globalization testing but I don't think that this is the sole cause.  I would love to see education of cross-cultural/cross-market personas to broaden everyone's horizons! 

The other issue with personas is that I do not believe they are updated often enough.  I have seen the same exact personas used for several years without a significant review/revision to them and often this results in personas that are out of date.  They are also often developed with a small team of people and never given a large scale review.  Although the review can be a big thing to organize, many valuable points of view can be collected and a more effective set of personas can evolve from it.

The other thing that can keep personas valuable is to review the customer satisfaction data of the software after release and see how well your personas match reality.  There may be cases where the match is great but the assumptions of testers on what was important TO that persona was off and this can be corrected and is a learning experience for everyone. 

Persona testing is a valuable way to start and focus customer experience testing but a few pitfalls need to be constantly monitored to prevent this technique from causing its own set of expensive issues:

  • Keep personas current
  • Consider all customer types
  • Consider all customer locales/markets
  • Think outside the persona "box" as well as inside it
  • Keep your own set of personas including "hacker" and "pirate"
  • Conduct large scale persona reviews with multiple software disciplines
  • Don't limit customer experience testing to only persona focii

Persona testing is definitely not a panacea but it can start testers thinking of customer experience and customer satisfaction