J.D. Meier's Blog

Software Engineering, Project Management, and Effectiveness

Scenario Scoreboards for Testing Your Product Success

Scenario Scoreboards for Testing Your Product Success

Rate This
  • Comments 0

Here is an example of a simple, but surprisingly effective way to test your product success. Around patterns & practices, a few of us affectionately called this a “consumer reports” view.  Whether your product is prescriptive guidance or code, the concept is the same – you’re testing your product against user scenarios.

To create a Scenario Scoreboard, simply put together a matrix of scenarios. Next, add criteria. Next, test against the scenarios and rate the criteria.  Note – There are two keys to making this successful: 1) have a useful collection of scenarios, and 2) get agreement on the criteria.  You want agreement on the criteria so that people can stand behind the results.  Tuning your criteria can help you tune your product success.  You’re basically creating a feedback loop to measure the success of your product against user scenarios.

Example of a Scenario Scoreboard
What’s important in the example is the simple frame:  scenarios and criteria.   The scenarios are grouped into buckets.

Category

Scenarios / Tasks

Criteria

Best Practice Compliance

Implementation Complexity

Implementer Competence

Time to Implement

General

  • How to push peak loads to the cloud to reduce the size of an on premise data center.
  • How to decide if you application is right for the cloud (on-premise vs. cloud, advantages/disadvantages)
  • How to run your own VM in the cloud.
  • How to run an on-premise app in the cloud.
  • How to create a small-to-medium Web app.
  • How to create a large Web application.
  • How to create a parallel processing application.
  • How to store blog data for an on premise application.

Arch / Design

  • How to structure your application.
  • How to partition based on latency.
  • How to scale horizontally.
  • How to scale vertically.
  • How to manage state.
  • How to design for Hybrid  (i.e. part of the application runs on premise, part in the cloud)

Auditing and Logging

  • How to log information.
  • How to throttle your logging.
  • How to determine your log destination
  • How to view logs

Authentication

  • How to authenticate callers.
  • How to perform single sign on (Federation).
  • How to build an STS.
  • How to identify callers.
  • How to manage personally identifying information / sensitive data in the cloud. 

Authorization

  • How to use roles to protect resources.
  • How to leverage claims.
  • How to turn your application into a claims aware application.
  • How to turn claims into roles.

Caching

  • How to use local storage.
  • How to leverage a distributed cache.
  • How to swap out cache providers.
  • How to cache data effectively.
  • How to expire the cache.

Configuration

  • How to cache configuration data.
  • How to decide what settings you should use.

Data Access

  • How to design an extensible schema that will never need to be updated.
  • How to choose a partition key for different entities.
  • How not to get too much data into one partition.
  • How to repartition your live data.
  • How to load initial/domain data.
  • How to do BI in the cloud.

Monitoring / Health

  • How to monitor web roles.
  • How to monitor worker roles.
  • How to alert/alarm.

Performance and Scalability

  • How to simulate load tests.
  • How to compare blog storage against local files.
  • How to measure performance against CRUD.
  • How to do capacity planning.
  • How to access/view Performance Counters

State

  • How to design statelessness.

Storage

  • How to decide which storage to use.
  • How to organize your containers and blobs efficiently.
  • How to track/retrieve additional information/properties about blobs
  • How to authorize access to containers/blobs

Transactions

  • How to roll back.
  • How to update multiple pieces of data at the same time.
  • How to implement 2-phase commit.
  • How to lock effectively.

Worker Model

  • How to schedule work.
  • How to group different types of work.
  • How to communicate between different types of worker roles.
  • How to determine the number of worker roles.
  • How to determine if multiple threads should be used

Workflow

  • How to design for asynchronous work.
  • How to design for integration (custom cloud applications / finished services / on premise / ESB.

Deployment

  • How to deploy seamlessly with no downtime
  • How to architect your application to allow data schema deployment to happen independent from app deployment

Ratings for the Evaluation Criteria
Here are some examples of ratings for the criteria:

Rating Category

Criteria

Best Practice Compliance Ratings

  1. Not possible
  2. Developer implements
  3. Developer extends
  4. Wizard
  5. Transparent

Implementation Complexity Ratings

  1. Large amount of code
  2. Medium amount of code
  3. Small amount of code
  4. Wizard +
  5. Wizard

Quality of Documentation and Sample Code Ratings

  1. Incorrect or inaccurate
  2. Vague or Incomplete
  3. Adequate
  4. Suitable
  5. Best Practice Documentation

Developer/Administrator Competence Ratings

  1. Expert (5+ years of experience
  2. Expert/intermediate (3-5 years of experience)
  3. Intermediate
  4. Intermediate/novice
  5. Novice (0-1 years of experience)

Time to Implement

  1. High (More than 4 hours)
  2. Medium to High (1 to 4 hours)
  3. Medium (16-60 minutes)
  4. Low to Medium  (6-15 minutes )
  5. Low (5 minutes or less )

Additional Resources

My Related Posts