Community Based Development - Let's Talk
One thing that's been a major point of frustration for me over the last year is a conversation that goes something like this.
- [Me] - "We should totally do exams around XYZ and ABC"
- [Management] - "We only have budget for N developer exams this year. See if the product group has money."
- [Product Group] - "We totally want to help in every way possible, but we don't have any budget either."
So what happens in cases like this? It's pretty simple really - an exam does not get built. I want to fix this problem by turning the way in which we develop exams completely upside down and I want your thoughts.
First, one piece of background information. It is REALLY expensive to create an exam. The reason behind the expense has to do with the fact that for every part of the process we bring in individuals who are subject matter experts across the entire exam content domain - and these sorts of individuals don't come cheap. My contention is that for question development, anybody is capable of writing a good question for a specific objective, and we should stop limiting ourselves to only the folks who are experts across all the objectives.
So what do I want to do? Here are the high-level thoughts...
- Objective domain development via Wiki. As you probably know, the functional outline to which we write exam questions is called the objective domain. I would like to get more eyes (and hands) on this document than those of the 5-7 folks that we currently fly to Redmond since the purpose of the document is to outline the relevant skills required to support an assertion of competency in a technology. In my mind, editing and validation by more people equals greater relevancy for the exam/certification.
- Community-based question development. Currently, we hire 5-7 individuals and require them to write up to 30 questions each depending on the size of the item pool. There are a few really bad potential ramifications here.
- Authors must be subject matter experts across the entire domain - this comes at a premium.
- Authors must write a lot of questions each - this can lead to author burnout.
- Authors must potentially write questions on topics that they are not all that passionate about.
- The total number of items is all we get for an exam. If somebody comes out with a dump, the exam is exposed without much recourse (other than starting the whole process over again). Additionally, any questions that we kill (for whatever reason) are simply lost - there's no new questions to fill the spot - the result is simply a smaller question pool.
I want to create an infrastructure whereby anybody who is passionate about an objective is able to submit a question. Additionally, if the question is accepted into the pool, I want the author given credit on the Microsoft publication notes (prep guide, etc...). There are a TON of detail to work out here, but again, these are the high level notes.
So is this just another scheme to cut costs internally? Sure - there's a cost component. However, there are also a lot of other components that I think outweigh the simple element of cost.
- More coverage of technology. Under the current development model, we simply don't have the ability to scale. Community based development pretty significantly changes the rules of the game and allows us much greater flexibility.
- Better coverage. I'm going to stick my neck out a bit here and suggest that 50 people writing 1 question each will yield a better question pool than 1 person writing 50 questions.
- Continuous re-seeding. One of the biggest problems in terms of cheating is that the brain dump people know that once the question pool is built, those are the only questions that they need to steal. Under a community-driven model, new questions are continuously accepted into the question pool, even after the exam has been released. In my mind, this really changes the game from "Microsoft vs. the brain dump people" to "Community vs. the brain dump people".
In the end, it's a matter of tradeoffs. If we continue along our current model, then we're going to assume that our costs are essentially fixed. This really only gives us 2 areas where we can make tradeoffs: coverage and quality. Practically, we will not sacrifice quality. Therefore, the only axis left is coverage. The mindset behind community based development in terms of cost is not to reduce MSL's total cost of doing business - it's to reduce the cost of developing an exam so that we can build more exams. It's an effort to do more with what we have - to do what we do better.
So what do I want from you? Your thoughts, obviously!
- Are you, the community of certified Microsoft developers, engaged enough to participate in this kind of initiative?
- What are your initial reactions/thoughts to the entire idea?
- Where do envision the biggest pain points to be in executing this kind of vision?
- What am I missing - what might I not have considered?
I look forward to hearing from you.
I am currently the Editor-in-Chief for MSDN Magazine. I joined Microsoft in 2006 as a product planner with the certification team at Microsoft Learning. Prior to that, I spent my career as a developer and later as an architect. My main technology passions include pretty much anything on language theory, agile development, and service-oriented architecture.
