Follow Up to Follow Up on Community Based Certifications

I'm responding to a response on my post about what it would look like to do community based question development.  The major take away for me on reading the following response is that I probably should take you through (at a high level) what our process actually looks like today.  I'll do this in a follow up post.  For now, my comments are in red.

I'm responding here so that I can get our readers involved in this conversation as well. I will be providing a link to this post on Howard Dierking's Blog as well.
I have read all the comments on Howard Blog, as well as comments on Trika's Blog and Deepak Kapoor's Blog
My Thoughts
Of all the points made about this subject, I have to agree with Deepak the most. The first of Deepak's statements that I'd like to elaborate on is:

quote:

---

Originally posted on Deepak Kapoor's Blog
"...creation of exam content by community must be moderated and channeled through strict review procedures. This is to ensure that candidates who will appear for exams will get the right value for money. Implementing a rigorous quality assurance process will require budgets, and this takes us back to the main issue here."

---

A point that I would like to add is the fact that involving a community would require years of preparation that would also cost alot of money that the current budget, obviously, doesn't have.

No disagreement that there would be cost involved. However, as I’ve said, I envision the cost being more related to the cost of infrastructure – and this type of cost is something that can be amortized across all exams. Additionally, I’m confused as to why it would take years to involve a community – especially if the community already existed (such as the MVP community).

Questions have to be answered:
1. Who will head-up the community involvement? (You can't just let a community loose on the creation of an exam with nobody to corroborate the information) :Budget+

I don’t think that we ever envisioned an unmanaged, un-moderated community development effort – Again, the cost is for infrastructure and can be scaled and amortized.

2. Once the information has been collected, who will weed out the numerous duplicate/redundant/worthless/pointless questions and make sure that only the relevant questions make it into the pool? :Budget+

Part automated workflow utilizing the community – part Microsoft internal content development managers. The first part falls under infrastructure costs – the second falls under Microsoft headcount costs (which are assumed as irrelevant for this discussion).

3. Who is going to keep people from bickering about those numerous duplicate/redundant/worthless/pointless questions and make sure that people focus on what is relevant and not what they BELIEVE other's should know? (SMEs KNOW what others should know) :Budget+

Firstly, a question should never be seen by the entire community. The entire pool would still go through QA review by a SME panel as I’m envisioning the process working now. However, I think that it’s a pretty bold assertion to suggest that SMEs know what others should know. I have witnessed SMEs do exactly what is suggested that they are above (bicker about what they believe people should know). The question is not whether or not the bickering is good (in my opinion, it is very good), but whether or not it should be just between 5-7 people. Additionally, we see complaints related to relevancy on SME designed and SME written exams – so I’m not sure that the SME designation quite gets a person to the level of “know[ing] what others should know).

4. Once a fraction of those questions have been processed into the pool, who will determine the Answers are worded so as to not create serious confusion amongst the candidates? :Budget+

Microsoft internal content development managers – the same way it happens today irrespective of SMEs.

5. Now the Questions and Answers have been collected, weeded, and sorted, who will approve them? :Budget+

Microsoft content development managers, myself, the product group, and a panel of QA SMEs – the same way it happens today.

Pain Point #1: The more hands that are in the pot, the harder it becomes to control the product. If you have a community of people writing questions, you're going to have a harder time regulating that community. :Budget+

This statement by itself doesn’t have any direct causal relationship to an increase in budget. Can you elaborate?

Pain Point #2: It would cost more money in the long run to recall those (now displaced/disgruntled) SMEs and pay them more money to repair a problem that they had under control in the first place. :Budget+

You’re making a pretty big assumption here about how things work today. I can’t really go into a lot of those details, so I’ll ask you to elaborate on a few points and maybe I can respond to more detailed statements. Firstly, why would it cost more to invite a SME to participate in something for which they make money? Secondly, what would need repairing exactly? Thirdly, how are you able to assert that the problem is under control today – and would continue to be going forward (under the current model)?

As it stands now (you say) there are 5-7 SMEs that are writing exams in specific areas where they are the experts. From what I understand, those Experts are involved in many different aspects of creating an exam. To replace them with a community of people that don't have the knowledge that the SMEs have, you're asking for trouble.

This is a faulty assumption (I say). We *very rarely* (though we have had to in 1 exam I’ve owned) utilize an expert in multiple phases of exam development. We typically bring in completely different SMEs for different phases. Again, also keep in mind that I’m not suggesting a complete “hand off” to the community. I’m suggesting community based question development. To take this statement and assume I am talking about the entire exam development process is to hold some invalid assumption about that process.

It takes professionals to Plan Examinations. What you're asking for would be similar to asking College Students what they would like to see on next week's exam. It is simply something that cannot be done.

Firstly, the initial planning of the exams is a collaborative effort between me and the product group. Secondly, while I did suggest that Wiki be used for the creation of the objective domain (functional outline), I’m not married to the idea. This could still be done via a SME panel and published to the community to use in writing the questions. However, I’m still not convinced that it couldn’t be done in a broader sense. It would simply need to be closely moderated.

Back to Howard's Points
Community-based question development.

quote:

---

Originally Posted on Howard Dierking's Blog
1. Authors must be subject matter experts across the entire domain - this comes at a premium.
2. Authors must write a lot of questions each - this can lead to author burnout.
3. Authors must potentially write questions on topics that they are not all that passionate about.
4. The total number of items is all we get for an exam. If somebody comes out with a dump, the exam is exposed without much recourse (other than starting the whole process over again). Additionally, any questions that we kill (for whatever reason) are simply lost - there's no new questions to fill the spot - the result is simply a smaller question pool.

---

My responses
1. (a)Good. That proves to the Candidates that the Exams will be worthy of taking, versus someone figuring that a community of misfits pulled the questions out of their....
(b)In addition, this 'premium' ultimately comes at a lower cost that the potential losses that may acrue because of the overall lack of relevant knowledge of a community based system.

Take exam 70-536. Is a person who is not capable of authoring a good question on multithreading or GDI+ capable of writing a good question on the collection classes in the BCL?

2. Highly Doubtful. Possible, yes, but doubtful. Writer's block would be a more feasible reach for an answer than Burnout. What do you think the burnout rate of 1000s of community members would be?

I was a question author before joining Microsoft. It is not even close to highly doubtful – in fact, it’s highly common.

3. Passionate or not, they get paid to write those questions because they know the answers before the questions are asked.

Also, this statement assumes a lot – when you see how difficult it is to write a good question as an author, you see that unless you have passion around that area, you tend to write questions that are more knowledge-based and less task based.

4. Ok, I do not agree with their methods on "Sealing" an exam once it has been released and I do believe that questions should be replaced once they are removed (If not for security purposes, at least to keep the candidates honest), but I still do not agree that a community of under-qualified IT Professionals should be writing the exams. If more people want the ability to write the exams, then more people need to work on their qualifications of becoming Microsoft Certified Trainers.
Besides, there is already a community that does this. It's called the MCT Community.
According to Microsoft

quote:

---

Originally published on http://www.microsoft.com/learning/mcp/mct/default.mspx
Microsoft Certified Trainers (MCTs) are the premier technical and instructional experts in Microsoft products, technologies, and solutions. To ensure a superior learning experience for our customers, MCTs are the only individuals that Microsoft authorizes to deliver training that is based on Official Microsoft Learning Products or Microsoft Dynamics Learning Products for the MCT program.
The MCT community spans more than 140 countries and includes classroom and e-learning instructors, learning consultants, authors, conference presenters, user group leaders, and more. All are united by passion and talent for helping Microsoft customers and partners realize their full learning potential.
MCTs meet high initial and on-going certification requirements and enjoy access to many benefits. These include free access to the complete library of Official Microsoft Learning products; substantial discounts on exams, books, and Microsoft products; members-only newsgroups and online community resources; and invitations to exclusive events and programs.

---

Yea – I’ve read the marketing materials too.

If anyone thinks they can do the job of an MCT, they need to EARN the rights to be an MCT.

Training is absolutely not certification. In fact, as a part of our accreditation, one of the rules is that those who develop certifications cannot be involved in creating training materials.

Read the MCT Program Guide AND the MCT Program Agreement. Then, follow the 4 basic steps to becoming and MCT
On top of all the 'question writing' that you're proposing, how do you propose that Microsoft handle this community? You are aware that Microsoft currently audits the MCT Community throughout the year to make sure they are in compliance with program requirements, correct? Talk about throwing a wrench in those gears.

The MCT audit process does not ensure that MCTs are up to speed on the latest technologies. MCTs out there, feel free to dispute this statement, but I talked to Ken Rosen about it today just to make sure that I wasn’t running on faulty assumptions. As such, I don’t see how this would throw a wrench in any MCT program gears.

Summary
MCTs (SMEs/Experts) = $$
Community = $$$
This community you're proposing would not be worth the time, the money, or the effort that you're proposing from Microsoft.