Multiple Choice Questions Are Like....Drugs?
I was really excited about offering up the community based development concept for public debate and scrutiny - and I am still very glad to have done it. I did however, get quite a kick out of this comment on Trika's blog.
Now, firstly, I want to mention that I have been in a few conversations with the folks over at CertGuard and they are great people who are trying boost the value of certification for all of us. Some of the concerns raised I agree with and have taken into consideration - others I don't - and the pilot will hopefully provide additional clarity. However, the commenter above did bring up one thing that I have heard many times since I joined the organization. That is, if you want to get rid of brain dumps, change the format of the exams.
Typically, when people make comments like this, they are referring to performance-based (PBT), or emulation-based testing. By that, I'm describing a test format whereby in order to pass the exam, you actually open a tool like Visual Studio and perform the task assigned.
Fundamentally, I have a significant problem believing that changing the way in which we ask questions is some kind of silver bullet to the cheating problem. Here are a couple of thoughts.
- PBT questions are still graded by a computer, which means that the objectives are deterministic. Therefore, why is it so hard to see brain dumps on these types of questions?
- Because PBT questions are graded by a computer, both the expected results and constraints must be entered into the scoring program. For example, let's say that I expect the candidate to write an addition function that returns 5 when I pass it 2 and 3. What is to say that you as the savvy candidate don't simply hard code 5 as the return value? Now granted - this is a huge oversimplification - but the burden is on the PBT question author to think of all the things that a candidate shouldn't do as well as the things he/she should do in completing the question requirements. This could potentially unlock a whole new type of cheating (the type that you see in the gaming world).
- The cost (initially, at least) of developing PBT questions is high. Therefore, the number of questions may be fewer. Even if a brain-dumped PBT question is more complex, if there are fewer of them, how much more difficult have we really made it to cheat?
- Standardized tests like the SAT, LSAT, and GRE (in the US) have relatively low(er) problems with cheating - and they are all multiple choice tests.
In my opinion, the way to curb cheating takes on a couple of forms. Firstly, you can go after the people that are cheating and helping others to cheat. We are pursuing that route as evidenced with TestKing settlement. Secondly, you can create question types that are harder to steal - PBT takes a step in that direction. Thirdly, you can significantly increase the amount of information required both to steal and to memorize in order to obtain a "paper" cert (perhaps even do adaptive testing) - community-based development takes a step in that direction.
Now, don't think that by this, I mean to say that we shouldn't do PBT. To the contrary, we absolutely should (and probably should have a long time ago) have this tool in our toolbox. My point is that it is just that - a tool - albeit a really bright, shiny one. We need to approach the problem of cheating from multiple dimensions - and both PBT and community-based development are a couple of those dimensions.
I am currently the Editor-in-Chief for MSDN Magazine. I joined Microsoft in 2006 as a product planner with the certification team at Microsoft Learning. Prior to that, I spent my career as a developer and later as an architect. My main technology passions include pretty much anything on language theory, agile development, and service-oriented architecture.
