[lpi-discuss] paper vs. computer based exams

[Alan McKinnon]

On Friday 05 September 2008 00:08:12 Anselm Lingnau wrote:
> doron wrote:
> > my question, is why we don't use the method of tasks based exam ?
>
> The party line is that, in contrast to tasks-based exams, multiple-choice
> exams actually have science (as in, reviewed papers) behind them that says
> that performance in an appropriately constructed multiple-choice exam
> correlates with knowledge of the subject matter tested. As far as
> tasks-based exams are concerned, there is a widespread warm fuzzy feeling
> suggesting that someone who can do contrisved tasks of the kind suitable for
> an exam situation will also be able to do similar tasks in the real world
> but that seems to be all there is at the moment.

As a long time supported of LPI and (until very recently) a practising RHCI, I 
can attest to the RHCE exam having very little to do with real-world 
situations. The exam is about as for removed from reality as the LPI exams 
are but in a different direction. This is completely unavoidable as like 
Anselm says, the exam room is a very very different thing from real-life.

Compare it to a car drivers license test. The road test is somewhat realistic, 
but the majority of people who've passed it will tell you that they drive in 
a certain fashion for the test in order to be able to pass the test. When 
they leave the test centre with license intact, they drive a very different 
way.

The RHCE exam is a lot like that. It measures something (it is far from 
worthless!) but it does not actually measure what candidates seem to think it 
measures. And it definitely does not measure what Red Hat's marketing 
department claims! (You can't fault them for that - they have a job to do and 
they do it well).g

I get asked a lot to opine about how good the RHCE exam is. I've figured out 
this answer:

"It tests if you can run a Red Hat system the way Red Hat designed it to be 
run. It tests if you can do routine ordinary everyday things in the 
prescribed manner."

Unfortunately it suffers from a very high false negative rate, and it's not 
just me - all 6 RHCIs in this country report similar numbers. And the irony 
is that almost universally the people who do pass are not our students, they 
are people who have been doing sysadmin stuff for years and can answer the 
test questions in their sleep. So one has to ask the question:

"Is a task-based exam really such a good idea when 80% of the people who, in 
the estimation of an experienced teacher and examiner, actually should pass 
the thing are failing?"

Lest anyone thinks I have a poor opinion of the RHCE exam, I don't. As exams 
go, this is an especially good one and Red Hat put a lot of work into making 
sure it is relevant and meaningful. Luckily, false positives (people who 
passed but didn't deserve to) don't appear to happen in my experience.



-- 
alan dot mckinnon at gmail dot com