[lpi-discuss] Re: [lpi-staff] RPM and Debian
alan at afribiz.co.za
Sun Mar 13 12:55:09 EST 2005
On Sunday, 13 March 2005 19:03, Mark Miller wrote:
> On Sat, 2005-03-12 at 20:46 -0800, ross e. brunson wrote:
> > Is it once again time for me to bring up that old chestnut of a
> > topic called "We should have a practicum-based exam option"? Not
> > as the primary option for exams, but perhaps we could offer the
> > ability to "CLEP" out of the different levels at LW's or an
> > annual event?
> While there are domains that a hands on test could measure better
> than a written test ours really isn't one of them. At the
> Association of Test Publishers convention some groups were pushing
> hands on heavily. They had to admit to me however that a well
> written multiple choice test can indeed assess the knowledge of
> people even better than hands on and even better than essay
> questions. Here is why:
> 1. A performance test is limited to a small number of examples
> within a given timeframe.
> 2. Those examples are necessarily limited in scope due to time.
> Like a code example in a magazine they must give a relatively
> trivial example due to time and space limitations.
> 3. It is resource intensive. Due to the need for computers,
> networks etc there is a limit on how many test centers can support
> this. This would limit the ability to reach out beyond major
> concentrations of candidates. Exams in poor areas would be nearly
> Now with that said I must tell you that I am by nature a
> performance based kind of guy. A certification isn't a performance
> based thing. It is a tool to decide if a job candidate has a
> certain level of knowledge. The ability to apply that knowledge
> must be assessed by judging experience and attitudes at an
> interview. Even then it comes down to a "feel" for the hiring
> person. Performance based measures are the only way to go for on
> the job training. Being able to prove to a domain expert that you
> can apply your skills directly on the job is priceless. That
> doesn't always translate to being able to apply them universally.
> To my knowledge no test method can guarantee that a person can
> apply a given skill in any domain outside of the one tested.
> In my mind, for IT certification, a hands on test is of limited
> value and that value is mainly to sell expensive training.
I must largely agree with Mark here. But, a large chunk of my
agreement is based on gut feel and experience with technical people.
An example from a dis-related field: like Mark, I used to be an
electronics tech. I ran the biggest and highest producing workshop of
it's kind in the city (consumer grade electronics - TV, VCR, audio,
etc). Every job applicant got tested the same way - I'd ask for a
random number x between 1 and 7. Together we'd go to the in-rack,
find job number x, and the applicant had to fix it while I kept on
eye on him. Here's what I would check for:
- logical thinking sequence to determine the fault
- did he ask for the proper test equipment (oscilloscope)?
- did he ask for the schematic diagram?
- quality of soldering work
- safety measures - were they applied?
I didn't give two hoots about whether the fault got fixed or not
(except if he happened to get a real easy job). If you don't know
Sony TVs and you happen to get one, your chances of fixing it are
slim indeed. What I did look for is general attitude and skill, the
candidates approach to the problem, and does he know how to be a
*technician*. A technician by definition is someone who knows
how_to_find_out how to fix an item, not someone who can fix it
(that's too narrow a definition for industry-wide tests).
So that's how a domain expert does it. Here are the flaws:
The system completely depended on my ability to spot good and bad
It relied completely on my personality type and what I needed and
wanted from a technician.
It was only valid if I was testing someone I would employ. Someone
that I deemed unsuitable could easily fit in elsewhere, and perform
If I wasn't around the applicant couldn't be tested.
It was impossible to train someone to replace me in this function when
I left the company.
No two applicants got the same test, which introduces inconsistency,
which brings us back to me being the only valid tester for the life
of the method.
No quantifiable measureables.
I had the option to change the test method to something that would
test basic ability, but chose not to as I was more interested in
finding people that would fit in with the existing group.
To come back to practical versus paper IT certification tests, a
practical test can be devised that tests basic ability, like Mark
says. It's no better or worse than a paper test, but costs a whole
We shouldn't forget that we would like to test for actual ability, but
that can't be done - ability is an intangible. We can only test for
the side effects, such as knowledge about a subject that comes from
real experience, and correlate that to statistical data on success of
candidates in the workplace. All an IT cert should give is a
reasonably high probability that the candidate knows his stuff.
It's easy to screw this up. Two prime examples:
1. To be a certified technician in South Africa, you sit a trade test,
which is a practical exam. I have yet to see a freshly graduated
technician who could perform on the job, despite the practical test.
Reason: the test is out of date. It tests if you can fix a limited
range of TVs that date back to 1985.
2. A fellow student on a technical course some years ago was part of a
University Engineering School study to correlate success at school
with success on the engineering course. The correlation was 0.19 - to
an engineer that means the school test system is completely and
My conclusions: LPI is going about this the right way with paper
tests. A large initial investment of effort gives us a test method
that is cheap to deliver and statistically very consistent.
That's the theory, what I don't have is numbers to back it up. I do
have empirical evidence from my own observations. Have any studies
been done to correlate success at certification exams with success in
the market place?
Head Trainer - AfriBiz
alan at afribiz.co.za
+27 82 337 1935
+27 12 331 8102 (W)
+27 12 349 9277 (F)
More information about the lpi-discuss