[lpi-discuss] Re: [lpi-staff] RPM and Debian

[Alan McKinnon]

On Sunday, 13 March 2005 19:03, Mark Miller wrote:
> On Sat, 2005-03-12 at 20:46 -0800, ross e. brunson wrote:
> > Is it once again time for me to bring up that old chestnut of a
> > topic called "We should have a practicum-based exam option"?  Not
> > as the primary option for exams, but perhaps we could offer the
> > ability to "CLEP" out of the different levels at LW's or an
> > annual event?
>
> No.
>
> While there are domains that a hands on test could measure better
> than a written test ours really isn't one of them. At the
> Association of Test Publishers convention some groups were pushing
> hands on heavily. They had to admit to me however that a well
> written multiple choice test can indeed assess the knowledge of
> people even better than hands on and even better than essay
> questions. Here is why:
>
> 1. A performance test is limited to a small number of examples
> within a given timeframe.
>
> 2. Those examples are necessarily limited in scope due to time.
> Like a code example in a magazine they must give a relatively
> trivial example due to time and space limitations.
>
> 3. It is resource intensive. Due to the need for computers,
> networks etc there is a limit on how many test centers can support
> this. This would limit the ability to reach out beyond major
> concentrations of candidates. Exams in poor areas would be nearly
> impossible.
>
> Now with that said I must tell you that I am by nature a
> performance based kind of guy. A certification isn't a performance
> based thing. It is a tool to decide if a job candidate has a
> certain level of knowledge. The ability to apply that knowledge
> must be assessed by judging experience and attitudes at an
> interview. Even then it comes down to a "feel" for the hiring
> person. Performance based measures are the only way to go for on
> the job training. Being able to prove to a domain expert that you
> can apply your skills directly on the job is priceless. That
> doesn't always translate to being able to apply them universally.
> To my knowledge no test method can guarantee that a person can
> apply a given skill in any domain outside of the one tested.
>
> In my mind, for IT certification, a hands on test is of limited
> value and that value is mainly to sell expensive training.

I must largely agree with Mark here. But, a large chunk of my 
agreement is based on gut feel and experience with technical people.

An example from a dis-related field: like Mark, I used to be an 
electronics tech. I ran the biggest and highest producing workshop of 
it's kind in the city (consumer grade electronics - TV, VCR, audio, 
etc). Every job applicant got tested the same way - I'd ask for a 
random number x between 1 and 7. Together we'd go to the in-rack, 
find job number x, and the applicant had to fix it while I kept on 
eye on him. Here's what I would check for:

- logical thinking sequence to determine the fault
- did he ask for the proper test equipment (oscilloscope)?
- did he ask for the schematic diagram?
- quality of soldering work
- safety measures - were they applied?

I didn't give two hoots about whether the fault got fixed or not 
(except if he happened to get a real easy job). If you don't know 
Sony TVs and you happen to get one, your chances of fixing it are 
slim indeed. What I did look for is general attitude and skill, the 
candidates approach to the problem, and does he know how to be a 
*technician*. A technician by definition is someone who knows 
how_to_find_out how to fix an item, not someone who can fix it 
(that's too narrow a definition for industry-wide tests).

So that's how a domain expert does it. Here are the flaws:

The system completely depended on my ability to spot good and bad 
technique.
It relied completely on my personality type and what I needed and 
wanted from a technician.
It was only valid if I was testing someone I would employ. Someone 
that I deemed unsuitable could easily fit in elsewhere, and perform 
well there.
If I wasn't around the applicant couldn't be tested.
It was impossible to train someone to replace me in this function when 
I left the company.
No two applicants got the same test, which introduces inconsistency, 
which brings us back to me being the only valid tester for the life 
of the method.
No quantifiable measureables.

I had the option to change the test method to something that would 
test basic ability, but chose not to as I was more interested in 
finding people that would fit in with the existing group.

To come back to practical versus paper IT certification tests, a 
practical test can be devised that tests basic ability, like Mark 
says. It's no better or worse than a paper test, but costs a whole 
lot more.

We shouldn't forget that we would like to test for actual ability, but 
that can't be done - ability is an intangible. We can only test for 
the side effects, such as knowledge about a subject that comes from 
real experience, and correlate that to statistical data on success of 
candidates in the workplace. All an IT cert should give is a 
reasonably high probability that the candidate knows his stuff.

It's easy to screw this up. Two prime examples: 

1. To be a certified technician in South Africa, you sit a trade test, 
which is a practical exam. I have yet to see a freshly graduated 
technician who could perform on the job, despite the practical test. 
Reason: the test is out of date. It tests if you can fix a limited 
range of TVs that date back to 1985.

2. A fellow student on a technical course some years ago was part of a 
University Engineering School study to correlate success at school 
with success on the engineering course. The correlation was 0.19 - to 
an engineer that means the school test system is completely and 
entirely worthless.

My conclusions: LPI is going about this the right way with paper 
tests. A large initial investment of effort gives us a test method 
that is cheap to deliver and statistically very consistent.

That's the theory, what I don't have is numbers to back it up. I do 
have empirical evidence from my own observations. Have any studies 
been done to correlate success at certification exams with success in 
the market place?

-- 
Alan McKinnon
Head Trainer - AfriBiz
alan at afribiz.co.za
+27 82 337 1935
+27 12 331 8102 (W)
+27 12 349 9277 (F)