Tarkblog

I’ve been doing a lot of testing lately (for customers), and I’ve noticed a trend that is quite surprising.

Well, surprising probably isn’t the right term. Lately, I’m not really surprised by much. Ridiculous is probably a better way of putting it.

See, the common trend I’ve found anymore is that if the test doesn’t generate the desired results, then the test is changed until it delivers the results the tester wanted. The proper, albiet more difficult, thing to do is to find out why the object being tested isn’t generating the proper data and fix the problem. Coercing the test to generate the desired results isn’t going to help anyone in the long term.

But that seems to be the norm these days.

A common test we run is to start an engine cold (say, -20 degrees F) and look at the hydrocarbon levels. High HC levels lead to visible smoke, which in a diesel is basically really annoying if you’re warming your pickup truck in your drive and your whole yard fills with smelly exhaust fumes.

One key aspect to this test is to start the engine, let it idle for a few minutes, then immediately jump the throttle to a higher percentage (torque peak speed), wait a few minutes, then jump the engine to high idle, wait a few minutes, then idle it down again. What’s key here are the jumps - that is, after the engine has had a chance to sit at a certain speed for a while, the hydrocarbon levels will fall off because things have warmed up. The step change in throttle allows you to see the dramatic effect of the hydrocarbon output.

This particular test sequence is written in a test specification; that is, someone deemed it important enough to mandate how this particular test is run.

Yet, one engine we’ve had lately doesn’t respond well to this test. That is, in cold conditions, the engine responds poorly to this step change in throttle. They’re running “experimental” software in the engine, so things like this are bound to happen.

But instead of fixing the problem in the engine, they want us to change the way we run the test. Specifically, they want us to slowly ramp the engine speed up instead of making the step change. Why? Because that way, the engine can actually run the test. Thus, running a modified test is more important than running the correct test, because getting some (wrong) data is better than getting no data.

What a joke. What will end up happening is that the customer’s management will be happy in the short term because they produced results, but in the long term when the problems still persist because they didn’t run the test that generates the proper results, they’ll end up having to rerun the test.

And the bigger threat to this is that, 6 months down the road, it will be asked “Well, why didn’t the people performing the test run the actual test? We have a spec and they didn’t follow it. This is their fault.”. And AEI will look bad because our test wasn’t run to spec.

I’ve got more examples, but due to problems with keeping certain things “secret” I can’t talk more about them. But let’s just say that one customer we deal with has two groups, a “vehicle” group and an “emissions” group. And the emissions group doesn’t care about making data match the “vehicle” while the “vehicle” group doesn’t care about the “emissions”. So we’re fighting a losing battle between the two. Each wants us to change the test setup to match how they run their own particular tests, which is fine, except it causes us not to be able to match the other one. Who knew.

Finally, I can’t say enough about professors who get angry when the whole class of students do poorly on a test, so for the next test they make the questions “easier” (and by easier, I mean they put 5 questions instead of 9 on the test). They change the testing method, instead of fixing the problem that caused the bad results - their own poor methods of teaching.

Tarkers. All of them.

This entry was posted on Wednesday, December 1st, 2004 at 10:31 am and is filed under Random Thoughts. You can follow any responses to this entry through the RSS 2.0 feed. Both comments and pings are currently closed.