I live in the country, and we’re not on a municipal water system. Instead, we have a 2500 gallon cistern, and I fill it with water that I haul myself in a 3/4 ton Chevy.
What the hell does that have to do with testing?
The truck is only ten years old. Practically a baby by my terms. That’s important, because around the end of the 20th century car manufacturers started to replace kilometer after kilometer of tangled wiring harnesses with microchips. There are sensors all over the place, and they all feed into the OBD. (Onboard Diagnostic System.) It’s a computer system and diagnostic data, for all that sensor data.
The year the Chevy was made they installed an unreliable fuel sensor. The things die like crazy, and since they’re sitting on top of a gas tank, it’s an obnoxious part to fix. Mine doesn’t work, and I haven’t bothered to fix it. This means that the needle fluctuates randomly between full, half full and empty. The dash is randomly lit up by the fuel light. Because the fuel sensor is broken, the fuel gauge feeds me meaningless garbage. I know this, because I have a sense of how fuel systems behave — clearly I’m not parking the truck with more gas than I started out with. Yet as far as the truck’s diagnostic systems are concerned, nothing is out of the ordinary.
This brings me to the next problem. A while ago, I was on my way to pick up water, when I noticed the orange glow of a check engine light. With that new glow beside the fuel warning, the dash was lit up like a christmas tree. Back at home with the truck parked and emptying into the cistern, I had time to go into the house and grab the OBD box. I knew that I could plug the OBD box into the truck and get access to all that diagnostic information, in the form of a bunch of codes that would give me more specific information than the very vague flashing of a check engine light.
Fail states are of course unreliable. The diagnostic system had no idea that the fuel sensor was broken, and as far as the fuel sensor was concerned the level in the tank was leaping wildly between full, empty, and half. The truck wasn’t capable of intuiting that something might be wrong from a bunch of weird sensor results. Failed systems can have an extremely difficult time self-diagnosing. In this case, you might be able to solve the problem by adding more sensors: measure the fuel poured into the tank, versus the fuel pumped out. Check the pressure in the tank. Check the sensor readings against average mileage. The problem is that by doing so you’re making the system even more complicated, and introducing more potential for error. Sometimes, simple is the most reliable, and when you’re talking about warning lights, reliable is important.
So I was sitting with the OBD reader and a book to translate the codes. The code was for the Emissions System. Aha! Now we’re talking. That narrowed things down a lot. Even more so, it’s a light a lot of people get. It’s become notorious, after too many people limped vehicles into the garage only to discover (120$ later) that their fuel cap was loose. The system was checking the pressure of fuel and vapor in the system, and a loose cap showed just the same way any other pressure loss would. This has been a major issue, because frequent false alarms lead to people ignoring warning systems, and in this case, they can be very expensive to people.
I wasn’t going to a mechanic, so I started to consider all of the things that could go wrong with an emission system and my truck. What is the sensor trying to tell me? The water tank emptied itself, the suspension lifted, and I went back to uncouple the hoses and store the equipment. I was just finishing up when I remembered what should have been obvious an hour prior. I had just prepared to climb under the truck and take a look at my exhaust system, but as I mentioned earlier, the truck, has seen better days. In fact, one of its many missing pieces is the muffler and the entire tail pipe. Exhaust pumps out beneath the box and finds its way off into the atmosphere from there. The emissions sensor was complaining because the entire exhaust system was gone. Hell, the sensor itself was probably gone. Of course the truck was complaining about the emissions system.
So there’s a point to all of this: Generic error messages are very good, they warn people quickly about issues. Warning systems are even better if there’s a set of logs and diagnostic data that’s accessible beneath the user level error message. Designers can build a very user friendly system, with very specific information beneath it for troubleshooting. But none of that is foolproof, and none of it replaces people. The best system in the world still won’t be enough, because machines in error states are very, very bad at self diagnosing. The self-diagnostic is either going to be very complicated and thorough, or very simple and reliable. Compromises have to be made on that scale, and regularly are.
What does this have to do with testing?
It’s a parable, right? Or an allegory. Or something literary. I’m testing a truck. The engine light comes on. Is it going to be enough to file a bug report that says, “Driving truck, engine light came on?” Clearly not. How about, “Driving truck to get water, engine light came on for emissions control error”? Still not enough. What could I have told them that was helpful? Let’s say I didn’t know that my muffler was missing. Maybe I could file a report that says, “Driving truck, when emissions control light came on. Exhaust has been bubbling up from under truck. Truck has been very noisy for some time. All other gauges look fine, except for fuel which shows empty, then full, then empty again.” Your mechanic still isn’t going to be able to tell you what’s wrong over the phone, but I’ll bet he’ll be starting to formulate a really good guess.
In the end, it’s always going to take investigative work to track down the source of problems. Even with all the tools in the world, computers and diagnostic systems are going to fall just shy. So give your mechanic all the information that they need. Don’t rely on self diagnosed symptoms. Your doctor won’t take phone calls, and your mechanic can’t give you an estimate based on a single diagnostic code. Your developer probably can’t either.