Debug Like a Plumber: Probe Hidden Bugs Indirectly
Production bugs hide like underground leaks—don't inspect directly; inject 'tracer gas' probes that force issues to surface, as a leak specialist did in 20 minutes without digging.
Force Hidden Problems to Reveal Themselves
A leak detection specialist fixed an underground pipe leak near a driveway without digging: he connected a compressor to inject tracer gas into the pipe, then walked the surface with a handheld detector. The gas escaped only through the break, bubbling up through soil to the detector. In 20 minutes, he pinpointed the spot: 'Dig here.' He succeeded by assuming the problem was invisible—buried under concrete—so his method didn't try direct visibility. Instead, it created an inescapable signal from the issue itself.
Engineers' Faulty Assumption Slows Debugging
Software teams facing production bugs assume visibility: code is readable, dashboards show metrics, logs capture events. They read code, stare at dashboards, add logging (often more logging). This direct inspection fails because production issues are 'underground'—intermittent, environment-specific, or emergent—making them hard to spot even when staring.
Adopt the Tracer Gas Mindset for Faster Fixes
Shift to the plumber's assumption: production bugs can't be seen directly, so inject probes that the problem can't hide from. Examples include targeted canary deployments, synthetic traffic simulating user paths, or chaos experiments flipping switches to surface weaknesses. These methods guarantee the bug announces itself, cutting debug time from hours/days to minutes, just as tracer gas did.