Experimental failure is data

The experiment failed.

Three days of preparation. A prototype. Measurements.

And the expected result isn’t there.

The temptation is immediate: fix something and start again.

Yet, before touching the prototype, one question is worth asking: what has this failure just taught us?

An experiment doesn’t need to succeed to be useful #

In R&D, the goal of an experiment isn’t to confirm that we were right. It’s to reduce an uncertainty.

A hypothesis can be confirmed, refuted, or simply left undetermined because the experiment wasn’t discriminating enough. These three outcomes are different. And confusing them loses valuable information.

If an architecture assumed to be viable doesn’t produce the expected phenomenon under certain conditions, we may just have eliminated part of the solution space.

It isn’t a solution yet. But it’s no longer the same problem.

Document what doesn’t work #

The value of an experimental failure then depends heavily on what you keep.

• What hypothesis were we testing?
• What did we actually change?
• What quantities did we measure?
• Under what conditions?
• What result would have confirmed or refuted the hypothesis?

Without that record, the failure simply becomes “we tried, it didn’t work.” A few months later, someone sometimes runs exactly the same experiment.

With a usable record, the result becomes project data instead. It lets you change the next hypothesis.

Fail more precisely #

R&D rarely advances in a straight line.

• You formulate a hypothesis.
• You build the minimal experiment that confronts it with reality.
• You measure.
• Then you gradually reduce the uncertainty.

Progress doesn’t necessarily mean having fewer and fewer failures. It means getting failures that are more and more informative.

A prototype that fails without your knowing why brings little. An experiment that lets you precisely rule out a hypothesis can, on the contrary, save weeks.

That’s an important difference between building to demonstrate and experimenting to understand.

But this logic has a trap. In trying to make every experiment clean, robust and compliant with best practice, you can end up eliminating exactly the most interesting ideas before you’ve even tested them.

Perspective 05 — next Monday: Optimising too early can kill invention.