Pilots Are Not Experiments

July 16, 2013  

In previous blog entries, I discussed why you need to make your innovation a form of experimentation.

Doing this redefines the meaning of failure. Experiments (and as a result, innovation) only fails when you don’t have enough data to support or refute your hypothesis.

Disproving your hypothesis is success because you can kill the project (or specific solution) and stop making more investments in a losing concept.

All too often, companies confuse experiments with pilots. They are not the same thing.

Typically, a pilot is not designed to test a hypothesis but rather to test the scalability of a solution. You want to make sure everything works as expected before rolling out to new divisions, geographies, or departments.

Pilots have a reasonable likelihood of success. You may identify some modifications and tweaks to your original solution. But the odds are that you will move forward with the plan.

Experiments, on the other hand, (should) have a high likelihood of disproving your original hypothesis.

If your experiments frequently support your hypotheses, you need to consider the possibility that you are doing something wrong.

  1. Maybe your experiments are really pilots. Experimentation typically starts earlier in the development cycle.
  2. Maybe your team is suffering from “confirmation bias;” you subconsciously look only for evidence that supports your hypotheses.
  3. Maybe your experiment is not well-formulated and is resulting in false positives.

In the world of innovation, there is no such thing as a “perfect” experiment. No amount of analysis or testing can predict what will happen under real world circumstances.

But when done right, experiments can help you improve your innovation ROI by identifying the “bad” ideas early in the process. Therefore, embrace the fact that good experiments disprove hypotheses more often than not.

Leave a Reply