top of page
  • Writer's pictureMatthew Lerner

Failed Experiment

Most growth experiments should fail. (If not, test bolder stuff). But, when an experiment fails, how do you know if you’re moving forward? Or wasting precious time? You can feel it.


Do you feel smarter? Does it inform your next experiment?

- Can you rule out an option?

- Did you find some qualitative insight?

- Did some other metric move? In an unexpected way?

- Did you learn something about your tech or process?


Ultimately you need to refine your understanding of the prospects' conception of the problem space, their expectations, assumptions, doubts and behaviours.


You absolutely should get smarter after each experiment. If not, you’re going sideways.


How to Fix This?

If your experiments are not making you smarter, fix that with two simple process changes:

  1. Better up-front experiment thinking: Get very clear on your (specific written) hypothesis and success metric. Ask people to write predictions. Isolate one variable (e.g. message or channel) and/or test bold things that challenge your most important beliefs & assumptions about your customers.

  2. More post-experiment thinking: People tend to overthink what might go wrong and underthink what actually did go wrong. For each failed experiment, force yourselves to spend a full 20 minutes looking through the results and talking through why the results differed from your predictions. List out as many ideas as possible.

It’s hard to know if you’re making progress when the numbers aren't going up. But it’s easy to know if you’re learning. And if you're learning, you're making progress.

124 views0 comments

Recent Posts

See All

Science vs. TED

If you suddenly needed life-saving medical treatment, which one would you choose: A drug whose safety and efficacy were validated in a randomized controlled trial One that was featured in a really ins

The 6 dysfunctions of OKRs

Unpopular opinion: OKRs don't cause problems, they surface them. If you have a clear strategy, and everyone understands the company's goals and their role, OKRs should write themselves. If not...focus

Comments


bottom of page