I was wrong about being wrong

My friend Erin used to run experiments at Booking.com. She said they tested 1,000 ideas per week, and 91% failed.

My first reaction: "Wow, if you test aggressively, you'll be wrong a lot." I completely missed her point.

On a bike ride last week, it hit me: Testing didn't cause them to be wrong, it caused them to discover they were wrong. Those ideas would’ve been bad whether they’d tested them or not.

But Booking.com isn't uniquely stupid. So if 91% of their ideas are wrong, mine probably are too. And so are yours.

I stopped for a snack and messaged Erin, "So what's the point of all that testing if the control variant usually wins?"

"That’s exactly the point,”she said. 

“Only 10% of our upside came from finding winners. 90% came from preventing bad ideas from ever seeing the light of day."

Without testing, they would have shipped hundreds of bad ideas. With testing, most of them died in the lab.

I used to think growth was about finding a big lever. Turns out it’s mostly "first, do no harm."

What would you do differently if you knew 9 out of 10 of your ideas were wrong?

Because they probably are.

Simple next steps

Instead of executing your big ideas, test them quickly with a minimum viable test:

1. Identify your riskiest assumption by asking: “If this idea fails, what’s the likely cause?”

2. Find a way to test only that assumption without testing the whole idea – here’s 20 example experiment designs.

3. Ask everyone to predict what they think will happen – document these to eliminate hindsight bias

4. Run the experiment, discuss any surprises in the results

Want to build a culture where bad ideas die in the lab? We run private workshops to help teams experiment faster and bolder.