Rigor vs. Ambition

I love working with former scientists, but I also kind of hate it.

I was reviewing an experiment recently. The new page was performing better, so I suggested we end the A/B test.

"Shouldn't we wait until it's statistically significant?" she asked.

“We’re not trying to publish a paper,” I replied. “We’re looking for things that are big. If an experiment takes weeks to reach significance, it’s not big. Don’t waste time chasing unnecessary precision – go look for something bigger.”

“Fair enough,” she responded, “but something doesn't make sense... We thought this would make a big difference, and it didn't. Why not?”

(Now I’m back to loving scientists!)

It's a brilliant question – when you get an unexpected result, revisit your prior assumptions.

In my experience, small results usually stem from three things:

• They don’t see it. We make a small change halfway down a page, or on a page that gets very little traffic, or in an email nobody opens.

• They don’t understand. The new copy / image / video was too long / complicated / clever – confusing, not persuasive.

• They don’t care. We talked about something that mattered to us, not our customers, like features rather than outcomes.

Small result –> big impact

We all drink too much coffee and fall in love with our ideas, expect they’ll be huge, and then they aren't. The natural response is to push those failures aside, a little embarrassed, and chase the next shiny thing. But if we can take a few minutes to reflect on why we overestimated our idea in the first place, we get better at dreaming up things that actually can be huge.

And that, my friend, can turn a small failure into a huge win.

I hope this helps.