How to make good decisions with bad data
- Matthew Lerner
- 20 hours ago
- 2 min read
Early in my career, we used this horrible thing called WebTrends that measured nonsense like "hits" and "time spent on page."
Eventually, I realized those reports were a Rorschach test – I could see anything I wanted in them. So I ditched them and started with business outcomes, working backwards.
Why am I telling you this? Because marketing analytics feel the same way today. AI overviews reduce search clicks. Social platforms suppress links. Apple, GDPR and LLMs all block tracking. The dashboards still light up, but the signal is mostly noise.
The fix is almost embarrassingly simple.
Each week, I used to pull a report of daily conversions and look for weird spikes or troughs. Then I’d research what happened:
“Does anyone know why we got so many customers on Tuesday of last week?”
“Our SEO traffic just tripled, but our sales didn’t budge. Are we ranking for a new term that’s sending us garbage traffic?"
I'd find things that worked (a press mention brought 40 customers) and things that didn't (an affiliate sent traffic that never converted). I’d drop the affiliate and befriend the journalist. Repeat.
Good news, old-school analytics still work:
Pull a report showing two metrics, daily:
Conversions – Signups, purchases, or paid conversions
Conversion Rate – The % of visitors who converted
Research unusual spikes and dips:
Total conversions spike – What worked?
Total conversions dip – What’s hurting us?
Conversion rate dip (traffic up, conversions flat) – Something's sending junk traffic. Stop spending on it.
Conversion rate spike (conversions up, traffic flat) – High-quality traffic source. Do more of that.
Double down on things that work and fix or drop the things that don’t
Forget fixing broken tracking. Watch outcomes and work backwards. It’s imprecise, but I’d rather have vague signal than detailed noise.
Is it dangerous to trust correlations?
Only a complete moron would base important decisions on correlations, right? Well… yes – if you stop there. Correlations show us where to dig. We assess causation by acting: Get another article, drop the affiliate, see what happens.
Credit to Amanda Natividad for this insight; she calls it a “correlation dashboard.” You can watch my conversation with Amanda here.