Correlation is not causation: how to check before acting
A finding that users who do X are more likely to convert may be pure correlation. Those users might simply be your most motivated ones, and would have converted anyway. Your existing data cannot tell you which one you are looking at, so check: look at the users who converted without doing X, run a small experiment on the people who have not converted, and talk to users.
You run a report and find that people who do action X are more likely to upgrade. The obvious insight follows. Let's get more people to do action X.
That could be pure CorrelationTwo things showing up together. It is where an insight starts, not where it ends: the users who do the thing may simply be your most motivated ones.Glossary. Nothing in the finding says X causes the upgrade.
Correlation means two things show up together. CausationOne thing making another happen. Your existing data cannot show it; an experiment, a look at the users who converted without the behaviour, or a conversation with them can.Glossary means one of them makes the other happen. The users who do X might just be your most motivated users, and they would have upgraded anyway.
Your existing data cannot tell you which one you are looking at. You have to go and check.
Take a quick example. You have a platform where you generate courses using AI and then publish them. The insight you found is that people who publish are 12 times more likely to buy your paid plan.
Here is what you need to do.
What did the users who converted without the action do instead?
Work out what their behavior looked like, and what the main thing they did was.
Say you find that these buyers did a lot of editing, and people who edit are also far more likely to purchase. Now you have ruled one thing out. It is not just publishing. There is a set of behaviors making people buy a plan, and publishing is one of them.
If nothing else stands out and almost every buyer published, publishing stays a strong suspect. Move to the experiment below to test it.
How do you test whether the action causes conversion?
Run a small experiment on the people who have not purchased. Take that group and split it randomly in two. Push one half to publish, and leave the other half alone.
You need a way to reach one half and not the other, you need the split to actually hold, and you need enough non-buyers for the difference to mean anything.
Then compare the two groups. Does the paid conversion rate go up in the half you pushed, stay flat, or go down?
| Paid conversion in the half you pushed | What it means |
|---|---|
| Up | Publishing is doing real work. Getting more people to publish should bring more upgrades. |
| Flat | It was correlation. People who publish were going to buy anyway, so pushing others to publish will not move revenue. |
| Down | The push itself is getting in the way. |
This is the fastest way to answer whether the correlation is causal. The random split is what makes the answer causal: Kohavi and colleagues' guide to controlled experiments explains that random assignment spreads confounding factors evenly across the groups, so the difference between them comes from the change you made.
Which users should you talk to about a correlation?
Whenever you land on an insight like this, talk to your users. There are three cohorts, or groups, worth talking to:
- people who published and upgraded
- people who did not publish and upgraded
- people who published and did not upgrade
Ask the first and second group why they upgraded. Ask the third group why they did not. That is what takes you down to the root cause.
Common questions
How do you know if a correlation is causation?
Three checks. Look at the users who converted without doing the action at all and see what they did instead. Run a small experiment prompting the action on users who haven't converted. And talk to users about why they did it.
Why is acting on correlation risky?
Because you may be pushing an action that marks motivated users rather than creating them. Driving everyone to do it then produces the action without the conversion, and you've spent a roadmap slot on nothing.
What is the difference between correlation and causation?
Correlation means two things show up together. Causation means one of them makes the other happen.
Which users should you talk to when you find a correlation?
Three groups: people who did the action and upgraded, people who did not do it and upgraded, and people who did it and did not upgrade. Ask the first two why they upgraded, and the third why they did not.