Metric definitions: why three people report three numbers
Three people report three different numbers for the same metric because they built three different formulas for the same metric.
One person reports sign up rate at 15%. Another says 20%. Another says 5%. You do not know what it is, and all three of them are right. They have just measured it differently.
Every metric is a formula: something you count, divided by something else, over some window of time. The problem is that for RetentionThe share of users who come back. Bounded retention counts people who returned on exactly that day; unbounded counts that day or any day after, and the two give very different numbers.Glossary or sign up rate, there are ten different ways of building that formula.
Take sign up rate. You could take new users who come to your website and count how many of them sign up. Or you could take every visitor on your marketing website, including people who have been before, and count how many sign up.
The time frame moves too. Within one session. Within one day. Within seven days. And there are more choices underneath that, each one giving you a different sign up rate.
When do different metric definitions become a problem?
It shows up once you have hired someone to own analytics and they have started building dashboards. Two people pull the same metric and get different numbers, and nobody has confidence in anything after that. That is the point to fix it. It is far cheaper then than after a year of reports have been built on top of both definitions.
Often the people reading these numbers never check with you, the founder, the growth lead, or the product lead. Someone looks at the 5% and says this is really bad. Someone else looks at the 20% and says this is really good. Both of them make decisions on it, and the number you actually agreed on might be 15%.
So you need someone, or a team, who makes sure metric definitions are the same across teams. Sign up rate is calculated exactly this way, and nobody measures it any other way.
How should you write down a metric definition?
The fix is someone who defines these metrics, and one place where the definition lives. Maybe that is one dashboard everyone uses for that metric. Maybe it is saved definitions in your data model (the layer that turns raw events into clean tables, covered in "Data modelling"), holding exactly how that metric is calculated. dbt's Semantic Layer documentation makes the same case: when a metric definition changes in one place, it is refreshed everywhere it is used.
A definition written down is four lines, not a document.
Sign-up rate
- Counted
- Users who complete account creation
- Out of
- New users who land on any page of the website
- Window
- Within one day of first landing
- Excludes
- Internal traffic and known bots
Sign up rate
Counted: users who complete account creation
Out of: new users who land on any page of the website
Window: within one day of first landing
Excludes: internal traffic and known bots
Every one of those four lines is a place where two people can quietly disagree with each other. Once the lines are written down, they cannot.
Go back to the three numbers with that in mind. The 5% counted every visitor on the marketing site and gave them one session to sign up. The 15% counted new users to the website and gave them a day. The 20% counted those same new users and gave them seven days. Nobody did anything wrong. Only one of the three was the company's number.
Is it a definition problem or a tracking problem?
There is a different problem that looks exactly like this one from the outside, and it is worth knowing which of the two you have.
I spent weeks with a client once trying to improve their new user to signup conversion. The dashboard said 5%. We changed the copy, redesigned the flow, ran experiments. Nothing moved. At the end, we realized that the signup event was not firing correctly. Once we fixed it, the real number was 12%.
No amount of agreeing on definitions would have caught that. A definition problem hands you several numbers that are all correct. A tracking problem hands you one number that is simply wrong. They feel identical in the meeting where somebody says the data looks off, and they need completely different fixes.
So ask one question. Are people bringing different numbers for the same metric? That is a definition problem. Is everyone looking at the same number, and nothing you do moves it? Suspect the tracking. "Event verification" covers how to check that events fire correctly.
| What you see | What you have | The fix |
|---|---|---|
| People bring different numbers for the same metric | A definition problem: several numbers that are all correct | Write the definition down in four lines and keep it in one place |
| Everyone looks at the same number, and nothing you do moves it | A tracking problem: one number that is simply wrong | Check that the events fire correctly |
Common questions
Why do two people get different numbers for the same metric?
Because they built different formulas. Sign-up rate could be new visitors who signed up, or every visitor including returning ones, within a session, a day or seven days. Each choice gives a different, defensible number.
How should you document a metric definition?
In four lines: what is counted, what it is counted out of, the time window, and what is excluded. That is short enough that people actually write it and specific enough that two people get the same number.
What is the difference between a definition problem and a tracking problem?
A definition problem hands you several numbers for the same metric that are all correct. A tracking problem hands you one number that is simply wrong, for example because the event is not firing correctly.
Where should metric definitions live?
In one place everyone uses: one dashboard for that metric, or saved definitions in your data model that hold exactly how the metric is calculated.
When should you standardise metric definitions?
Once you have hired someone to own analytics and two people pull the same metric and get different numbers. It is far cheaper to fix then than after a year of reports have been built on top of both definitions.