The analytics playbook

Set up analytics you can trust & use

Playbook for Founders & PMs on what to track, how to check the data is right, and how to turn the numbers into decisions.

By Ansh Agrawal · 28 chapters · 6 parts

Part 01

Foundations

What product analytics is, how it differs from the other two things people call analytics, and the funnel model every later chapter sits on.

5 chapters · 18 min read

  1. 01What is product analytics?Product analytics is the analysis of what users do inside your product. The test is whether you can point at a row of data and name the user who did it. Revenue by country, sales by region and anything else that can't name a user is not product analytics.2 min read
  2. 02Product vs data vs business analyticsData analytics is the umbrella; product analytics and business analytics both sit under it. Product analytics studies how users behave inside the product. Business analytics answers questions about business objectives: revenue, markets and internal reporting.3 min read
  3. 03How to hire a product analyst: screen for the mindsetFounders screen product analysts for SQL and tools, then hire someone who pulls any number but can't say which number to ask for. The skill that separates a good product analyst is understanding how users move through the product well enough to turn a vague problem into one specific question. You can test for it in ten minutes by asking a candidate to map your product and say where users would drop off.5 min read
  4. 04Every product is a funnel: how to model yoursEvery product is a set of steps people go through in order, and so is every feature inside it. Count how many people reach each step, find the step where the biggest share drops off, and your problem shrinks from the whole product to one step.3 min read
  5. 05Product analytics by stage: what to build at your sizeSign-ups a month, not visitors, decides your analytics stage. Under 1,000 sign-ups a month, one product analytics tool on its free tier is enough. From 1,000 upward, track all the core actions in your product and use the data to find where and why people drop off.5 min read
Part 02

What to measure

Picking a north star you can explain, building the metric tree under it, and deciding who owns analytics at your stage.

2 chapters · 11 min read

  1. 06North star metric, metric tree and counter metricsPick one north star metric that matches what the business is trying to do right now. Break it into a metric tree so you can see which branch moved. Then watch a counter metric next to it, so a short-term bump does not read as a win.5 min read
  2. 07Analytics strategy: choosing metrics that are actionableA metric belongs in your analytics strategy only when it can change a decision. On its own, a metric usually tells you nothing. Set it next to two or three others, like sign-ups by channel or by country, and write down for each one what it counts and the question it helps you answer.6 min read
Part 03

Trustworthy data

The tracking plan, verifying what ships, keeping it accurate six months on, and the two things that quietly corrupt every per-user number you have.

6 chapters · 27 min read

  1. 08How to create a tracking plan before anyone writes codeA tracking plan is one sheet that contains every event you want to track and the properties that go on each event. Write it before anyone writes code, give it one owner who is not a developer, and that's your source of truth.6 min read
  2. 09Event verification: how to QA tracking before you trust itAfter your team ships event changes, somebody has to check that what's arriving in your analytics tool is what you asked for. Verification runs after implementation and before anyone uses the data, in three layers: what fired, whether what fired is right, and whether every surface was checked.5 min read
  3. 10Data accuracy as a system: keeping numbers rightVerifying an event tells you it worked that day. Staying accurate takes three habits: alerts on events and properties, new-event checks, database checks. The three run on their own, so you find out what your data is doing without going looking for it. Alerts catch problems as they happen, verification catches new events that were never checked, and the database check catches a gap that suddenly widens.3 min read
  4. 11Analytics data loss: why you're missing 15-20% of eventsIf your tracking runs entirely in the browser, ad blockers take at least 15 to 20%. Route through your own domain and send core events from your backend. If your users are technical, developers and designers and anyone who lives in a browser with an ad blocker running, the loss is worse than that.3 min read
  5. 12Identity stitching: why your user counts don't matchYour analytics tool counts one person as several until you call identify at login and signup. Get it wrong and every per-user number is wrong with it. Call identify with your own internal user ID, and only after the user has authenticated.4 min read
  6. 13MMP vs UTM tracking: knowing where users came fromIf your product is web only, UTM tags on every link are enough. The moment you have a mobile app, UTMs break at the install and you need an MMP. If you have both a website and an app, use the MMP for every link, so one system tells you where each user came from.6 min read
Part 04

Your data stack

What to add and when, how to make your tools agree on the same user, and the point where data modelling starts paying for itself.

2 chapters · 7 min read

  1. 14Unifying data: making your tools agree on one userUnifying data means lining up records from different tools so they describe the same user or campaign. You join at the least detailed level both sides share. User-level sources join on your internal user ID; ad spend only goes down to campaign and day.4 min read
  2. 15Data modelling: when it's worth doing, and what it fixesData modelling earns its place once you have a warehouse and people writing SQL against it. It cuts wasted compute and time spent rediscovering joins. You write the cleaning steps once, in dbt, in three layers: staging, intermediate and marts.3 min read
Part 05

Reading the data

Frameworks for taking apart any product problem, defining metrics so they point at something, and reading a number instead of just looking at it.

8 chapters · 28 min read

  1. 16A framework for any product problem: four questionsLow retention, low activation and low signup-to-paid conversion are the same problem from a data perspective. Take apart any of them and it boils down to: who converts, what they did that everyone else did not, what the two say together, and which of them actually caused the conversion.5 min read
  2. 17Metric definitions: why three people report three numbersThree people report three different numbers for the same metric because they built three different formulas for the same metric.3 min read
  3. 18Rate and mix analysis: why your conversion rate movedWhen a conversion rate moves, two different things can move it: the rates inside each segment, and how much volume each segment got. Build one counterfactual period and the two separate cleanly.4 min read
  4. 19Segment analysis: finding which segments move a metricFugly is a segment analysis that finds which segments move with a metric you care about. You build one table with every factor you suspect, run a classification model over it, and read off which segments convert well and which do not.3 min read
  5. 20The art of reading data: definition plus contextReading data well means defining a metric so a movement points at something concrete, then reading it with the business context to know what it means.4 min read
  6. 21Qual plus quant analysis: why half the picture misleadsQuantitative analysis tells you what is happening. Qualitative analysis tells you why. Most people do one half and stop, thinking they have the whole picture.3 min read
  7. 22Correlation is not causation: how to check before actingA finding that users who do X are more likely to convert may be pure correlation. Those users might simply be your most motivated ones, and would have converted anyway. Your existing data cannot tell you which one you are looking at, so check: look at the users who converted without doing X, run a small experiment on the people who have not converted, and talk to users.3 min read
  8. 23Sense checking: testing a number before you report itA sense check is holding a number against what you already know about the business, to see whether it is in the right ballpark before anyone acts on it. Compare it with numbers you have heard, with your own experience, and with a rough rebuild from visits and sign-ups a day. If it fails, verify it before you decide anything on it.3 min read
Part 06

Acting on what you find

Testing the insights you have collected, working out whether your ad spend creates demand, and telling the story so somebody acts on it.

5 chapters · 18 min read

  1. 24How to run experiments at different stages?Most companies collect insights and never test them. You might not have the traffic for a textbook A/B test, and that is fine. Run a smaller version instead, such as a before-and-after comparison or a group of users held back, and keep running it.3 min read
  2. 25Incremental testing: are your ads creating demand?Incremental testing tells you whether your ad spend brings signups that would not have arrived anyway. You switch ads off in a holdout group of markets, leave them running in a matched group, and compare signups. The gap between the two groups is your incremental signups.4 min read
  3. 26Media mix modelling: measuring untrackable channelsMedia mix modelling measures the channels you cannot track with a link, such as influencers, offline campaigns and brand spend. You give it one row per week with spend per channel, total signups and what else was going on. It works backwards to split your signups into base and the part each channel caused.3 min read
  4. 27Data storytelling: building a dashboard people understandA dashboard people understand opens on a question, orders its reports so each one follows from the last, and lands on an answer.3 min read
  5. 28Analytics in the age of AI: making agents trustworthyAI can answer your analytics questions, but only with clean data, a written explanation of what it means, and the exact calculation for each core metric. Skip those and it still answers but a good chunk of the time the answer is wrong.5 min read