Back to the playbook

Reading the data

Frameworks for taking apart any product problem, defining metrics so they point at something, and reading a number instead of just looking at it.

  1. 16A framework for any product problem: four questionsLow retention, low activation and low signup-to-paid conversion are the same problem from a data perspective. Take apart any of them and it boils down to: who converts, what they did that everyone else did not, what the two say together, and which of them actually caused the conversion.5 min read
  2. 17Metric definitions: why three people report three numbersThree people report three different numbers for the same metric because they built three different formulas for the same metric.3 min read
  3. 18Rate and mix analysis: why your conversion rate movedWhen a conversion rate moves, two different things can move it: the rates inside each segment, and how much volume each segment got. Build one counterfactual period and the two separate cleanly.4 min read
  4. 19Segment analysis: finding which segments move a metricFugly is a segment analysis that finds which segments move with a metric you care about. You build one table with every factor you suspect, run a classification model over it, and read off which segments convert well and which do not.3 min read
  5. 20The art of reading data: definition plus contextReading data well means defining a metric so a movement points at something concrete, then reading it with the business context to know what it means.4 min read
  6. 21Qual plus quant analysis: why half the picture misleadsQuantitative analysis tells you what is happening. Qualitative analysis tells you why. Most people do one half and stop, thinking they have the whole picture.3 min read
  7. 22Correlation is not causation: how to check before actingA finding that users who do X are more likely to convert may be pure correlation. Those users might simply be your most motivated ones, and would have converted anyway. Your existing data cannot tell you which one you are looking at, so check: look at the users who converted without doing X, run a small experiment on the people who have not converted, and talk to users.3 min read
  8. 23Sense checking: testing a number before you report itA sense check is holding a number against what you already know about the business, to see whether it is in the right ballpark before anyone acts on it. Compare it with numbers you have heard, with your own experience, and with a rough rebuild from visits and sign-ups a day. If it fails, verify it before you decide anything on it.3 min read