Qual plus quant analysis: why half the picture misleads

Quantitative analysis tells you what is happening. Qualitative analysis tells you why. Most people do one half and stop, thinking they have the whole picture.

I've seen this split over and over. Some Founders and PMs watch session recordings, or talk to a couple of users, and stop there. Others plot charts and funnels and stop there.

Quantitative analysisQuantitative analysisThe numbers half of the work: charts, funnels, conversion rates. It tells you what is happening, and how widespread it is.Glossary is the numbers: charts, funnels, conversion rates. Qualitative analysisQualitative analysisThe people half of the work: watching session recordings and talking to users. It tells you why something is happening, which the numbers never will.Glossary is the people: watching session recordings and talking to users.

Look at either half on its own and you'll get some good insights. That's exactly the problem. The insight is good enough that you think you have the whole picture, and you start acting on it. In a lot of cases you don't have it.

The people who plot the charts go deep into the data. But they rarely take the time to check what they found by watching recordings or talking to users. The people who rely on recordings and conversations never go deep into the data.

The numbers leave out why. Say your funnelFunnelThe steps a user goes through in order. Every product is one, and so is every feature inside it. Counting who reaches each step is what shrinks a whole-product problem to a single step.Glossary tells you that sign up to activation is 60%. Activation is the first key action that shows a user got value from your product.

You take the other 40% and split them into buckets. Some ended their session right after signing up. Some went to the page where the activation action happens and never took it. A few other patterns.

Now you pick a bucket. The biggest one is people who reach that page and do nothing, so you decide to run experiments there.

You still have no answer to why. Why did those people go to the page and do nothing?

That answer comes from session recordings and talking to users. You take that group, watch their recordings, speak to them, and you get a much better idea of what's happening and what you need to solve.

All three tools let you go from the funnel straight to the sessions.

• In Mixpanel you click a point on a funnel report and select View Replays, as Mixpanel's Session Replay docs describe.
• In PostHog you drill into a funnel step and watch recordings of the users who converted or dropped off at it (PostHog's guide to watching recordings).
• In Amplitude you click a converted or drop-off point on a funnel step and select Watch Session Replays (Amplitude's Session Replay docs).

Without the qualitative half you end up with a list of experiments that takes you nowhere. You're testing guesses one by one and hoping one of them lands.

Recordings and conversations leave out how many users a problem affects. It goes the same way in reverse. You watch session recordings, spot a problem, speak to a few users who point at the same thing, and you spend time fixing it.

In reality it might not be much of a problem. Maybe five or ten users hit it in a month, and you're better off spending that time somewhere else. The users you spoke to are telling you about something real. It's just not real for the larger group.

Before you fix it, check your data for how many users actually hit that problem. If it's a handful, park it. If it's a big share of the funnel, it's worth your time.

1. Start in the numbers to find where users drop off and which bucket is biggest.
2. Watch that group's recordings and talk to them to find out why.
3. When recordings or conversations show you a problem first, go back to the numbers to check how many users it affects before you spend time on it.