How to hire a product analyst: screen for the mindset

Founders screen product analysts for SQL and tools, then hire someone who pulls any number but can't say which number to ask for. The skill that separates a good product analyst is understanding how users move through the product well enough to turn a vague problem into one specific question. You can test for it in ten minutes by asking a candidate to map your product and say where users would drop off.

Founders keep asking me the same question. They want to hire a product analyst, and they want to know how to tell a good one from a bad one.

The reason it's hard is that a product analyst looks a lot like the other two analytics roles on paper. Same tools, same SQL, often the same job titles. But a good product analyst thinks very differently from someone with a business analyticsBusiness analyticsAnalytics answering questions about business objectives rather than user behaviour: revenue by country, cohorts against targets, reconciling revenue with finance data. It sits alongside product analytics under data analytics.Glossary or a data analyticsData analyticsThe broad term: taking data, pulling insight out of it and understanding what is happening. Product analytics and business analytics both sit under it.Glossary mindset. I've written about the three roles separately in Product vs data vs business analytics. In my experience product is the toughest of the three.

Understanding how users move through the product. Past the SQL and the Python, that's what it comes down to. I call these paths flows: sign up, set up, invite a teammate, come back next week. There are so many different flows a user could take.

Someone hands you a problem statement. You have to condense all of those flows into one specific question, or one metric, that will actually get you the answer.

That sounds easier than it is. It mostly comes with experience, from working as a product analyst long enough to know: this is the flow, so this is the metric we look at here. Different flow, and a different set of metrics makes more sense.

Here's what that looks like on two example problems.

If someone tells me retentionRetentionThe share of users who come back. Bounded retention counts people who returned on exactly that day; unbounded counts that day or any day after, and the two give very different numbers.Glossary has gone down, here's roughly how I go at it.

First I ask what retention means here. Day 7, day 30, monthly? Bounded or unbounded?

Then I split the drop. Are new users retaining worse, or are existing users leaving? Those are two different problems with two different fixes, and the overall number hides which one you have.

If it's new users, I go back to the flows. I take a look at what the users who stuck around did in their first session, and what the ones who left did instead. Compare the two groups on the actions they took, the channel they came from, the platform they were on. The difference between the two groups is where the answer usually sits.

If it's existing users leaving, I look at what changed in the product around the time the drop started. A new release, a bug, a pricing change, a feature that moved. Then I check whether the drop is spread across everyone or sitting in one cohort, because a change that only affects one plan or one platform shows up as one group leaving, not all of them.

Activation is the moment a new user first gets real value from the product. If the question is how do we increase activation, I start somewhere else:

1. Ask why a user would not activate.
2. Turn each answer into a hypothesis, a guess you can check in the data.
3. Go deeper into the data for each one.
4. Validate or invalidate it.

We did this with VideoTap. Most signups were dropping out during onboarding, and the ones who finished it still weren't uploading a video, which is where the product's value actually shows up. So we audited the onboarding step by step, then went deeper into the upload failures. The upload went from four steps to two, we added guidance where users were getting stuck, and we fixed the YouTube upload errors that were killing uploads outright. Onboarding completion went from 28% to 80%, and activation went from 7% to 19%.

That's the product analyst mindset.

Give the candidate your product and ask them to map it out and tell you where people would drop off. Let them use the product for a few minutes first.

You're not grading the answer against what you know to be true. You're watching whether they ask what the product is for and who's using it, whether they walk the flows in order instead of listing features, and whether the drop-off points they name are places a real user would get stuck rather than just the last screen before payment.

If they can't do that, they don't understand how users move through the product, or how products get built, and they're not going to be good at this job.

The same mindset shows up in how someone answers a question you asked them. You're looking for someone who:

• asks what a metric means before answering a question about it
• splits a number into its parts before explaining it
• goes back to how users move through the product to find the cause
• builds hypotheses and checks them, instead of jumping to a fix

And the same test applies to you. Could you map out your own product and say where people would drop off?

Hope this was helpful. If you're hiring for this and want a second pair of eyes on a candidate, or on a number that isn't adding up, hit us up.