Incremental testing: are your ads creating demand?

Incremental testing tells you whether your ad spend brings signups that would not have arrived anyway. You switch ads off in a holdout group of markets, leave them running in a matched group, and compare signups. The gap between the two groups is your incremental signups.

You are spending $1,000 a month on ads. One campaign is doing well and you are ROI-positive, so you scale it to $2,000, then $3,000.

Signups and revenue do not go up with it.

You tell yourself it needs time. Meanwhile you are paying three times as much for the same result.

An incremental signupIncrementalityThe share of an outcome that only happened because of the spend. An incremental sign-up is one that would not have arrived anyway through the product, content or word of mouth.Glossary is one that only happened because of the ad. Everything else would have arrived anyway.

For most products, a lot of the demand at any given moment already exists. You created it earlier, through the product, through content, through word of mouth.

Once you are big enough, that demand reaches you through several channels at once. Advertising harder to people who were coming anyway does not add signups. It adds cost.

Take a simple case. I know Mixpanel is an analytics tool. If I search for Mixpanel on Google, it makes no difference whether I see the ad. I already know what I want and I am going to sign up either way. Somebody is paying for that click and getting nothing.

This is not a hunch. In a study published through NBER, eBay ran two tests. It switched off its branded search ads on Yahoo and MSN, and 99.5% of that traffic came back through the unpaid search results instead. It also switched off its other paid search ads on Google in 68 US markets, and sales did not move in a way you could tell apart from zero.

The same pattern shows up with branded search, meaning ads that show when someone searches your company's name.

This is not true of every channel. Ads pointed at people who have never heard of you can genuinely create demand. What you cannot do is tell which case you are in by looking at a dashboard. You have to test it.

Split your markets into two matched groups, turn ads off in one, and compare signups. I am not going to go deep into the setup, but the idea is simple.

Say you operate in multiple cities or countries. Split them into two groups, matched on customers, revenue, and growth rate. Both groups should already have been behaving the same way before you touch anything. If you get the matching wrong, nothing after it can be trusted.

Then turn ads off in one group and leave them running in the other. The group with ads off is your holdoutHoldoutA group deliberately kept away from a campaign so the difference between them and everyone else shows what the spend actually caused.Glossary, also called the control group.

Over the test period, compare signups in the two groups. Because they moved together before the test, the gap that opens up between them is your incremental signups.

Note: put several cities on each side, not one against one. City-level numbers bounce around week to week. With a single city per group, that noise can be larger than the effect you are measuring. You would read a flat result and conclude your ads do nothing, when the test was never able to detect the lift.

You are switching ads off in half your markets for four weeks. If the ads were doing something, you lose those signups while the test runs and you do not get them back.

Two things make it worth paying. You run this once per channel, not every month. And the alternative is scaling spend on a campaign you cannot evaluate, which is the more expensive way of finding out.

Someone also has to own it: splitting the markets, checking every channel is off in the holdout, and then leaving the test alone for four weeks. That last part is the one teams get wrong.

The cities where you turned ads off have to actually have no ads reaching them, and that is harder than it sounds.

Plenty of things will still reach people in your control group:

• a retargeting audience (ads that follow people who already visited your site)
• a national email blast
• a WhatsApp campaign
• one influencer with a national following

When that happens, the gap you measure shrinks toward nothing. You conclude your ads do not work, when what actually happened is that you never stopped advertising to those people.

Before you trust a holdout, go through every channel you run and confirm that none of them is quietly serving the cities you switched off.

Four weeks is a reasonable floor. Two things take time to show up. There is a delay between someone seeing an ad and signing up. And people who saw the ad before you switched it off keep signing up for a while after. Cut the test off after a week and you miss both. Run it longer if your sales cycle is slow.

As for size, you need at least 200,000 new users on your website a month before this test is worth running.

Below that, once you split your traffic across several cities on each side, each group gets too few signups a week. The gap you are trying to measure gets lost in normal week-to-week swings, and you end up with a flat result that tells you nothing.

If you are under that, hold off. Spend the time on the channels you can track directly, where each signup can be tied back to the link or campaign it came from. Come back to this test once you have the traffic.