What is a holdout group in marketing?

In a holdout test, your audience is split in two. The test group (sometimes called the treatment group) sees your ads as normal. The holdout group (also called the control group) doesn't. Everything else stays the same: prices, listings, emails, other channels.

If the two groups started out alike, any difference in sales during the test came from the ads. That's the whole idea, and it's why holdout tests are the most direct way to measure incrementality.

lift = (test group sales ÷ holdout group sales, adjusted for size) − 1
incremental sales = test group sales − what the holdout says they'd have been

There are two ways to build a holdout group, and the choice decides what sales you can measure.

• A user holdout. The ad platform randomly picks some users and doesn't show them your ads. Meta's conversion lift studies work this way. Random selection by person is the cleanest design, but the platform can only compare purchases it can see, from its pixel or the data you send it. Sales on Amazon usually aren't among them.
• A geo holdout. You turn ads off in some regions, like states or metro areas, and keep them on elsewhere. You then compare sales by region from your own records. Amazon reports where each order ships, so a geo holdout can measure Amazon sales directly. See geo testing.

Random selection matters. If you pick the holdout yourself, say your weakest regions, you'll measure the difference between regions, not the effect of the ads. With only a few dozen regions, pure chance can also give you two groups that don't match, so geo tests choose groups that have sold in step historically. That's matched market testing.

A bigger holdout makes a clearer answer, but it pauses more ads, which costs sales if the ads work. Three things set the size you need:

• Sales volume. More orders means less noise. A brand with 10 Amazon orders a day needs a much larger or longer test than one with 100.
• The lift you expect. A 30% lift is far easier to see than a 5% one.
• Test length. More weeks means more orders counted, which narrows the uncertainty.

Many brands underestimate this. In 25 large ad experiments at US retailers and brokerages, the median uncertainty range on return on ad spend was more than 100 percentage points wide, because individual sales are so noisy (Lewis and Rao, 2015). Work out the minimum detectable effect before you start: the smallest lift your test could reliably see. The holdout test calculator does it from your weekly sales.

Splits between 30% and 50% of sales in the holdout are common for geo tests. A 50/50 split gives the most precise answer for a given number of weeks. A smaller holdout pauses less, but needs longer.

A made-up brand sells about 40 Amazon orders a day across the US and spends on Meta ads nationwide.

1. It splits the states into two groups that have each had about half its orders, and whose ratio barely moved week to week.
2. For 6 weeks, it pauses Meta ads in one group (the holdout) using location targeting. Nothing else changes.
3. Before the test, the ads-on group sold 1.2 orders for every 1 in the holdout. During the test, it sold 1.4 for every 1. The ratio grew by 1.4 ÷ 1.2 − 1 ≈ 17%.
4. So the holdout states would have had about 17% more orders with ads on. That's the lift, with an uncertainty range around it.

The incremental ROAS calculator turns results like these into incremental sales and iROAS.

• Changing other things mid-test. A price cut or a promotion in only one group breaks the comparison.
• Stopping early when it looks good. Peeking and stopping on a good week makes false wins much more likely. Fix the length in advance.
• Running into big sales events. Prime Day or Black Friday can swamp the effect you're trying to measure.
• Letting ads leak in. Location targeting isn't perfect, and people travel. Some holdout shoppers will see ads, which pushes the measured lift toward zero. Read the result as conservative.
• Showing the holdout a different ad. Some older designs showed the holdout a charity ad in place of yours. Researchers have shown that logging when your ad would have been shown, called "ghost ads", measures the same thing far more cheaply (Johnson and others, 2017).

Spillmark runs geo holdouts across US states for Meta ads and measures Amazon orders by ship-to state. It chooses the groups from your own order history, sizes the test before anything pauses, checks the calendar for sales events, and locks the design with a fingerprint before the test starts. The steps and the math are on how lift is calculated.