In short
Holdout test brand size is really a noise question: with your monthly Amazon sales split across regions, how small a Meta lift can a geo test still see? Chance alone sets a floor on week-to-week swings. A region with few orders moves a lot in percentage terms, so the minimum detectable effect stays large until you add weeks, regions, or sales. This calculator turns monthly sales and average order value into an estimated orders-per-region-week, a planning coefficient of variation, and the same MDE math as the full holdout sample-size tool. If the MDE sits above the lift you care about, don't pause ads for a test that can't speak. Grow the window, wait for steadier volume, or accept that only a large effect would show up.
What does brand size have to do with a holdout?
A geo holdout pauses Meta ads in some regions and leaves them on in others, then compares Amazon sales. The test only works if the lift you care about is larger than ordinary week-to-week noise. Small brands feel that noise hardest: each state sees few orders, percentages swing, and a short test can't tell a 15% lift from chance.
Brand size here means national Amazon sales and how those sales split into region-weeks. More sales per region per week lowers the chance floor on variability. More regions and more weeks shrink the minimum detectable effect further. This page answers the first question founders ask: whether the brand is even big enough to run one.
How to calculate detectable lift from brand size
weekly national orders = monthly sales ÷ AOV ÷ weeks-per-month orders per region-week = weekly national orders ÷ regions CV (planning) = max(1 ÷ √orders-per-region-week, 25%) MDE = (z_95 + z_80%) × CV × √(2 ÷ (regions-per-group × weeks))
- Monthly sales: a typical Amazon month, not your best week.
- AOV: sales ÷ orders, so dollars become order counts.
- 25% CV floor: promotions, stockouts and season add noise beyond pure chance.
For full control of CV, confidence and power, use the holdout test calculator. For an example state list at a target share, use the geo holdout states calculator.
Worked example
A made-up brand does about $60,000 a month on Amazon at a $40.00 average order. It can split 50 regions and hopes to see a 20% lift in 6 weeks. These are the calculator's example values.
- About 6.9 orders per region per week → planning CV 38.1%.
- Split into 25 holdout and 25 ads-on regions.
- Minimum detectable lift ≈ 12.3%.
- Verdict: this design can see the hoped-for lift.
Compare a smaller shop at $8,000/month hoping for 15% in four weeks: MDE ≈ 41.3%, verdict too-small. Pausing ads there would mostly buy uncertainty.
When is a brand big enough?
Big enough means the MDE sits below the lift that changes a decision, usually the lift ads need to beat break-even after Amazon fees. If break-even needs 25% incremental lift and your MDE is 40%, the test can't clear the bar you care about.
Ready designs detect your target lift in the weeks you will actually wait. Stretch designs need a longer window but stay realistic (often under about twelve weeks). Too-small designs need so long that season and creative drift swamp the answer. Wait, or use a larger effect size as the only honest target.
Common mistakes
- Using peak month sales. Size on a typical month or the test will look easier than it is.
- Ignoring the CV floor. Sparse regions are noisy even when the national chart looks smooth.
- Running anyway "to learn." An underpowered test teaches you to distrust measurement. That's expensive education.
- Skipping matched groups. Equal-weight region counts are a start. Real designs match on your ship-to history.
How this connects to true incrementality
Incrementality is the sales you would lose without the ads. A holdout estimates that only when it's powered. After a clear test, turn the result into dollars with the incremental ROAS calculator and check waste with the ad waste calculator.
Spillmark plans geo holdout tests for Meta ads across US states from your Amazon order history and reports the range. See how lift is calculated.
Frequently asked questions
How big does my brand need to be for a holdout?
Big enough that the minimum detectable lift sits below the lift you care about, in a window you will actually run. This calculator estimates that from monthly sales and AOV.
Why does average order value matter?
Noise is about orders, not dollars. AOV turns sales into order counts so the chance floor on variability is realistic.
Can a small brand ever run a geo test?
Sometimes, with more weeks, fewer regions, or a larger effect size as the target. If weeks needed stretch into many months, wait for volume.
Is this the same as the holdout sample size calculator?
It uses the same MDE math, but starts from brand size instead of asking you for CV up front. Use the full holdout calculator when you already know variability.
Should I include Shopify sales?
Only if you will measure and pause ads against those regions the same way. This page is aimed at Amazon ship-to state tests.