In short
The minimum detectable effect, or MDE, is the smallest lift a holdout test can reliably tell apart from normal week-to-week noise. If your ads lift sales by 15% and the test's MDE is 30%, it will most likely report no clear effect even though the ads work. The MDE gets smaller with more regions, more weeks and steadier sales. Chance alone sets a floor on the noise: a region with 25 orders a week swings at least 1 ÷ √25 = 20%. Use 1.96 for 95% confidence or 1.64 for 90%. A good MDE sits below the lift your ads need to break even, so a null result still tells you something. Measure variability from past weekly sales by region, and fix the test length before you start.
What is a minimum detectable effect?
A geo holdout test measures what your Meta ads cause. You split regions, usually US states, into two groups. One group keeps its ads on. The other, the holdout, pauses them for a few weeks. Then you compare Amazon sales between the groups, before and during the test.
The minimum detectable effect (MDE) is the smallest lift a test can reliably tell apart from normal week-to-week noise. If your test's MDE is 30% and your ads really lift sales by 15%, the test will most likely say "no clear effect", even though the ads work. That's the most common way holdout tests fail: not a wrong answer, but a test too small to give one.
So size the test first. This calculator tells you the MDE of a design, and how many regions or weeks you'd need to see the lift you expect.
How to calculate the minimum detectable effect
regions per group = regions available ÷ 2, rounded down n = regions per group × weeks MDE = (z_confidence + z_power) × CV × √(2 ÷ n) weeks needed = 2 × ((z_confidence + z_power) × CV ÷ expected lift)² ÷ regions per group
- CV (coefficient of variation): how much one region's weekly sales swing, as a share of its average. Chance alone sets a floor. A region with 25 orders a week swings at least 1 ÷ √25 = 20%; one with 4 orders a week, at least 50%. Promotions, stockouts and seasons add more.
- z_confidence: 1.96 for 95% confidence (two-sided), 1.64 for 90%.
- z_power: 0.84 for 80% power, 1.28 for 90%.
- Expected lift: how much higher sales are with ads than without.
The MDE shrinks with the square root of the test's size. Four times as many region-weeks halves it. That's why small brands often need longer tests than they'd like.
Worked example
A made-up brand sells about $1,500 a week on Amazon in each of 20 regions. Its weekly sales per region swing by about 35%. It hopes Meta ads lift sales by 20% and plans a 6-week test. These are the calculator's example values.
- 20 regions split into 10 with ads off and 10 with ads on. n = 10 × 6 = 60 region-weeks per group.
- At 95% confidence and 80% power, z = 1.96 + 0.84 = 2.80.
- MDE = 2.80 × 35% × √(2 ÷ 60) = 17.9%. That's about $268.54 per region per week.
- 17.9% is below the 20% it hopes for, so the test is big enough. 5 weeks would just do.
If the ads only lift sales by 10%, the same design is too small. It would need 20 weeks with these regions, or 66 regions in 6 weeks. Halving the lift you want to see means four times the test.
The test has a cost. If the 20% lift is real, the holdout regions give up about $15,000 of sales over 6 weeks, while $6,000 of Meta spend is paused.
| Weeks | Minimum detectable lift |
|---|---|
| 4 | 21.9% |
| 6 | 17.9% |
| 8 | 15.5% |
| 12 | 12.7% |
What's a good minimum detectable effect?
A good MDE is one below the smallest lift that would change your decision. For most brands, that's the lift your ads need to pay for themselves:
break-even lift ≈ break-even ROAS × weekly Meta spend ÷ weekly sales without ads
If your ads need a 15% lift to break even and your test can only see 25%, a "no clear effect" result can't tell you whether the ads pay. Get your break-even ROAS from the break-even ROAS calculator.
There's no universal good number. It depends on your order volume, how steady your sales are and how many regions you can use. Brands with few orders per region tend to have high variability, so they need more weeks.
Common mistakes
- Guessing variability too low. A low CV makes any test look big enough. Measure it from past weekly sales by region, or use the 1 ÷ √orders floor as a minimum.
- Stopping when the result looks good. Checking every day and stopping at the first good-looking week inflates false alarms. Pick the length up front and stick to it.
- Letting budget move. If the paused regions' budget flows to the ads-on regions, the test measures more spend, not the same spend. Keep each group's spend steady.
- Testing over a big event. Prime Day or a big promotion in only some regions adds noise the plan didn't count.
- Forgetting the lag. Some shoppers buy days after seeing an ad. Leave time after the test ends before you read the result.
How this connects to true incrementality
Sizing the test is step one. Once it's run, the incremental ROAS calculator turns the results into the sales your ads caused, and the attribution gap calculator shows how far Meta and Amazon Attribution were from that.
This page uses a textbook approximation. Spillmark plans tests from your own order history instead, runs a placebo check on past weeks to measure real noise, and locks the design before the test starts. See how lift is calculated for the full method.
Frequently asked questions
What is a geo holdout test?
A test that pauses ads in some regions and keeps them on in others, then compares sales. The difference, adjusted for how the regions normally compare, is the effect of the ads.
How long should a holdout test run?
Long enough that its minimum detectable effect is below the lift you care about. For small brands that's often 6 to 12 weeks. Use the calculator to check your own numbers.
How many regions do I need?
More regions make a test more sensitive, but each must be big enough to measure. US states are a common choice because Amazon order reports show the ship-to state.
Is minimum detectable effect the same as the lift I expect?
No. The expected lift is what you think the ads do. The minimum detectable effect is the smallest lift the test can see. A good test has an MDE below the expected lift.
Is this the same as an A/B test sample size calculator?
It's the same idea, but for regions over weeks instead of visitors. Region sales are noisier than single clicks, so the variability input matters more.