How lift is calculated

• Spillmark splits the US into two groups of states that normally sell in step.
• During the test, your Meta ads keep running in one group and pause in the other. The paused group is the holdout.
• If the holdout falls behind its usual share of your Amazon orders, the gap is what your ads were adding.
• Every result comes with a range. If the range is wide, the test couldn't tell, and Spillmark says so.

Meta counts a sale when someone saw or clicked your ad and then bought. Amazon Attribution credits Meta when a shopper clicked a tagged ad and bought within 14 days. Both are rules for handing out credit. Neither can tell you if the shopper would have bought anyway. That gap is why incremental sales differ from attributed sales.

They also miss people. Someone who sees your ad on Instagram, then searches for you on Amazon a day later, never clicked a tag. A holdout test doesn't depend on tracking any click. It counts every Amazon order by the state it ships to, so it sees those shoppers too. For how to choose a design, see incrementality measurement.

Orders come from Amazon's All Orders report (or the same report through Amazon's API). Spillmark counts each order once, by ship-to state and local date. It drops cancelled orders, orders outside the US, and Multi-Channel orders from other sites.

Then it draws 4,000 random splits of the 50 states and DC, each close to the share of orders you want in the holdout. It keeps the split whose weekly ratio between the two groups was steadiest. Steady groups make a small change easy to see.

The most recent 8 weeks are left out of that choice and used afterwards to check the noise. A split chosen for being steady always looks steadier on the weeks it was chosen on, so this keeps the check honest.

Spillmark tries every combination of 30%, 40% or 50% of orders in the holdout and 4 to 12 weeks of testing. For each one it works out the smallest lift the test could reliably detect (the minimum detectable effect) and how much Meta spend you'd pause.

It recommends the cheapest option that can detect the lift you care about. If none can, it tells you before you start, so you don't pause ads for an answer you won't get.

Here is how order volume and test length change what a test can see. These are best cases with a 50/50 split and 12 weeks of history:

Real sales are noisier than this. Promotions, stockouts and seasons all add noise, so Spillmark measures it from your own history and the real numbers come out larger.

To try rough numbers for your own brand, use the free holdout test calculator.

A test that overlaps Prime Day, Prime Big Deal Days, Black Friday week or December can be thrown off by the sale, not the ads. Spillmark checks those dates, your own promotions and price changes, and any earlier test (plus 2 weeks after it). A serious overlap blocks the test unless you choose to accept the risk, and that choice is written into the test design.

Before the test starts, Spillmark writes down the groups, the dates, the holdout share, the expected minimum detectable effect and the noise level. It takes a SHA-256 fingerprint of that record. Any later change to the design would change the fingerprint.

Each record in a brand's ledger includes the fingerprint of the one before it, and the latest fingerprint is time-stamped by an independent timestamp authority (RFC 3161). Results are added when the test ends and are never edited. So you can show a test report to an investor, a buyer or your agency, and they can check it wasn't redesigned after the fact.

You pause Meta ads in the holdout states using location targeting and leave everything else alone. When the test ends, Spillmark compares each group's orders during the test with the same group before it. The math is below.

Lift. Before the test, the two groups sell in a steady ratio. That ratio predicts what the holdout would have sold with ads. The shortfall is the effect of the ads:

lift = (N_on,test / N_off,test) ÷ (N_on,pre / N_off,pre) − 1

N is the number of orders, "on" is the group that kept its ads, "off" is the holdout, "pre" is the baseline before the test (12 weeks by default) and "test" is the test itself. A lift of 20% means the holdout states would have had 20% more orders with ads on. This is a difference-in-differences on the log scale: it cancels out anything that moves both groups together, like a holiday or a price change everywhere.

Error range. Order counts behave roughly like Poisson counts, so the variance of log N is about 1/N. The four counts are independent, so their variances add:

SE(log lift) = √( φ · (1/N_on,pre + 1/N_off,pre + 1/N_on,test + 1/N_off,test) )
95% range = exp(log lift ± 1.96 · SE) − 1

φ (phi) is the noise multiplier: how much more the weekly ratio between the groups wanders than pure chance would allow. It's measured from your history and is never below 1. At analysis time Spillmark uses the larger of the φ locked at registration and the φ measured in the baseline, so a noisy baseline can only widen the range. The p-value is the usual two-sided test of "no lift".

Minimum detectable effect. For 95% confidence and 80% power:

MDE = exp((1.96 + 0.84) · SE) − 1

The planner doesn't trust this formula alone. It also runs placebo tests: the same calculation on every past stretch of weeks when nothing changed. Those should find no lift, and about 5% of them will look "significant" by chance. Spillmark uses whichever is larger, the formula's error or the spread of the placebo results.

From lift to orders and profit. The lift turns into orders per day across the whole country:

extra orders a day = (orders a day with ads on everywhere) × lift ÷ (1 + lift)
extra sales a day  = extra orders a day × average order value

Divide by Meta spend per day to get incremental ROAS (iROAS). That dollar lift is also what people mean by marketing incrementality. Then Spillmark takes off Amazon fees and your product costs, because an extra sale only helps if it makes money after fees, costs and the ads that caused it.

We ran 200 simulated tests at 30 Amazon orders a day, a 50% holdout and 6 weeks, with a true lift of 25%. The data was simulated, not any brand's:

• The average estimate was 24.7%, so the method isn't biased.
• The 95% range contained the true lift 96 to 97% of the time.
• With no real effect, 3% of tests came out "significant", under the 5% you'd expect.
• For this design, the planner's minimum detectable effect was about 23%.

Before you run a test, Spillmark estimates the effect from how your Amazon orders moved when Meta spend changed, day to day and state to state. It uses a state-by-day model with state and day effects (errors clustered by state), checked against a national daily model with weekday and trend terms (Newey-West errors). This is weaker than a test, because spend doesn't change at random. Spillmark shows it with its range and uses it to decide whether a test is worth running. A test gives the firmer answer.

• Small brands. Under about 10 Amazon orders a day, most tests can't see a realistic lift. Spillmark tells you before you start.
• Ads leak across state lines. People travel, and location targeting isn't perfect. Leakage makes the two groups look more alike, so it pushes the measured lift toward zero. Read a result as conservative.
• One test is one moment. It measures your ads at one level of spend, with one set of creative, in one season. Effects change, so test again when those change.
• A holdout costs some sales. If your ads work, the holdout states sell less during the test. Spillmark shows the expected cost before you start.
• Ship-to state, not ad location. Amazon reports where an order ships, not where the shopper saw the ad. Gifts and travel add a little noise.

Geo experiments are a published, peer-reviewed method. Spillmark's estimator is the simple ratio form described in Google's geo experiment papers. Meta's GeoLift uses synthetic control, a related method.

• Vaver & Koehler, Measuring Ad Effectiveness Using Geo Experiments (Google, 2011)
• Kerman, Wang & Vaver, Estimating Ad Effectiveness using Geo Experiments in a Time-Based Regression Framework (Google, 2017)
• Ben-Michael, Feller & Rothstein, The Augmented Synthetic Control Method (2021), the method behind Meta's open-source GeoLift
• GeoLift, Meta's open-source geo experiment package