Geo testing: how geo lift tests measure ads

Geo testing (also called a geo experiment or geo lift test) splits a country into regions and treats them differently. Ads stay on in some regions and pause, or increase, in others. After a few weeks, you compare how sales moved in each set of regions. The difference is the effect of the ads, or the lift.

The regions can be states, metro areas (in the US, often Nielsen's 210 Designated Market Areas, or DMAs), or postal codes. Ad platforms like Meta and Google let you target or exclude any of these, which is what makes the test possible.

Google researchers described the approach in 2011 and later added a time-based regression version (Vaver and Koehler, 2011; Kerman and others, 2017). Meta publishes an open-source tool for it, GeoLift. For how geo tests sit next to user holdouts and models, see incrementality measurement.

Ad platforms can run tests by person: show the ads to some users and not others. That's a clean design, but the platform can only count the sales it can track, such as purchases its pixel sees on your site. Sales on Amazon, in stores, or by phone are invisible to it.

Geo tests get around this. You only need sales by region, and most sales records already have that:

• Amazon order reports include the ship-to state and postal code.
• Shopify and other store platforms record shipping addresses.
• Retail sales can be grouped by store location.

Geo tests also capture effects that tracking misses, like someone who sees an ad and searches for you on Amazon a day later. That sale is counted in their region whether or not anyone tracked a click.

1. Collect history. Get several months of daily or weekly sales by region.
2. Choose the groups. Pick regions for the test and control groups that have sold in step in the past. See matched market testing.
3. Size the test. Work out the smallest lift the design could reliably detect, and how much ad spend it pauses. If it can't see a realistic lift, change the design before you start. The holdout test calculator gives a first estimate.
4. Check the calendar. Avoid big sales events and your own promotions, which can swamp the effect.
5. Run it. Change ads in the test regions using location targeting, and change nothing else.
6. Measure. Compare how each group moved against its own history. The common methods are difference-in-differences, time-based regression and synthetic control.

• Holdout (go dark). Pause ads in the test regions. This measures what the current spend is doing. It's the most common design and the easiest to read.
• Scale-up (heavy-up). Raise spend in the test regions. This measures what more spend would do, which is useful before a budget increase, since returns usually shrink as spend grows. The marginal ROAS calculator explains why.
• New channel launch. Turn a new channel on in some regions first, then compare. This answers "is it worth rolling out?" before you commit nationally.

• Leakage. People travel, commute and move across region lines, and location targeting isn't perfect. Leakage makes the groups more alike, so it pushes the measured lift toward zero.
• Small brands. With only a handful of orders a day, region-level sales are too noisy to see a realistic lift in a few weeks.
• Fewer units than a user test. There are 50 states, not millions of users, so groups can't be fully random and balanced by chance. Careful matching makes up for it.
• One moment in time. A result holds for the spend, creative and season you tested. Re-test after big changes.

Spillmark runs geo holdout tests for Meta ads across US states and measures Amazon orders by ship-to state. It picks the groups from 4,000 random splits of your own order history, sizes the test before anything pauses, checks the calendar, and locks the design before the test starts. The full method is on how lift is calculated.