Synthetic control: the method behind GeoLift

Every ad test needs an answer to "what would sales have been without the change?" In difference-in-differences, the answer comes from one comparison group. That works when a good comparison group exists. Often none does: no single state or city moves quite like the one you're testing.

Synthetic control builds a better comparison. It takes the regions that weren't changed (the donor pool) and finds the weights that make their blend track the test region as closely as possible before the test. That weighted blend is the synthetic control.

synthetic test region = w₁ × region 1 + w₂ × region 2 + … + wₙ × region n
weights are 0 or more and add up to 1, chosen to match the test region before the test
effect = actual test region − synthetic test region, during the test

Economists developed synthetic control to study events that happened to one place, where no single comparison fits.

• Abadie and Gardeazabal (2003) estimated the economic cost of terrorism in Spain's Basque Country by comparing it with a blend of other Spanish regions.
• Abadie, Diamond and Hainmueller (2010) measured the effect of California's 1988 tobacco control program, Proposition 99, on cigarette sales. They built a "synthetic California" from other states and compared the two after the law passed.

The method later moved into marketing, where it suits geo tests well: you usually have a few test regions and many possible comparison regions.

A made-up brand pauses ads in one test region. Three untouched regions, A, B and C, are in the donor pool. The weights that best tracked the test region before the pause were 0.5, 0.3 and 0.2. Average weekly orders:

Before the pause, the blend matched the test region exactly: 0.5 × 120 + 0.3 × 80 + 0.2 × 100 = 104. During the pause, the blend says the test region should have sold about 109 orders a week. It sold 96, about 12% fewer. That gap is the estimated effect of the ads in that region. Real tests use many more weeks and regions, plus a range of uncertainty.

GeoLift, Meta's open-source package for geo tests, uses the augmented synthetic control method (Ben-Michael, Feller and Rothstein, 2021). Plain synthetic control struggles when no blend of donors tracks the test region well. The augmented version adds a correction for the part of the gap the blend can't close.

GeoLift also helps plan the test: it simulates candidate sets of test markets on your history and estimates which would detect a given lift. That's the same planning question as matched market testing, answered with a different model.

It works well when:

• You have a long history before the test, so the weights are fit on many weeks.
• The blend tracks the test region closely before the test. A poor fit before means a poor estimate after.
• The donor regions weren't affected by the change, including by ads leaking across borders.

It's less suited when:

• The test region is unusual, like your biggest market, and no blend of smaller regions can match it.
• The history is short or very noisy, so the weights mostly fit noise.
• You've split the country into two large groups. Then each group already tracks the other, and simpler ratio methods like difference-in-differences do the job.

Abadie (2021) sets out these conditions in detail and is the best single reference on when the method is credible.

Spillmark splits all US states into two large groups and has you pause Meta ads in one. With groups that size, a log-ratio difference-in-differences with placebo checks is simpler to read and to audit than a synthetic control. GeoLift-style synthetic control is the better tool for tests on a few small markets. The full method is on how lift is calculated. To size a test from your own sales first, try the holdout test calculator.