Difference-in-differences, explained with ad sales

Say you pause ads in some states and their sales fall 10%. The ads might have caused that drop, or sales might have fallen everywhere because the season changed. A simple before-and-after can't tell.

Difference-in-differences (often shortened to DiD or diff-in-diff) fixes this with a second group. Measure how the group you changed moved (the first difference), measure how a similar untouched group moved over the same weeks (the second), and compare the two. Anything that hit both groups cancels out.

It's one of the oldest tools in economics. A famous study compared fast-food jobs in New Jersey, which raised its minimum wage in 1992, with nearby Pennsylvania, which didn't (Card and Krueger, 1994). Geo tests for ads use the same logic with sales.

In its simplest form, with sales counts:

effect = (test after − test before) − (control after − control before)

Ad tests usually compare groups of different sizes, so it's easier to work with ratios. Compare the ratio between the groups before and during the test. This is the same idea on the log scale:

lift = (ads-on ÷ holdout, during the test) ÷ (ads-on ÷ holdout, before) − 1

A lift of 20% means the holdout would have sold 20% more with ads on. This is the estimator in Google's geo experiment papers and the one on how lift is calculated.

A made-up brand pauses Meta ads in half its states (the holdout) for 6 weeks. Amazon orders, compared with the 6 weeks before:

1. Before the test, the ads-on states sold 1.2 orders for every 1 in the holdout.
2. During the test, they sold 1.4 for every 1.
3. Lift = 1.4 ÷ 1.2 − 1 = 16.7%.

A before-and-after on the holdout alone would say the pause cost 10% of orders. But the ads-on states grew 5% over the same weeks, probably the season. Difference-in-differences accounts for both: with ads on, the holdout would have sold about 16.7% more. The incremental ROAS calculator does this math and turns it into iROAS.

Difference-in-differences only works if, without the test, the two groups would have kept moving together. This is called the parallel trends assumption. You can't prove it for the test period, but you can check how well it held before:

• Plot the ratio between the groups week by week before the test. It should be flat, with small wobbles.
• Run placebo tests: apply the same math to past stretches when nothing changed. They should find no lift, and only about 1 in 20 should look "significant" by chance at 95% confidence.
• Choose the groups carefully in the first place. That's what matched market testing is for.

When no single group tracks the test group well, synthetic control builds a better comparison from a weighted blend of regions.

A single number isn't enough, because sales wobble even with no test. Order counts behave roughly like Poisson counts, so the uncertainty of a ratio can be worked out from the four order totals, then widened by how noisy your groups have been historically. The result is a range, like "17%, between 6% and 29%". If the range includes zero, the test couldn't tell. The method page shows the exact formula.

• Using only one group. A before-and-after on the test group alone mixes the ads' effect with everything else that changed.
• Picking a control that was already drifting. If the groups were moving apart before the test, the trend will look like lift.
• Changing other things in one group only. A regional promotion breaks the comparison.
• Reporting a number without a range. A 5% lift with a range of −10% to +20% isn't a finding.