Matched market testing: how to pick test regions

A matched market test is a geo test where the test and control regions are chosen to be alike. The idea is old: brands have long launched products in a few "test markets" and compared them with similar markets that didn't get the launch. For ads, you change spend in the test markets and use the matched control markets to see what would have happened otherwise.

"Alike" doesn't mean the same size. It means their sales move together: when one has a good week, so does the other. Two states whose sales rise and fall in step make a good pair even if one sells three times as much.

In a test with millions of people, random assignment balances the groups by chance. With 50 states or a few hundred metro areas, it doesn't. A random split might put most of your fastest-growing states on one side, and the test would mistake that growth for the effect of the ads.

Matching also cuts noise. If the two groups' ratio barely moves from week to week before the test, a small change during the test stands out. If the ratio swings around, only a large effect can be seen. That's why the minimum detectable effect depends on how steady the groups are, not just on total sales.

1. Get history by region. At least 8 to 12 weeks of sales, ideally more, grouped by state or metro area.
2. Try many splits. Generate many candidate groupings, each close to the share of sales you want in the test group.
3. Score each split. Measure how steady the ratio between the two groups has been, week by week. Lower variation is better.
4. Check on held-back weeks. A split chosen for being steady will always look steady on the weeks it was chosen on. Keep the most recent weeks aside and check the winner on those.
5. Sanity-check the result. Look for odd groupings, like every big state on one side, and for regions with their own events coming up.

Other approaches build the control from a weighted blend of regions instead of a fixed group. That's synthetic control, the method behind Meta's GeoLift, which also has tools for choosing test markets.

Two candidate splits for a made-up brand, each with about half its Amazon orders in the holdout. The table shows the ratio of orders in the ads-on group to the holdout, week by week, before any test:

Split A's ratio stays within 2% of 1.00. Split B's swings by more than 10% with no test running. In split A, a 10% lift would stand out clearly. In split B, it would look like an ordinary week. Same brand, same sales, very different tests.

• Matching on size, not movement. Two regions with the same sales can still move very differently.
• Hand-picking test markets. Choosing your "best" or "worst" regions builds in a difference before the test starts.
• Too little history. A few weeks can make any split look steady. Use months if you have them.
• Ignoring local events. A regional promotion, retailer launch or weather event in one group can look like a lift.
• Skipping the size check. Even a well-matched test can be too small. Check what it can detect before pausing any ads.

Spillmark draws 4,000 random splits of the 50 states and DC from your own Amazon order history, keeps the one whose weekly ratio was steadiest, and checks it on the most recent 8 weeks, which were left out of the choice. Then it sizes the test and locks the design. Details are on how lift is calculated.