Tests
What should the next holdout test be, what will it cost, and can its results be trusted?
Loading tests…
Brands you can access
Example Coffee Co.When each source last synced
Amazon fees are estimates until orders settle (about 14 days).
Amazon Ads attributes purchases up to 7 days after a click, so the last week can still rise.
Amazon Attribution keeps adding purchases to each day for 14 days.
Meta revises the last few days of results as delayed conversions arrive.
What should the next holdout test be, what will it cost, and can its results be trusted?
Planned on Amazon orders through Oct 24: about 31 a day at $46.80 each.
Meta → Amazon pays for itself at a 7.5% lift in Amazon orders ($38.00 a day of spend, first orders only). Holding out 40% of orders for 8 weeks can detect 7.2% or more, enough to tell whether Meta pays back.
Missed orders use this brand's own estimate of what Meta → Amazon adds, with its range; no other brand's data is used. A negative net cost means pausing saves more than it loses.
Earliest start that avoids the serious ones: Dec 28.
| Holdout | Weeks | Detects | Orders missed | Contribution missed | Meta spend saved | Net cost |
|---|---|---|---|---|---|---|
| 30% | 4 | 10.9% | 15 | $255 | $319 | −$63.71 |
| 30% | 6 | 8.9% | 22 | $383 | $479 | −$95.57 |
| 30% | 8 |
Each result is computed from the locked design and added to the ledger. Earlier results are never edited.
Each prediction is locked, with a hash, before its result is known, then scored against what happened.
Track record: no predictions scored yet; the first is due Nov 1.
Every test design, result and prediction is chained by hash, so changing or deleting any earlier record would show.
4 entries, latest Oct 1. Each record still matches the hash recorded for it.
| 7.7% |
| 29 |
| $511 |
| $638 |
| −$127 |
| 40% | 4 | 10.2% | 20 | $341 | $426 | −$84.95 |
| 40% | 6 | 8.3% | 29 | $511 | $638 | −$127 |
| 40%recommended | 8 | 7.2% | 39 | $681 | $851 | −$170 |
| 50% | 4 | 10.0% | 24 | $426 | $532 | −$106 |
| 50% | 6 | 8.2% | 37 | $639 | $798 | −$159 |
| 50% | 8 | 7.1% | 49 | $852 | $1,064 | −$212 |
“Detects” is the smallest lift the test finds 80% of the time at 95% confidence, checked against placebo tests on this brand's past orders. Greyed out: can't see the 7.5% target.