In short
An A/B test of two ads compares how often a click becomes an order. Divide each ad's orders by its clicks. In the example, ad A turns 2% of clicks into orders and ad B turns 3%. The gap is 1% of clicks. A two-sided test at 95% confidence asks whether a gap that large would often show up when the two ads truly convert the same. Here it would not, so ad B won on order rate, and it also costs less per order ($10.00 against $15.00). The test says nothing about whether either ad caused a sale that would not have happened anyway. That takes a holdout, not a creative test.
What is an A/B test for ads?
An A/B test runs two ads, or two creatives, and compares them on the same outcome. For a store, the outcome that pays the bills is an order, not a click and not a thumbs-up. The order rate is orders divided by clicks. The ad with the higher rate won the test, but only if the gap is bigger than the bounce you get from chance.
Chance is the whole problem with small tests. Twenty orders against twenty-five, from a thousand clicks each, looks like a win in the ads manager. It usually isn't. Swap a few buyers from one ad to the other and the lead flips. A statistical significance calculator draws the line: it tells you when the lead is large enough, given how many clicks you have, that it would rarely appear if the two ads were equal.
This one is built for ads. General significance calculators ask for abstract sample sizes. Website tools ask for visitors and conversions on a page. An ad report gives you spend, clicks and orders, and you need the order rate and the cost per order from those three numbers.
How to calculate whether the gap is real
order rate = orders ÷ clicks cost per order = spend ÷ orders gap = order rate B − order rate A pooled rate = (orders A + orders B) ÷ (clicks A + clicks B) z = gap ÷ standard error of the pooled rate
- Pull spend, link clicks and orders for each ad over the same dates. In Meta Ads Manager or Amazon Ads, use one row per ad.
- Divide orders by clicks for each ad. That is the order rate the test compares.
- Run a two-proportion z-test, two-sided, at 95% confidence. The pooled rate is what both ads would share if neither were better. The standard error comes from that rate and the two click counts.
- Call the gap real only when the p-value is under 0.05 and each expected cell is at least 5. Expected orders for an ad are its clicks times the pooled rate. Expected non-orders are the rest. Below 5, the normal curve is a poor fit and the result stays "not enough orders."
Spend is not in the test. It answers a different question: what each order cost. An ad can win the order rate and still be the expensive one if its clicks cost more.
Worked example
Two made-up ads, same spend and same clicks, which is the calculator's example. Only the orders differ.
- Ad A: $2,400 spend, 8,000 clicks, 160 orders. Order rate = 160 ÷ 8,000 = 2%. Cost per order = $15.00.
- Ad B: $2,400 spend, 8,000 clicks, 240 orders. Order rate = 3%. Cost per order = $10.00.
- The gap is 1% of clicks, which is a 50% higher order rate for ad B. The p-value is under 0.001, and both ads have far more than 5 expected orders, so the gap clears 95% confidence. Ad B won.
A quieter pair does not. One thousand clicks and 20 orders against 25 is a gap of 0.5%, from 2% to 2.5%. The 95% range on that gap still includes no difference at all, so the lead is noise. The ads manager will still highlight the higher number.
What's a result you can act on?
Three outcomes, and only one of them is a decision.
- A winner. The order-rate gap clears 95% confidence, and there are enough orders for the test to apply. Keep the winner if its cost per order is a price you can pay. Check that price against your margin with the break-even ROAS calculator.
- Too close to call. One ad is ahead, but a gap this size shows up often when the ads are equal. Do not kill the other ad yet. More clicks shrink the noise.
- Not enough orders. The rates can look far apart, and a raw z-score can look decisive, when each ad has only a handful of orders. Wait. A 0-for-20 against an 8-for-20 is not a finished test.
There is no universal order rate that counts as good. A 3% rate that costs more than the order is worth is a loss. A 1% rate on cheap clicks can be the better ad. Read the rate and the cost per order together.
Common mistakes
- Stopping when the ads manager draws a star. Platform badges use their own rules and their own windows. Check the order rate and the p-value on the same dates yourself.
- Comparing different weeks. Ad A in a sale week against ad B in a quiet week is not a test. The calendar changed, not the creative.
- Mixing clicks and orders from different columns. Link clicks with purchases. Unique outbound clicks with a view-through conversion will invent a gap.
- Calling the cheaper cost per order the winner when the rate test says otherwise. Cost per order moves when the click price moves. The significance test is on the order rate. Use cost per order to decide whether the winner is affordable.
- Peeking every morning and stopping at the first green day. A gap that is noise on Monday can look real by Wednesday and be noise again on Friday. Decide the click count, or the end date, before you look.
- Treating the winner as incremental. Both ads can be collecting people who would have bought anyway. Beating the other creative is not the same as beating a world with no ads.
How this connects to true incrementality
This calculator answers which ad got more orders per click, and whether you have enough clicks to trust that. It does not answer how many of those orders would have happened with the ads off. Two creatives can split the same demand. The winner of that split still may not be causing the sale.
A geo holdout pauses ads in some regions and leaves them on in others, then compares Amazon sales. That is the test for what the spend caused. Size it with the holdout test calculator before you turn anything off, and read what incrementality means if the two ideas are getting mixed together. The incremental ROAS calculator turns a finished holdout into a return number. Use this page to pick a creative. Use a holdout to decide whether the ads earn their spend.
Frequently asked questions
How do you know if an A/B test on ads is significant?
Divide each ad's orders by its clicks, then run a two-sided two-proportion z-test at 95% confidence. The gap is significant when the p-value is under 0.05 and each ad has enough orders for that test, at least 5 expected orders and 5 expected non-orders under the pooled rate.
What numbers do you need from the ad report?
Spend, link clicks and orders for each ad, over the same dates. Spend gives the cost per order. Clicks and orders give the order rate the test compares.
Is a higher order rate always the winner?
Only if the gap clears the noise. A small lead on a small sample is usually chance. And a real lead can still be a bad buy if the cost per order is higher than the order is worth.
Why not just use a general statistical significance calculator?
Those ask for sample sizes and conversions in the abstract. An ad test starts from spend, clicks and orders, and the decision needs the cost per order next to the order rate. The math is the same two-proportion test.
Does a winning ad mean the ads caused the sales?
No. The test says one ad beat the other on orders per click. It does not say those orders would have disappeared if you had run neither ad. That question needs a holdout test, with ads on in some regions and off in others.