How to test whether your ads actually cause sales

What incrementality means

Ad platforms report conversions: sales that came after someone clicked or saw an ad. Incrementality asks a harder question. How many of those sales happened because of the ad, and how many would have happened anyway?

The difference can be large. A 2015 field experiment at eBay, published in Econometrica, found that returns from paid search were a fraction of non-experimental estimates, and that ads on eBay's own brand name had no measurable short-term benefit. People searching for eBay were going to find eBay with or without the ad. Your branded search and retargeting ads may work the same way.

The basic recipe: a holdout group

Every incrementality test follows the same pattern, borrowed from medical trials:

• Split your audience into two groups that are as alike as possible.
• Show ads to one group (the test group) and not to the other (the holdout or control group).
• Run the test long enough for buyers to make decisions.
• Compare real sales in each group, adjusting for any difference in group size.
• The extra sales in the test group are your incremental sales. Divide incremental revenue by ad spend to get incremental return on ad spend, or iROAS.

Three tests a small business can run

Platform lift studies. Google Ads offers Conversion Lift, which randomly withholds ads from some users. Its setup guide lets you set a holdback of 1% to 50% and reports incremental conversions, incremental cost per action and iROAS. Meta offers lift studies too, though availability depends on your account and spend. These are the most precise option because the platform controls who sees what, but the platform is still grading its own ads, and small budgets may not produce a clear answer.

Geo tests. Pick matched regions, such as similar cities or states, and run ads in some but not others. Then compare sales by region using your own records. This works well if you sell across many areas and can tell where each order came from.

Pause tests. Turn a campaign off for a few weeks and watch total sales. It is the crudest test, since seasons and news also move sales, but it costs nothing. Branded search is a good first candidate because it so often claims sales that arrive anyway.

Getting a result you can trust

Small tests are noisy. A few habits make results more reliable:

Measure with your own sales data, not the platform's conversion count. The whole point is to step outside the platform's view of itself. If you use Stripe or a store platform, payments by date and region are usually enough.

• Run long enough. Google recommends at least 14 days and notes up to a 17% drop in measured lift for studies with a long conversion delay that run for less than 14 days.
• Change one thing at a time. Don't launch a sale or a new product in the middle of a test.
• Write down what you expect before you start, so you can't move the goalposts afterward.
• Repeat. One test is a data point. Two or three pointing the same way is a finding.

Reading the answer

Compare your incremental return with your break-even point, the return you need just to cover product costs. If your margin is 40%, you need $2.50 back for every $1 of ad spend to break even. Our break-even ROAS guide shows the math.

A platform may report a strong return while the test shows a modest one. That doesn't always mean you should cut the campaign. It means you now know its real contribution and can size the budget to match. For channels that are too small or too slow to test directly, marketing mix modeling is the next step.

Matrix ROI, from Smith App Studio, keeps verified daily sales beside each platform's claims, which is the comparison you need before, during and after a test. The live demo shows what that looks like.