Campaign Incrementality: Did Advertising Create New Customers?

Business question

The growth team needs to know whether paid advertising creates new customers. Reported conversions alone cannot answer that question. Some people would have converted even without seeing the campaign.

Why it matters

If the business counts every conversion touched by an ad as campaign value, it can spend money on customers who did not need the ad. The right measure is the extra conversions caused by assigning the campaign.

Decision brief

• Recommendation: run the live campaign with a randomized holdout; scale only when incremental CPA is below contribution value.
• Evidence: control conversion was 0.194% and treatment conversion 0.309%: absolute lift +0.115 percentage points (95% CI +0.108 to +0.122pp), or about +59.3% relative.
• Potential value: about 115 extra conversions per 100,000 assigned users in this benchmark, before economics.
• Evidence strength: High for the released randomized comparison; low for transporting the rate to a current campaign.
• Cost / resource requirement: Current ad cost and contribution cannot be estimated from this dataset; reserve a randomized holdout and measurement capacity. Break-even CPA is contribution value per incremental conversion, not total attributed conversions.
• Main risk: the file has no ad cost, margin, LTV, or current-campaign population.
• Cost of inaction: Cannot be estimated from this dataset; the risk is paying for conversions that would have happened anyway.
• Success / stop rule: Continue only if incremental CPA stays below contribution value; stop or redesign if the holdout shows no useful lift or the guardrail fails.
• Next action: pre-register the holdout, minimum useful effect, sample size, and stopping rule.

My role

I owned the analysis design, streaming validation, intention-to-treat calculation, chart, and business recommendation for this portfolio case. I did not run the campaign or observe a live commercial outcome. I would hand the decision owner a holdout design, break-even CPA rule, and segment hypotheses to retest. Analysis code, SQL, and validation are available for review.

Data used

I used the Criteo AI Lab uplift dataset. It combines results from randomized advertising tests. Each row represents one user record and shows whether the user was assigned to advertising, visited, or converted.

The validated release contains 13,979,592 rows. The user features are anonymized, so they can show different response patterns but cannot be turned into real customer groups. Criteo also warns that the file combines experiments and was sampled in a way that prevents the original campaign effect from being recovered. It is a benchmark for learning, not a forecast for a current campaign.

Approach

1. Check that the advertising and control groups are present in the expected proportions.
2. Compare conversion and visit rates using the group each user was assigned to.
3. Measure the extra response in the advertising group.
4. Explore whether response differs across anonymized user features.
5. Define the information a real campaign needs before the business spends more.

Key findings

The advertising group converted more often

The advertising group converted at 0.309%. The control group converted at 0.194%. The difference was 0.115 percentage points, with a 95% confidence range from 0.108 to 0.122 percentage points. This is the intention-to-treat difference within the released benchmark; it is not a forecast for a current campaign.

Chart takeaway: Criteo randomized advertising benchmark: assigned advertising increased conversion by 0.115 percentage points, with the 95% confidence range shown

The same pattern appeared in visits. The difference was 1.034 percentage points, with a 95% confidence range from 1.006 to 1.063 percentage points.

Business meaning: the released benchmark shows a clear difference between assigning users to advertising and holding them out. It does not tell us that a current campaign will produce the same result.

The headline rate is not the same as profitable growth

The benchmark difference is about 115 extra conversions per 100,000 assigned users in this released sample. That number explains the experiment. It cannot calculate current campaign profit because the file has no ad cost, conversion value, margin, or later customer value.

Business meaning: statistical confidence tells us that the benchmark difference is unlikely to be random noise. It does not tell us whether the campaign earns more than it costs.

Recommendation

What: Run the current campaign with a randomized holdout and measure extra conversions, not only attributed conversions.

Where / who: Start with an audience definition the business can explain and reach. Use anonymized feature bands only as hypotheses for the next test.

Why: The benchmark shows that assignment can change conversion, but response is not even across users.

Risk: The benchmark's effect cannot be transported to a current campaign, and targeting the wrong group could reduce reach or increase acquisition cost.

Next test: Compare broad targeting with uplift-based targeting. Set the minimum useful effect, sample size, stopping rule, and multiple-segment rule before launch.

What internal data would improve the decision

I would add ad spend, contribution margin, conversion value, customer identity, channel, frequency, and later retention. Then I would calculate incremental CPA and incremental return by a business-defined audience. The campaign would scale only when incremental CPA stays below contribution value and customer-experience guardrails remain healthy.

Key takeaway

More reported conversions do not automatically mean profitable advertising. The important decision is to measure the conversions created by the campaign, then spend more only where a current holdout shows that the extra customers are worth the cost.

Technical appendix

The primary analysis uses intention-to-treat. In plain language, each user stays in the group they were assigned to, even if the ad was not actually shown. This protects the fairness of the randomized comparison. Analysis code, SQL, and validation are available for review.

Exploratory subgroup check

I divided the complete f0 feature into four equal-sized bands as an exploratory check. The second band had the largest difference, 0.386 percentage points; the other bands were 0.038, 0.019, and 0.010 percentage points. These are not real customer personas. The feature has no business label and the split was chosen for exploration, so a new campaign should retest any targeting hypothesis in a holdout.