Design a test that can change a decision

A small incrementality test is useful only when its design, uncertainty and commercial threshold are agreed before the team sees a favourable result.

A marketing team can run an experiment, obtain a positive number and still learn too little to make its next investment. The problem often begins before launch: nobody agreed how much improvement would justify the cost, or whether the test could detect that improvement.

Start with a decision sentence. For example: “We will expand this campaign if the evidence supports enough additional contribution to cover its additional cost, within our cash and service constraints.”

That sentence is less exciting than promising a definitive lift percentage. It is far more useful when the result is uncertain.

Put economics before sample size

Consider a hypothetical campaign that costs €6,000 and generates €120 contribution per additional completed order, before campaign cost. It needs 50 additional orders merely to cover that spend. A management requirement for further contribution would raise the threshold.

The question is therefore not simply whether sales increase. It is whether the increase can plausibly exceed the commercial requirement. Contribution must include the variable costs relevant to this decision, including likely returns or service costs where appropriate.

Do this calculation before choosing a treatment group. Otherwise the organisation may celebrate a statistically detectable effect that is too small to pay for the campaign.

Choose a credible comparison

A randomised holdout assigns eligible people or accounts to groups before exposure. One group receives the treatment and the other receives the agreed comparison. Randomisation helps prevent the team from giving the campaign only to customers most likely to buy anyway.

The assignment unit needs to match how the treatment spreads. Randomising individual employees is unhelpful if colleagues in the same buying account share the offer. Assigning accounts may be more credible, although it reduces the number of independent units.

Geographic designs can be useful when person-level assignment is impractical. GeoLift’s methodology describes pre-test analysis to assess detectable effects and candidate test designs. A geographic comparison still needs scrutiny for market differences, spillover and concurrent local changes.

Do not select a control region solely because its sales total looks similar. Its trajectory, seasonality, distribution and exposure to other activity may differ in ways that matter more than size.

Check whether the test can answer the question

Give an analyst historical outcome data and the minimum commercially relevant effect. Ask for a prospective assessment of sample size, duration and uncertainty under the proposed design.

Advertising effects can be difficult to isolate against ordinary purchasing variation. Lewis and Rao’s field-experiment research is a useful warning that even substantial experiments can produce imprecise return estimates.

If the available sample cannot support the decision, change the plan before spending. You might lengthen the observation period, combine eligible cohorts under a consistent design, test a larger meaningful intervention, or choose a narrower operational question.

An underpowered sales test may still reveal delivery failures or customer confusion. Report those as operational findings, rather than presenting the absence of a clear sales effect as proof of zero value.

Freeze the rules that invite hindsight

Record the primary outcome, eligibility criteria, assignment method, exclusions, analysis approach and planned end date. Specify how refunds, delayed purchases and duplicate accounts will be handled.

Choose one main outcome that represents the decision. Additional metrics can explain what happened, but searching dozens of outcomes for a favourable result changes the meaning of the evidence.

Agree how the team will monitor harm. A campaign that causes complaints or overwhelms fulfilment may need to stop for operational reasons. That is different from repeatedly checking results and ending the experiment at the first attractive number.

Protect the comparison during execution

Give sales and support a clear explanation of the experiment. A well-intentioned salesperson who sends the offer to holdout accounts can weaken the comparison without realising it.

Log exceptional events: stock shortages, outages, competitor promotions, regional disruption and changes in sales coverage. These records help interpret the result and reveal whether the planned treatment was actually delivered.

Avoid removing inconvenient participants after seeing their outcomes. Excluding unsuccessful accounts because they “did not engage properly” can make the treatment group look stronger by construction. Any additional analysis should disclose the change and retain the planned comparison.

Report the range and its consequence

Return to the €6,000 example. Suppose the estimated additional orders are 60, but the uncertainty interval spans a modest loss through a large gain. The point estimate clears the 50-order break-even threshold; the evidence may still leave the investment decision unresolved.

The report should show that uncertainty in commercial terms, using the same contribution assumptions as the original decision. Avoid converting an uncertain estimate into a confident annual forecast through multiplication.

Management could approve a limited extension, gather more evidence or stop because the downside exceeds its tolerance. These are judgement calls, not statistical facts. Record why the chosen action fits the business’s constraints.

Leave a reusable record

Keep the test brief, implementation exceptions, result and resulting decision together. Future teams should be able to distinguish an ineffective intervention from an inconclusive study or a failed implementation.

Also record the scope of the finding. A campaign tested on existing customers in one market does not automatically establish performance among new prospects elsewhere.

A good experiment makes the next decision more disciplined, even when it does not produce a satisfying headline. Its value lies in reducing a consequential uncertainty enough to act, and showing honestly what remains unknown.