Projects / Which customers to send an email offer to

Which customers to send an email offer to

Send the email to all customers and keep a 10% holdout. It earns $13k per campaign, with no targeting model.

Self-built case study. Hillstrom public email test data. Not client or employer work.

R
Board deckFor leadershipTechnical documentMethod, results, notebook and files

The men's email to all customers earns $13k for each campaign of 100,000 customers

Incremental spend

+$0.77

for each customer, in 2 weeks

Incremental profit

$13,095

for each 100,000 customers

Holdout

10%

confirms the effect in each campaign

Decision: send the men's email to all customers, with a 10% holdout.

Source: Hillstrom email test, 64,000 customers; analysis. Public data; illustrative results.

A randomized test gives a clean answer: the three groups differ only by the email

Men's email

21,306

customers, chosen at random

Women's email

21,306

customers, chosen at random

No email

21,306

customers: the control group

Complication: each email has a contact cost, and most customers buy nothing in 2 weeks, so the spend data is noisy.

Spend, visits and conversions were measured for 2 weeks after the email. Source: Hillstrom email test, 64,000 customers; analysis. Public data; illustrative results.

The men's email adds $0.77 of spend for each customer, above the $0.33 break-even

Spend effect of the men's email against no email

$ for each customer, 2 weeks after the email

Measured effect
+$0.77
Low end of the 95% interval
+$0.49
Break-even effect
$0.33

Why the result holds

  • The groups match before the email: the randomization check passes
  • 21,306 customers in each group; 5,988 are enough for 80% power
  • A bootstrap gives the same interval

Break-even = contact cost of $0.10 divided by a 30% margin. Source: Hillstrom email test, 64,000 customers; analysis. Public data; illustrative results.

At $0.10 an email, the campaign earns $13,095 and pays at any cost below $0.23

Incremental profit for each campaign of 100,000 customers

$, margin on incremental spend less the contact cost

$13,095

Men's email

$2,733

Women's email

$0.13

profit for each customer emailed

How the profit adds up

  • 30% margin on $0.77 of spend: $0.23
  • Less the contact cost: $0.10
  • 95% interval for the campaign: $4,554 to $21,635

Margin and contact cost are assumptions. Women's email = $10,362 less than the men's email. Source: Hillstrom email test, 64,000 customers; analysis. Public data; illustrative results.

A targeting model adds nothing at this cost; a shorter list pays only at $0.25 or more

Contact cost below $0.23

Send to all

No rule and no model earns more. Today's assumption is $0.10.

Contact cost of $0.25 or more

Top 25% only

Rank customers by purchase history and send to the top quarter.

Uplift models

Not used

Two models did not beat the simple ranking by purchase history.

The full contact cost decides the rule, so Marketing must confirm it before the launch.

Source: Hillstrom email test, 64,000 customers; analysis. Public data; illustrative results.

Send the men's email to all customers, and keep a 10% holdout

Actions

  1. Send the men's email to all customers$13,095 for each 100,000 customers
  2. Keep a 10% holdoutcosts $1,309 a campaign, 87% power
  3. Confirm the full contact costbefore the launch

Risks and controls

The full contact cost is higher than $0.10: above $0.23, an email to all loses moneyAt $0.25 or more, send only to the top 25% by purchase history
The effect falls after the launchIf incremental spend stays below $0.33 for two campaigns, switch to the top 25%

Source: analysis. Public data; illustrative results.

1 / 6

Campaign profit splits into audience, margin and contact cost

  • Incremental profit of a campaign$13,095 for 100,000 customers
    • Customers who get the emailThe lever: all customers, or a targeted list

      Does a targeted list beat an email to all customers?

      No. At $0.10 an email, no rule and no uplift model beats send-to-all. The top 25% by purchase history pays only at $0.25 or more.

    • Profit for each customer emailed
      • Incremental margin
        • Incremental spendMeasured by the randomized test

          Does the email raise spend?

          Yes. +$0.77 for each customer (95% CI $0.49 to $1.05).

        • Margin on spendFixed: assumed 30%
      • Contact costThe limit: assumed $0.10 an email

        Does the email pay at the assumed cost?

        Yes. It pays at any cost below $0.23 an email.

Lever what the recommendation changes Limit what must not get worse Fixed outside the decision