Projects / Which customers to send an email offer to / Technical document
Which customers to send an email offer to: technical document
Five tests, with rules set before the analysis
In the notebook, the champion is the current policy and the challenger is the new policy.
| Test | Measure | Result | Rule | Pass |
|---|---|---|---|---|
| 1. Validity | Largest absolute standardized difference; sample-ratio p-value | 0.014; 0.90 | Less than 0.05; more than 0.01 | Yes |
| 2. Effect | Welch 95% CI of the spend effect; customers in each group / customers for 80% power | [$0.49, $1.05]; 21,306 / 5,988 | Excludes 0; at least the customers for 80% power | Yes |
| 3. Robustness | Bootstrap 95% CI; largest change of the CUPED and the regression effect | [$0.48, $1.06]; 0.3% | Excludes 0; within 10% | Yes |
| 4. Business value | Profit for 100,000 customers, with its 95% CI | $13,095 [$4,554, $21,635] | More than 0; lower end more than 0 | Yes |
| 5. Targeting | Best ML model against rank by history (Qini area); 90% interval of the gain of the top 25% at $0.10 | -123%, 0 of 10; [-$11,245, $11] | +20% or more and 9 of 10; interval above 0 (then targeting replaces send to all) | Yes |
Results
Baseline: what does the control group do without an email?
The groups are the same before the email. Thus the control group shows what all customers do without an email. Each effect in this notebook is a difference against this baseline. The baseline also shows why spend is a noisy metric.
The code measures four values for the control group. They are the visit rate, the conversion rate, the mean spend and the share with $0 spend.
| Customers | Visit | Conversion | Mean spend | Share with $0 spend | Mean spend of a buyer |
|---|---|---|---|---|---|
| 21,306 | 10.6% | 0.57% | $0.65 | 99.4% | $114 |
How to read: each bar is the share of the 21,306 control customers who reach a stage in the two weeks of the test. The control group got no email.
So what: 99.4% of the customers without an email spend $0, and the mean spend is only $0.65. The few buyers spend $114 each, so a small number of orders sets the mean. The standard deviation of the spend is about 18 times its mean. Thus the spend effect of an email has a wide confidence interval, and the test needs many customers (Step 3c).
Business note. The test file has 64,000 valid customers, no empty cells and one consistent outcome for each customer. The random assignment works: the largest standardized difference is 0.014, and the sample ratio p-value is 0.90. Without an email, a campaign of 100,000 customers gives about $65,279 of spend in two weeks. Each email must add to this baseline more than its cost. Step 3 now fixes the rules of the test and checks that the test is large enough to find such an effect.
Branch 4: business value
The question: does the recommended policy make money, also at the low end of the effect? A positive effect is not sufficient. Each email has a contact cost, so the effect must be more than the breakeven effect. The rule uses the base margin of 30% (as-01) and the base contact cost of $0.10 (as-02), for one campaign of 100,000 customers (as-04).
| Check | Result | Rule |
|---|---|---|
| Incremental profit for 100,000 customers | $13,095 | More than 0 |
| 95% CI of the profit | [$4,554, $21,635] | Lower end more than 0 |
| Breakeven effect (contact cost / margin) | $0.33 | |
| Lower end of the 95% CI of the effect | $0.49 | More than the breakeven effect |
At a margin of 30% and a contact cost of $0.10, the men’s email to all gives $13,095 for 100,000 customers. The 95% CI of the profit is [$4,554, $21,635]. The lower end of the effect, $0.49, is above the breakeven effect of $0.33. The policy passes branch 4. The margin and the contact cost are assumptions, so Step 7c changes them.
Rollout
The test result covers two weeks (as-03). After the launch, the effect can become smaller, and customers can unsubscribe. A holdout is a random group of customers that gets no email after the launch. It shows the effect of the email over time, with the same causal logic as the test.
Which share of customers do we keep as a holdout after the launch?
Why this question matters. Each customer in the holdout gets no email, so the holdout costs profit. A holdout that is too small cannot show if the email still pays for itself. Then a fall of the effect stays hidden, and the campaign can lose money for months.
Ways to do it. The size of the holdout is one number. Thus the options are points on one scale: none, small, medium and large.
| Option | Advantage | Disadvantage |
|---|---|---|
| A. No holdout | No lost profit. | We cannot see if the effect becomes smaller or if customers unsubscribe. We screen it out. |
| B. 5% holdout | Small lost profit. | Less power. |
| C. 10% holdout | Good power to show that the email still pays for itself. | More lost profit. |
| D. 20% holdout | Most power. | Twice the lost profit of option C. |
Evidence. The holdout must show that the email still pays for itself. Thus the test asks: is the effect more than the breakeven effect of $0.33? The table gives the power of this test for one campaign of 100,000 customers, with a one-sided test at 5%. The lost profit is the profit that the holdout customers do not give.
| Option | Holdout customers | Half-width of the 95% CI | Power if the effect stays at the test result | Power if the effect falls by 20% | Lost profit for each campaign |
|---|---|---|---|---|---|
| B. 5% | 5,000 | $0.43 | 64% | 36% | $655 |
| C. 10% | 10,000 | $0.31 | 87% | 56% | $1,309 |
| D. 20% | 20,000 | $0.23 | 98% | 77% | $2,619 |
Decision. Option C, a 10% holdout.
Why. If the effect stays at the test result, a 10% holdout shows that the email pays for itself with 87% power. A 5% holdout gives only 64%. A 20% holdout gives 98%, but it doubles the lost profit to $2,619. The lost profit of option C is $1,309 for each campaign. This is a small price to know if the email still makes money.
If the effect falls by 20%, the power of a 10% holdout is only 56% in one campaign. Thus the kill criteria below use two campaigns, not one.
What would change the decision. If a campaign has more than 200,000 customers, use 5%, because the holdout keeps the same number of customers. If the campaigns repeat each month, combine the holdouts of two or three campaigns.
Metrics and kill criteria
The holdout gives the primary metric for each campaign. The table gives the metrics, the limits and the action for each limit.
| Metric | Type | Target or limit | Action |
|---|---|---|---|
| Incremental spend for each customer (email group minus holdout) | Primary | More than $0.33 (the breakeven effect at $0.10) | If it is less for two campaigns, stop “send to all”. Send only to the top 25% by history. |
| Conversion rate | Secondary | More than the holdout | If the spend effect falls but conversion stays, examine the order size. |
| Unsubscribe rate | Guardrail | Less than 0.5 points more than the holdout. Confirm with Marketing. | If it is more, update the contact cost (as-02). Then use the targeting rule of 7a again. |
| Complaints | Guardrail | No increase | Stop the campaign. |
Business note. At the base contact cost, send the men’s email to all customers. It gives $13,095 for each campaign of 100,000 customers (range $4,554 to $21,635). If the full contact cost is $0.25 or more, send only to the top 25% by history. Keep a 10% holdout. It costs $1,309 for each campaign.
If the effect stays below $0.33 for two campaigns, stop “send to all”. Part 2 shows how other teams can use the same method.
What a reviewer should challenge
These limits can change the result most. Each one has a control.
| Limitation | Effect on the result | Control, or what to do next |
|---|---|---|
| The contact cost of $0.10 is an assumption (as-02). The data does not give unsubscribes. | The contact cost decides if a targeting rule beats “send to all”. | The profit chart of Step 7a and the sensitivity check of Step 7c. Measure the unsubscribe rate as a guardrail and update the cost. |
| The margin of 30% is an assumption (as-01). | The profit changes with the margin. At a low margin and a high cost, the top 25% is the better rule. | The sensitivity check of Step 7c uses 20% and 40%. Finance confirms the margin. |
| The effect window is two weeks (as-03). | The long-term effect is not known. It can be larger or smaller. | Keep the 10% holdout for three months (Step 7d). |
| Less than 1% of customers convert. | Each segment effect has a wide interval. The ML uplift models fit noise. | Rank by history. Repeat the comparison with each new test, with the model rule of Step 3a. |
| The gain of the top 25% at the base cost is near the limit of the policy rule. | A small change of the data or of the cost can change the targeting rule. | The decision uses the 90% interval, not the point estimate. Confirm the cost before a high-cost campaign. |
| I examined the segment effects before I selected the uplift method, on the same data. | The gain of a targeting rule can be too high. | The cross-fitted uplift reduces this risk. A new test confirms the rule before a high-cost campaign. |
| The data is from one retailer in 2008 (as-05). | The size of each effect is different for a card offer. | The method stays the same. Repeat the analysis on the bank data. |
Other limits have a small effect. The data source cut the spend at its largest value, so a few large orders can be larger in reality. The test has only one email version for each group, so it cannot show the best combination of versions.
Next steps
- Business case deck (6–8 slides): the men’s email to all customers, the breakeven contact cost and the 10% holdout.
- Confirm the contact cost (as-02) and the margin (as-01) with Marketing and Finance.
- Design the pricing test for grades A to C with case 01 (Part 2: Pricing). Add it to the pricing one-pager.
- Design the collections test (debt sale or in-house collection) with case 01 (Part 2: Recoveries). Add it to the collections one-pager.
What each tool does in this case
| R | Runs the whole analysis in one notebook in base R. |
| Statistics | Designs and reads the experiment: power and sample size, Welch t-test, bootstrap, CUPED and the Holm correction. |
| Machine learning | Tests two uplift models (T-learner and uplift tree). They do not beat the ranking by purchase history, so the recommendation does not use them. |