Projects / Which card transactions to approve, review or decline / Technical document
Which card transactions to approve, review or decline: technical document
Five tests, with rules set before the analysis
In the notebook, the champion is the current policy and the challenger is the new policy.
| Test | Measure | Result | Rule | Pass |
|---|---|---|---|---|
| 1. Order of risk | PR-AUC and dollar recall in the top 2%, new policy against current rules | 0.486 against 0.269; 26.5% against 17.0% | Both above the current rules; gain of 10% or more | Yes |
| 2. Level | Mean score minus actual fraud rate, months 5–6 | -0.06 points | Less than 0.5 points | Yes |
| 3. Stability | PSI of the score; share of the PR-AUC that the score keeps in later months | 0.008; 85% | PSI less than 0.1; 80% or more | Yes |
| 4. Business value | Total cost each month, new policy against current rules; decline rate of good transactions | $172,612 against $299,432; 2.46% | Lower total cost; 3% or less | Yes |
| 5. Explainability | Share of declines that readable rules explain | 0% | 50% or more | No |
Results
Baseline: what does fraud cost if the merchant approves all transactions?
The data is now clean. Before we build a score, we must know the cost that the score must decrease. The simplest policy approves all transactions. It has no false declines and no reviews, so its total cost is the fraud loss.
The fraud loss of a transaction is its amount if it is fraud (as-01: the business pays 100% of the chargeback). The table gives the cost for each 30 days, so that months with a different number of days compare correctly.
| Transactions | Days | Fraud rate | Amount ($) | Fraud amount ($) | Fraud loss, approve-all ($ each 30 days) | Fraud share of $ | |
|---|---|---|---|---|---|---|---|
| month | |||||||
| 1 | 130,968 | 29 | 2.48% | 16,756,862 | 439,590 | 454,748 | 2.62% |
| 2 | 89,838 | 30 | 4.04% | 11,990,567 | 496,493 | 496,493 | 4.14% |
| 3 | 91,768 | 30 | 4.03% | 12,773,159 | 515,459 | 515,459 | 4.04% |
| 4 | 98,027 | 30 | 3.93% | 13,719,550 | 643,237 | 643,237 | 4.69% |
| 5 | 85,303 | 30 | 3.47% | 11,520,251 | 493,821 | 493,821 | 4.29% |
| 6 | 94,636 | 33 | 3.47% | 12,978,561 | 495,244 | 450,222 | 3.82% |
How to read: the left panel gives the fraud loss of each month if the merchant approves all transactions, for each 30 days. The right panel gives the share of fraud transactions in each month; the dashed line is the rate of all months.
So what: the fraud loss stays between $450,222 and $643,237 each 30 days, and the fraud rate stays between 2.48% and 4.04%. Fraud is a constant cost, not one bad month. Month 4 has the highest loss. On the test months 5–6, approve-all costs $470,983 each 30 days.
Business note. The data has 590,540 clean transactions, about 3,245 each day, with a fraud rate of 3.50%. Fraud has 3.87% of the dollars. Approve-all costs about $508,326 each 30 days: this is the ceiling of the fraud loss. Two facts limit what a score can promise: the customer key is a proxy, and the label of a customer can follow a later chargeback. Step 3 now splits the data by time, so that the test looks like production.
Branch 4: business value
The question: at the same review capacity, does the challenger decrease the total cost against the champion? This is the main business test. Both policies have the same three tiers and the same capacity of 120 alerts each day (as-03). Only the score and the value that ranks the transactions change:
- Champion (as-06). The rules of a decision tree with 3 levels, fit on month 1. The code sets its decline and review thresholds on month 4, with the method of Step 5c.
- Challenger. The GBM score after the recalibration, and the expected cost in dollars (Step 5d).
The guardrail is the decline rate of good transactions. A decline of a good customer has a cost in the model (as-02), but it also hurts the brand. Thus the rule adds a hard limit of 3% (as-08). Marketing must confirm this limit before the test.
| Fraud loss ($/month) | False decline cost ($/month) | Review friction ($/month) | Total cost ($/month) | Fraud $ stopped | Decline rate of good transactions | Alerts each day (mean) | Days over capacity | |
|---|---|---|---|---|---|---|---|---|
| Policy | ||||||||
| Champion: static rules, three tiers | 290,176 | 3,013 | 6,243 | 299,432 | 38.4% | 0.28% | 119 | 48% |
| Challenger: GBM, expected cost, three tiers | 135,768 | 31,933 | 4,911 | 172,612 | 71.2% | 2.46% | 90 | 5% |
On the test months, the challenger decreases the total cost by $126,820 each month (42%) against the champion, with the same review capacity. It stops 71.2% of the fraud dollars. The champion stops 38.4%.
The challenger declines more good transactions: 2.46% against 0.28% for the champion. This rate is inside the guardrail limit of 3%. The challenger passes branch 4.
The champion approves almost all transactions and accepts the fraud loss. With the cost of a false decline in as-02, this choice is expensive. The champion also has a capacity problem. Its rule segments change in size from day to day. Thus its alerts are more than the capacity on 48% of the test days, against 5% for the challenger.
The chart splits the total cost of each test month into its three parts.
| Fraud loss ($/month) | False decline cost ($/month) | Review friction ($/month) | Total | ||
|---|---|---|---|---|---|
| Month | Policy | ||||
| 5 | Champion | 296,354 | 2,622 | 5,872 | 304,848 |
| Challenger | 141,778 | 27,463 | 4,674 | 173,914 | |
| 6 | Champion | 284,560 | 3,368 | 6,580 | 294,508 |
| Challenger | 130,305 | 35,996 | 5,127 | 171,428 |
How to read: each bar is one policy in one test month. The red part is the fraud loss, the navy part is the cost of false declines, and the gray part is the review friction. The number on top is the total cost.
So what: the challenger moves cost from fraud loss to false declines, and the sum goes down in each month. Thus the result does not depend on one good month.
The challenger in this test uses the model from months 1–3 with no retrain. Part 2 shows that a monthly retrain improves the score, so this decrease of the total cost is a low estimate. The dollar values are for this data. The data is a sample of the payment flow of one merchant, so the values for the full flow are larger.
How much of the gain depends on the champion?
The champion is our assumption (as-06, Step 0b). A weak champion makes any challenger look good. Thus we repeat branch 4 against four champions, from weak to strong. Each champion uses the same three tiers, the same review capacity and the same method to set its thresholds on month 4 (Step 5c). Only its score changes:
| Champion | Score | Why it is in the ladder |
|---|---|---|
| 1. Static rules (as-06) | Decision tree, 3 levels, fit on month 1 only | The base champion of this notebook: a small rule set that does not change. |
| 2. Rules from more data | Decision tree, 5 levels, fit on months 1–3 | A rule set that a strategy team updates with the same data as the challenger. |
| 3. Simple model | Logistic regression of Step 4b, fit on months 1–3 | A team with a model, but with a simple model. |
| 4. Same model, score thresholds | The GBM of the challenger, ranked by the fraud score (Step 5d, option A) | The strongest champion: the same score as the challenger. Only the dollar order is missing. |
| Total cost ($/month) | Fraud $ stopped | Decline rate of good transactions | Saving of the challenger ($/month) | |
|---|---|---|---|---|
| Champion | ||||
| 1. Static rules (as-06) | 299,432 | 38.4% | 0.28% | 126,820 |
| 2. Rules from more data | 276,518 | 65.4% | 8.10% | 103,906 |
| 3. Simple model | 295,739 | 58.6% | 7.90% | 123,127 |
| 4. Same model, score thresholds | 236,886 | 64.3% | 4.95% | 64,274 |
How to read: each bar is the gain of the challenger against one champion: the decrease of the total cost each month. The top bar is the base champion of this notebook. The bottom bar is the strongest champion: the same GBM score, with thresholds on the score instead of the expected cost.
So what: the gain is $126,820 against the static rules, but only $64,274 against the strongest champion. Thus 49% of the headline gain depends on the weak base champion, and 51% stays against the strongest champion. The part that does not depend on the champion is the dollar order of Step 5d. The challenger saves money against every champion in the ladder. A real test (Step 7c) measures the gain against the real rules of the merchant.
Rollout plan
The result of Step 6 comes from past transactions. Fraud patterns, customers and costs change. A controlled test gives the proof before a full rollout. The plan has five parts.
1. Test design. Send a random 10% of transactions to the challenger. The other 90% stay with the champion. A random allocation makes the two arms equal in all other ways, so the difference comes from the policy.
Each arm gets the same share of the review capacity as of the transactions. Thus the challenger arm gets 10% of the alerts each day, and the test keeps the condition “same review capacity”.
The test must run long enough to find the difference in cost. The code estimates the number of days from the cost of each transaction on months 5–6. It uses a significance level of 5% (two-sided) and a power of 80%.
| Value | |
|---|---|
| Transactions each day (months 5–6) | 2,856 |
| Transactions each day, challenger arm | 286 |
| Fraud transactions each day, challenger arm | 10 |
| Alerts each day, challenger arm | 12 |
| Difference of the mean cost for each transaction | $1.48 |
| Days to find the difference (formula) | 7.7 |
| Days to find the difference (doubled) | 16 |
The challenger arm gets about 286 transactions and 12 alerts each day. The challenger saves $1.48 for each transaction. To find this difference, the test needs about 16 days of transactions. Then it needs time for the chargebacks. The data has no chargeback dates, so Fraud Operations must give this delay.
2. Main KPI. The total cost for each 1,000 transactions in each arm, after the chargeback delay. The total cost includes the fraud loss, the false declines and the review friction (Step 5b). The cost for each 1,000 transactions makes the two arms of different size comparable.
3. Guardrails. Check these values each week:
- The decline rate of good transactions of the challenger arm: 3% or less (as-08). The label arrives late, so the decline rate of all transactions is the early signal.
- The alerts each day against the capacity of the arm.
- The PSI of the score against months 1–3, and the PR-AUC of each month with labels (Part 2).
4. Kill criteria. Stop the test in any of these cases:
- The decline rate of all transactions in the challenger arm is above 3% for two weeks in a row.
- The fraud loss for each 1,000 transactions of the challenger arm is above the champion arm for one full month with labels.
- The PSI of the score is more than 0.25.
5. Top risks.
| Risk | Effect | Control |
|---|---|---|
| The cost of a false decline (as-02) is higher than the assumption. | The challenger declines too many good customers. The decrease of the total cost is too high. | Finance and Marketing confirm as-02 and as-08 before the test. The guardrail and the kill criteria stop the test. |
| The fraud patterns change after the training months. | The score loses power, but the PSI stays low (Step 6c). | Retrain each month (Part 2). Check the PR-AUC of each month with labels. |
| Fraud operations cannot read why the policy declines (branch 5). | A partner complaint has no short answer. Fraud operations cannot check the declines. | Give the surrogate rules to fraud operations as patterns, not as reasons. If a partner asks for reasons, use the alternative of Step 4b. |
| The champion is an assumption (as-06). | The real champion can be stronger or weaker. The decrease of the total cost changes. | The test measures the real champion, because the 90% arm uses it. |
| The labels arrive late (as-07). | The test result and each retrain use labels that are not complete. | Read the KPI only after the chargeback delay. Train on all months except the last one when the labels are late. |
Decision at the end. If the challenger arm has a lower total cost for each 1,000 transactions and stays inside the guardrail, roll out the challenger. Increase the share step by step and keep the champion arm as a control.
Business note. At the current capacity, the challenger decreases the total cost by $126,820 each month. In each cost case that we tested, the decrease is at least $53,544. The best team size changes the cost by only $1,534 each month.
The recommendation is a test on 10% of transactions for about 16 days, with the guardrail of 3% (as-08). The test uses one score with no retrain. Part 2 examines how fast this score loses power, and when to retrain it.
What a reviewer should challenge
These limits can change the result most. Each one has a control.
| Limitation | Effect on the result | Control, or what to do next |
|---|---|---|
| The costs are assumptions (as-01 to as-05). | The dollar values and the size of the decline tier change. | The sensitivity check of Step 7b. Finance and Fraud Operations confirm the values. |
| The guardrail limit of 3% is an assumption (as-08). | A lower limit forces fewer declines and a smaller decrease of the total cost. | Marketing confirms the limit before the test. The kill criteria use it. |
| The champion is an assumption (as-06). We built it because the data has no real rules. | The gain of the challenger depends on the champion: $64,274 to $126,820 each month. | The ladder of four champions in Step 6d. Both policies use the same review capacity and cost method. The test of Step 7c measures the gain against the real rules. |
| The surrogate tree does not explain most declines (branch 5). | Fraud operations cannot give a short reason for each decline. | A fraud decline needs no adverse action notice. If a partner asks for reasons, use the alternative of Step 4b. |
| The PSI does not find a change of the fraud patterns. | A score can lose value with a stable PSI. | Check the PR-AUC of each month with labels. Retrain each month (dec-07). |
| The label includes later transactions of a customer after a chargeback (as-07). | The history features can look stronger than in production. | History from earlier transactions only (dec-02). |
| The data does not give the date of the chargeback. | The retrain test assumes that all labels are complete. | Train on all months except the last one when the labels are late. |
| The policy result uses the score with no retrain. | The decrease of the total cost is a low estimate. | Part 2 shows the gain of a monthly retrain. |
Other limits have a small effect. The data has no customer ID, so uid is a proxy. Vesta masked most features, so reason codes from the raw features have little meaning. The data is a sample of the payment flow of one merchant, so the dollar values are small. Scale the result with the actual volume. Real-time latency is a task for engineering.
Next steps
- Business case deck (6–8 slides): the challenger policy for a champion/challenger test on 10% of transactions.
- Confirm the cost assumptions with Finance (as-01, as-02), Fraud Operations (as-03, as-05) and Marketing (as-08).
- A one-pager for fraud operations: the review capacity, the value of one more analyst and the surrogate rules as patterns.
- Add a feature “earlier chargebacks of the customer” when the chargeback dates are available (dec-02).
- Set up the checks in production of Part 2 before the start of the test.
What each tool does in this case
| SQL | Joins the transaction table and the identity table, with DuckDB. |
| Python | Runs the whole analysis in one notebook that gives the same result on each run. |
| Machine learning | Scores fraud with gradient boosting, ranks transactions by expected cost, and sets the three tiers. |
| Statistics | Validates over time, tracks drift with the population stability index, and tests the sensitivity to costs. |