Case 02: Fraud strategy

Which approve, review and decline policy gives the lowest total cost of fraud, false declines and reviews?

Disclaimer. This notebook uses public data. The results are for illustration only. The data is not the data of an employer or a client. The costs, the review capacity and the champion policy are assumptions. The assumption register in the appendix gives the source of each assumption.

# Purpose: Load the libraries and the chart style. Set the paths, the random seed, the cost assumptions
#          (as-01 to as-05, as-08) and two helpers.
# Input:   None.
# Output:  Paths of the data files, `COST`, `TEST_SHARE`, `N` (numbers for the text), `fmt()` (table format),
#          chart style `cs`.

from pathlib import Path

import duckdb
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
from scipy.optimize import brentq
from sklearn.compose import ColumnTransformer
from sklearn.ensemble import HistGradientBoostingClassifier
from sklearn.impute import SimpleImputer
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import average_precision_score, roc_auc_score
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import OneHotEncoder, QuantileTransformer
from sklearn.tree import DecisionTreeClassifier, export_text

import chart_style as cs  # One chart style for all notebooks (deck style guide)

cs.setup()

# Fixed seed: each run gives the same samples and the same models.
SEED = 42
rng = np.random.default_rng(SEED)

# Paths are relative to this notebook (cases/02-fraud-strategy/).
DATA_DIR = Path("../../data")  # The raw data is not in the repository. See the README.
RAW_TXN = DATA_DIR / "raw" / "train_transaction.csv"
RAW_ID = DATA_DIR / "raw" / "train_identity.csv"
INTERIM_FILE = DATA_DIR / "interim" / "fraud_txn.parquet"
# Stop early with a clear message if a data file is missing.
if not INTERIM_FILE.exists() and not (RAW_TXN.exists() and RAW_ID.exists()):
    raise FileNotFoundError(
        f"Download train_transaction.csv and train_identity.csv to {RAW_TXN.parent}. See the README.")

# Cost assumptions (assumption register in the appendix).
COST = {
    "loss_share": 1.00,        # as-01
    "fd_margin": 0.03,         # as-02, share of the amount
    "fd_friction": 10.0,       # as-02, $ for each false decline
    "alerts_per_analyst": 40,  # as-03
    "analysts": 3,             # as-03
    "analyst_day": 300.0,      # as-04
    "review_catch": 0.90,      # as-05
    "review_friction": 2.0,    # as-05
    "fdr_limit": 0.03,         # as-08: guardrail, the highest decline rate of good transactions
}
TEST_SHARE = 0.10  # Share of the transactions in the champion/challenger test (Step 7)

N = {}  # Numbers for the text after each cell.
# Assumption values that the text of Step 0 uses.
N.update({
    "as_capacity": f"{COST['analysts'] * COST['alerts_per_analyst']}",
    "as_analysts": f"{COST['analysts']}",
    "as_fdr_limit": f"{COST['fdr_limit']:.0%}",
    "as_test_share": f"{TEST_SHARE:.0%}",
})



def fmt(table, formats):
    """Return a copy of the table with the number formats applied (for display only).
    A format is a format string, or a function for a special format (for example a dollar value)."""
    out = table.copy()
    for col, f in formats.items():
        out[col] = out[col].map(f if callable(f) else f.format)
    return out

How to read this notebook.

• The notebook hides the code. Click Show code to see the code of a step.
• A blue block is a decision. A green block is a business note. A grey block (Technical note) gives detail for data scientists. A business reader can skip it.
• The appendix holds the decision log, the assumptions, the glossary and the data dictionary.

Step 0. What and why

This step gives the business problem, the question for leadership and the method that answers it with past transactions. Each later step answers one part of the question.

Purpose. This step answers: what must a new fraud policy do for the business, and how can we test it before a real launch? It supports this decision: approve a challenger policy for a champion/challenger test.

0a. The business problem

Situation. An online merchant accepts card payments. The policy gives one of three actions to each transaction: approve, review or decline. A review sends the transaction to an analyst before the approval. A set of static rules selects the action. This rule set is the current policy: the champion.

Complication. The fraud loss goes up. Stricter rules decline more good customers, and a false decline also costs money: lost margin and customer friction. The analyst team can examine only 120 alerts each day (as-03). Fraud patterns change over time, so static rules lose value.

Question. Which policy gives the lowest total cost of fraud loss, false declines and reviews, with the same review capacity?

Resolution. Build a fraud score with a model. Put each transaction in a tier by its expected cost in dollars, inside the review capacity. Compare this challenger with the champion on the same past transactions. Give fraud operations readable rules for the decline tier. Retrain the score each month and monitor the drift.

If the challenger wins on the past transactions, test it on 10% of real transactions before a full rollout.

Decision that this notebook supports. Approve the challenger policy for a champion/challenger test on 10% of transactions.

Success metric. The total cost each month at the same review capacity. Guardrail metric: the decline rate of good transactions. Its limit is 3% (as-08).

0b. Method and data strategy

A technical reader asks one question first. How can we compare two policies without a live test?

The data has the fraud label of each past transaction. Thus we know the actual cost of each action on each transaction. An approved fraud transaction costs its amount. A declined good transaction costs lost margin and customer friction. We use an offline replay in four moves:

1. Build the champion. The data does not give the rules of the merchant. Thus we build a champion ourselves (assumption as-06): a decision tree with three levels on month 1. It does not change after month 1, like static rules.
2. Build the challenger. A model gives a fraud score from the data of months 1–3. The policy ranks each transaction by its expected cost in dollars.
3. Set both policies on the same month. Both policies set their thresholds on month 4. They use the same three tiers, the same costs and the same review capacity.
4. Replay both policies on months 5–6. Each policy gives an action to each test transaction. The fraud label gives the actual cost of each action. We compare the total cost each month.

The replay keeps all conditions the same, except the score and the value that ranks the transactions. Thus a difference in the total cost comes from the policy, not from the data or the costs.

The champion is our assumption, not the real rules of the merchant. A reader must know this before any result. Each gain in this notebook is a gain against a champion that we built.

Question	Answer
What does a real merchant have?	A production rule set and an analyst team. In a real project, these rules are the champion, and this section is not necessary.
What does the data not give?	Vesta published the transactions and the fraud labels. It did not publish the rules of the merchant, or the action of the merchant on each transaction.
What do we build?	A decision tree with three levels, fit on month 1 (as-06). It gives at most eight rule segments, like a small rule set. The fraud rate of a segment is the champion score. Inside a segment, the analysts review the largest amounts first. The tree does not change after month 1, like static rules.
Why this choice?	Many merchants start with a small static rule set. A tree gives such rules from the data, with no hand-made thresholds. The champion uses the same three tiers, costs and review capacity as the challenger (as-03).
Which assumptions go with it?	as-06: the champion rules. as-03: the review capacity. as-01, as-02, as-05: the costs. as-08: the guardrail limit.
How can it be wrong?	Real rules can be stronger: more segments, data from more months, regular updates, or a vendor score. Then the gain of the challenger is smaller. Real rules can also be weaker, but we do not assume that.
How do we test it?	Step 6d repeats branch 4 against a ladder of four champions, from our static rules to the same GBM with thresholds on the fraud score. Step 6d gives the gain against each champion.
What does not depend on it?	Three comparisons use the same GBM on both sides. The first is the dollar order against the score order (Step 5d). The others are three tiers against two tiers (Step 5c) and the monthly retrain (Part 2).
What does it mean for the claim?	The headline gain is the upper end of a range. The lower end, against the strongest champion, is the gain that we can defend. The champion/challenger test of Step 7c measures the gain against the real rules.

Why months. The model scores future transactions in production. A split by time gives the same condition: the model never sees the months that test it. A month is also the unit of the business decisions: the cost each month and the retrain each month.

A random split puts transactions of the same customer in the training set and in the test set. Then the label after a chargeback (as-07) copies into the test, and the result is too good. Step 3b shows the size of this effect.

The limits of the method.

• The replay does not see the effect of an action on later transactions. In production, a decline can stop a fraudster or lose a good customer.
• The data has no chargeback dates. The replay assumes that all labels are complete when the model trains.
• The costs are assumptions (as-01 to as-05). Step 7b repeats the result for other costs.
• The champion is an assumption (as-06). The gain of the challenger depends on it (see the block above).
• The data has no customer ID. The customer key uid is a proxy (Step 2).
• The data is a sample of the payment flow of one merchant. The dollar values are for this sample.

Technical note. The replay has these parts. Each later step gives the code.

• Actual cost. A transaction has the amount \(a\) and the fraud label \(y\). An approval costs \(y \cdot a \cdot L\). A decline costs \((1-y)(m \cdot a + f)\). A review costs \(y \cdot a \cdot L (1-q) + (1-y) f_r\). Step 5b defines the symbols and gives the expected cost.
• Champion score. The fraud rate of the leaf of the tree. A small term from the rank of the amount puts the larger amounts first inside a leaf.
• Champion thresholds. A grid search on month 4 finds the decline threshold with the lowest total cost. The review tier takes the next highest scores, up to the capacity.
• Challenger score. A GBM on months 1–3, with a recalibration on month 4 (Step 5a). The policy gives each transaction the action with the lowest expected cost. The review tier takes the transactions where a review decreases the expected cost most.
• Same review capacity. A real-time decision cannot wait for the end of the day. Thus each policy uses a threshold, not a daily rank. The code sets the threshold on month 4, so that the mean number of alerts each day equals the capacity.
• Test months. No threshold and no model uses months 5–6. They give only the final result.

0c. How the notebook answers the question

Each step answers one part of the question:

• Step 1. Which data gives the fraud label of each transaction, and what does it contain?
• Step 2. How do we put the transactions in time order, and which features can the model use fairly? What does fraud cost if the merchant approves all transactions?
• Step 3. Which rules must the challenger pass, and which months test it? What does the champion do on these months?
• Step 4. Which columns and which model type put the fraud transactions first? How do we handle the low fraud rate?
• Step 5. How does the score become an action: approve, review or decline?
• Step 6. Does the challenger pass each rule? Is it better than the champion in dollars?
• Step 7. What is the value of one more analyst, and how do we test the policy with real transactions?
• Part 2. Does the score stay good over time, and how often do we retrain it?

Business note. The question is a trade between fraud loss and false declines, inside a fixed review capacity. The replay compares the two policies on actual labels, so each cost is a fact of the past under the cost assumptions. Step 1 shows the data that gives these labels.

Step 1. Get data

Step 0 set the question and the method: replay two policies on labeled past transactions at the same review capacity. This step answers: which data gives these labels, and what does it contain?

Purpose. This step gets the data and its meaning. It supports this decision: if the data can answer the business question.

1a. Why this data

The data is the IEEE-CIS Fraud Detection dataset of Vesta Corporation. Source: Kaggle competition ieee-fraud-detection, files train_transaction.csv and train_identity.csv. The two files use TransactionID as the key. The data is a fair proxy for the fraud decision of a card issuer or a merchant, for four reasons:

• Real card-not-present payments. The transactions come from real e-commerce payments. Card-not-present fraud is the largest type of card fraud online.
• The same data at the decision. Each transaction has the amount, the product, the card, the address, the email and the device. A fraud system has the same data at the time of the payment.
• A fraud label for each transaction. The label comes from chargebacks. Thus we can calculate the actual cost of each action on each transaction.
• Time order. TransactionDT gives the time of each transaction. Thus we can split the data by month and replay each month.

The data cannot show these things:

What the data does not show	Why it matters	What the notebook does
A customer ID	History features need a customer key.	A proxy key uid from the card, the address and the first day of the card (Step 2).
The meaning of most columns	Vesta masked most columns. A reason code from a masked column has little meaning.	A surrogate tree with readable features (Step 6).
The chargeback dates	We do not know when each label becomes available.	The retrain plan trains on all months except the last one when the labels are late (Part 2).
The rules and the actions of the merchant	We do not know the real champion.	A champion from a decision tree on month 1 (as-06, Step 0b).
The result of a different action	A declined fraudster can come back. A declined good customer can leave.	A live test on real transactions (Step 7).
The full payment flow	The dollar values are for a sample.	Scale the result with the actual volume.

1b. Key variables

The table gives the column groups that the story uses. The target is isFraud. Appendix A gives the meaning and the use of all columns of the two files.

Column group	Columns	Meaning in plain words	Use
Label	isFraud	1 = fraud. The label comes from a chargeback. It also marks later transactions of the same customer (as-07).	Target
Time	TransactionDT	Seconds from a reference time. It is not a calendar date.	Time order, day, month and hour (Step 2). Not a feature.
Amount	TransactionAmt	The amount of the payment in US dollars.	Feature. Base of all dollar costs.
Product	ProductCD	The product code of the transaction (masked).	Feature
Card	card1–card6	Card data: for example the card network (card4) and debit or credit (card6). Vesta masked the other values.	Feature. card1 is a part of uid.
Address and distance	addr1, addr2, dist1, dist2	Billing region and country, and two distances (masked).	Feature. addr1 is a part of uid.
Email	P_emaildomain, R_emaildomain	The email domain of the buyer and of the recipient.	Feature. Step 2 adds email_match.
Counts	C1–C14	Counts, for example the number of addresses for the card (masked).	Feature
Time differences	D1–D15	Days between events. D1 is the age of the card in days.	Feature. D1 gives the first day of the card for uid.
Match flags	M1–M9	Matches, for example the name on the card against the address (masked).	Feature
Vesta features	V1–V339	Features from Vesta: ranks, counts and other relations (masked).	Feature. Step 1c keeps a part of them.
Identity	DeviceType, DeviceInfo, id_30, id_31, id_33	The device, the system of the device, the browser and the screen. Only a part of the transactions has these values.	Feature. Step 1c adds the flag has_identity.

A column with “Not a feature” never goes to the model. A masked column can be a feature, but it cannot give a readable reason for a decline.

1c. Load the data and look at it

The code joins the two files with SQL (DuckDB). The first run reads the two CSV files one time and writes a small Parquet copy. Later runs read the copy in a few seconds.

The V features come in groups with the same pattern of missing values. Two features of one group often carry the same information. To keep the model small, the code keeps two features of each group: the two with the most distinct values. This rule uses no label, so it does not cause leakage. Step 4a gives all column decisions.

The key parameters of the load are:

• Features of each missing pattern of V: two, with the most distinct values.
• Identity columns: DeviceType, DeviceInfo, id_30, id_31 and id_33.
• Join: a left join, so each transaction stays in the data. The flag has_identity marks a transaction with identity data.

# Purpose: Join the transaction file and the identity file with SQL (DuckDB). Keep two V features of each
#          missing pattern. Keep a Parquet copy for fast reloads.
# Input:   data/raw/train_transaction.csv, data/raw/train_identity.csv (first run only).
# Output:  `txn` (one row for each transaction), `V_KEEP` (V features that the code keeps),
#          `BASE_COLS`, `ID_COLS` (the other columns that the code reads).

BASE_COLS = (
    ["TransactionID", "isFraud", "TransactionDT", "TransactionAmt", "ProductCD"]
    + [f"card{i}" for i in range(1, 7)] + ["addr1", "addr2", "dist1", "dist2",
       "P_emaildomain", "R_emaildomain"]
    + [f"C{i}" for i in range(1, 15)] + [f"D{i}" for i in range(1, 16)]
    + [f"M{i}" for i in range(1, 10)]
)
ID_COLS = ["DeviceType", "DeviceInfo", "id_30", "id_31", "id_33"]

# First run only: read the two CSV files and write one Parquet file.
if not INTERIM_FILE.exists():
    INTERIM_FILE.parent.mkdir(parents=True, exist_ok=True)
    con = duckdb.connect()
    con.sql(f"""CREATE VIEW t AS SELECT * FROM read_csv('{RAW_TXN.as_posix()}',
                header = true, sample_size = -1)""")
    con.sql(f"""CREATE VIEW i AS SELECT * FROM read_csv('{RAW_ID.as_posix()}',
                header = true, sample_size = -1)""")
    # V features: count the values and the distinct values of each column.
    v_cols = [c for c in con.sql("DESCRIBE t").df()["column_name"] if c.startswith("V")]
    stats = con.sql("SELECT " + ", ".join(
        f"COUNT({c}) AS n_{c}, approx_count_distinct({c}) AS d_{c}" for c in v_cols
    ) + " FROM t").df().iloc[0]
    vstat = pd.DataFrame({"col": v_cols,
                          "non_null": [stats[f"n_{c}"] for c in v_cols],
                          "distinct": [stats[f"d_{c}"] for c in v_cols]})
    # The same count of values = the same missing pattern. Keep the two columns with the most distinct values.
    v_keep = (vstat.sort_values("distinct", ascending=False)
              .groupby("non_null").head(2)["col"].tolist())
    select = ", ".join([f"t.{c}" for c in BASE_COLS + v_keep]
                       + [f"i.{c}" for c in ID_COLS]
                       + ["i.TransactionID IS NOT NULL AS has_identity"])
    # Left join: each transaction stays, with or without identity data.
    con.sql(f"""
        COPY (SELECT {select} FROM t LEFT JOIN i ON t.TransactionID = i.TransactionID)
        TO '{INTERIM_FILE.as_posix()}' (FORMAT parquet)
    """)

# Later runs: read the Parquet file (a few seconds).
txn = duckdb.sql(f"SELECT * FROM read_parquet('{INTERIM_FILE.as_posix()}')").df()
V_KEEP = [c for c in txn.columns if c.startswith("V")]
print(f"Transactions: {len(txn):,} | V features kept: {len(V_KEEP)} of 339")

Transactions: 590,540 | V features kept: 30 of 339

The next cell shows the initial state of the data before any change. It gives the size, the time range, the fraud rate and the share of missing values in each column group.

# Purpose: Show the initial state of the data before any change: size, time range, fraud rate and
#          the share of missing values in each column group of Step 1b.
# Input:   `txn`, `V_KEEP`, `BASE_COLS`, `ID_COLS`, RAW_TXN, RAW_ID (only the header line).
# Output:  Numbers for the text (`N`), `col_groups` (column group -> columns),
#          table of the initial state (printed), table of missing values by column group.

# Column groups of Step 1b. The V group holds only the V features that the load keeps.
col_groups = {
    "Label": ["isFraud"],
    "Time": ["TransactionDT"],
    "Amount": ["TransactionAmt"],
    "Product": ["ProductCD"],
    "Card": [f"card{i}" for i in range(1, 7)],
    "Address and distance": ["addr1", "addr2", "dist1", "dist2"],
    "Email": ["P_emaildomain", "R_emaildomain"],
    "Counts": [f"C{i}" for i in range(1, 15)],
    "Time differences": [f"D{i}" for i in range(1, 16)],
    "Match flags": [f"M{i}" for i in range(1, 10)],
    "Vesta features (kept)": V_KEEP,
    "Identity": ID_COLS,
}
MISS_CUT = 0.5  # A column with more missing values than this share counts as "mostly missing"
miss = pd.DataFrame([
    {"Column group": g, "Columns": len(cols),
     "Share missing": txn[cols].isna().to_numpy().mean(),
     "Columns with more than 50% missing": int((txn[cols].isna().mean() > MISS_CUT).sum())}
    for g, cols in col_groups.items()
]).set_index("Column group")

# Size of the raw files: only the header line, so no full read of the CSV files.
raw_cols = {f: len(pd.read_csv(f, nrows=0).columns) if f.exists() else None for f in [RAW_TXN, RAW_ID]}
days = txn["TransactionDT"] / 86400  # TransactionDT is in seconds
feature_cols = [c for cols in col_groups.values() for c in cols]
N.update({
    "s1_rows": f"{len(txn):,}",
    "s1_cols": f"{txn.shape[1]}",
    "s1_raw_txn_cols": f"{raw_cols[RAW_TXN]}" if raw_cols[RAW_TXN] else "n/a",
    "s1_raw_id_cols": f"{raw_cols[RAW_ID]}" if raw_cols[RAW_ID] else "n/a",
    "s1_v_keep": f"{len(V_KEEP)}",
    "s1_day_first": f"{days.min():.0f}",
    "s1_day_last": f"{days.max():.0f}",
    "s1_days": f"{days.max() - days.min():.0f}",
    "s1_fraud_n": f"{int(txn['isFraud'].sum()):,}",
    "s1_fraud_rate": f"{txn['isFraud'].mean():.2%}",
    "s1_identity_share": f"{txn['has_identity'].mean():.0%}",
    "s1_miss_all": f"{txn[feature_cols].isna().to_numpy().mean():.0%}",
    "s1_miss_id": f"{miss.loc['Identity', 'Share missing']:.0%}",
    "s1_miss_v": f"{miss.loc['Vesta features (kept)', 'Share missing']:.0%}",
    "s1_miss_d": f"{miss.loc['Time differences', 'Share missing']:.0%}",
    "s1_cols_over_half": f"{int(miss['Columns with more than 50% missing'].sum())}",
    "s1_miss_cut": f"{MISS_CUT:.0%}",
})

# Initial state: one row for each fact.
state = pd.Series({
    "Rows (transactions)": N["s1_rows"],
    "Columns after the load": N["s1_cols"],
    "Columns in the raw files (transaction / identity)": f"{N['s1_raw_txn_cols']} / {N['s1_raw_id_cols']}",
    "V features kept": f"{N['s1_v_keep']} of 339",
    "Time range (days from the reference time)": f"{N['s1_day_first']} to {N['s1_day_last']}",
    "Fraud transactions": N["s1_fraud_n"],
    "Fraud rate": N["s1_fraud_rate"],
    "Transactions with identity data": N["s1_identity_share"],
    "Share missing, all columns": N["s1_miss_all"],
}, name="Value")
print(state.to_string())

fmt(miss, {"Share missing": "{:.0%}"})

Rows (transactions)                                    590,540
Columns after the load                                      91
Columns in the raw files (transaction / identity)     394 / 41
V features kept                                      30 of 339
Time range (days from the reference time)             1 to 183
Fraud transactions                                      20,663
Fraud rate                                               3.50%
Transactions with identity data                            24%
Share missing, all columns                                 38%

	Columns	Share missing	Columns with more than 50% missing
Column group
Label	1	0%	0
Time	1	0%	0
Amount	1	0%	0
Product	1	0%	0
Card	6	1%	0
Address and distance	4	44%	2
Email	2	46%	1
Counts	14	0%	0
Time differences	15	58%	8
Match flags	9	50%	4
Vesta features (kept)	30	45%	14
Identity	5	81%	5

The data has 590,540 transactions and 91 columns after the load. The raw files have 394 columns (transactions) and 41 columns (identity). The load keeps 30 of the V features. The data covers 182 days, from day 1 to day 183 after the reference time.

The data has 20,663 fraud transactions: a fraud rate of 3.50%. This rate is low, so a random score finds few fraud transactions. Step 3a explains why the notebook uses the PR-AUC for this reason. Step 4c examines the effect of the low rate on the training.

Missing values are frequent. Of all values, 38% are missing. Only 24% of the transactions have identity data, so the identity group has 81% missing values. The kept V features have 45% missing values, and the time differences have 58%. In total, 34 columns have more than 50% missing values. Step 2 explains why the GBM uses these values without a fill.

Business note. The data gives a fraud label, an amount and the data at the time of the payment for each transaction. Thus the replay can calculate the actual cost of each action. The data has no customer ID, no chargeback dates and few readable columns, and the notebook says where each limit matters. Step 2 puts the transactions in time order and builds the features that the model can use fairly.

Step 2. Clean and prepare data

Step 1 showed that the two Vesta files can answer the business question. They give the amount, the time and the fraud label of each transaction. But the data has no customer ID, the label of a customer can change after a chargeback, and many cells are empty. This step answers: which rows, which columns and which values can the model use, and what does fraud cost today?

Purpose. This step makes sure that the data can answer the business question without leakage. It supports this decision: the rows, the label and the features that the fraud score can use, and the baseline that the challenger must beat.

The data preparation follows one tree with three levels. Each level has one split axis. Each axis divides all cases into groups that do not overlap and that together cover all cases (MECE).

Level	Split axis	Groups	What this data needs	Section
Rows	Is the record a valid transaction with a label?	Not valid · valid	Four technical checks. Then a time order and six months.	2a
Columns	Does the business know the value at the time of the decision?	Known before the decision · known only after the decision	A customer key from earlier data. A label that can come from a later chargeback (as-07). A date that moves each month (D1n).	2b
Values	Why is a cell empty or extreme?	Structural · masked · extreme but real	No imputation for the GBM. No cap on the amount.	2c

Then Step 2d makes new features from earlier transactions, and Step 2e measures the baseline: the fraud loss if the merchant approves all transactions.

2a. Rows: which transactions can the model learn from?

The first split asks if a record is a valid transaction. The code checks four conditions. A record is not valid if it repeats a transaction or if a key field is empty. It is also not valid if the amount is $0 or less, or if the label is not 0 or 1.

The data has no product or channel that is outside the business question. Thus each valid row is in scope.

# Purpose: Check the rows, sort them by time and add `day`, `month` and `hour`.
# Input:   `txn` (from the load cell of Step 1).
# Output:  `txn` sorted by `TransactionDT`, with `day`, `month` and `hour`. Tables `row_checks` and `months`.

n_loaded = len(txn)
# Each check gives True for a row that is not valid.
row_checks = {
    "Duplicate TransactionID": txn["TransactionID"].duplicated(keep="first"),
    "Empty key field (ID, time, amount, label)": txn[
        ["TransactionID", "TransactionDT", "TransactionAmt", "isFraud"]].isna().any(axis=1),
    "Amount of $0 or less": txn["TransactionAmt"] <= 0,
    "Label not 0 or 1": ~txn["isFraud"].isin([0, 1]),
}
invalid = pd.concat(row_checks, axis=1).any(axis=1)
txn = txn[~invalid]

# Time order. A stable sort keeps the file order for two transactions in the same second.
txn = txn.sort_values("TransactionDT", kind="stable").reset_index(drop=True)
# TransactionDT counts seconds from a reference time that Vesta did not publish.
txn["day"] = (txn["TransactionDT"] // 86400).astype(int)
# Month = 30 days. The last 3 days go into month 6, so that no month has only a few days.
txn["month"] = np.minimum(txn["day"] // 30 + 1, 6).astype(int)
txn["hour"] = (txn["TransactionDT"] // 3600) % 24  # Hour of the reference clock, not local time

months = txn.groupby("month").agg(
    First_day=("day", "min"), Last_day=("day", "max"), Days=("day", "nunique"),
    Transactions=("isFraud", "size"), Fraud_rate=("isFraud", "mean"))
months.columns = ["First day", "Last day", "Days", "Transactions", "Fraud rate"]
N.update({
    "n_loaded": f"{n_loaded:,}", "n_invalid": f"{int(invalid.sum()):,}",
    "n_days": f"{txn['day'].nunique()}", "days_m6": f"{months.loc[6, 'Days']}",
    "days_m1": f"{months.loc[1, 'Days']}",
})
print(pd.Series({k: int(v.sum()) for k, v in row_checks.items()}, name="Rows that fail").to_string(), "\n")
fmt(months, {"Transactions": "{:,.0f}", "Fraud rate": "{:.2%}"})

Duplicate TransactionID                      0
Empty key field (ID, time, amount, label)    0
Amount of $0 or less                         0
Label not 0 or 1                             0 

	First day	Last day	Days	Transactions	Fraud rate
month
1	1	29	29	130,968	2.48%
2	30	59	30	89,838	4.04%
3	60	89	30	91,768	4.03%
4	90	119	30	98,027	3.93%
5	120	149	30	85,303	3.47%
6	150	182	33	94,636	3.47%

0 of the 590,540 rows fail a check. The file is clean at the row level.

Time order and months. TransactionDT gives the seconds from a reference time. Vesta did not publish this time, so the data has no calendar date. The code sorts all rows by TransactionDT before any other step. Each feature that uses other transactions then can read only the earlier rows.

The data covers 182 days. The notebook calls each block of 30 days a month. Month 1 has 29 days, because the first day of the data is day 1. Month 6 has 33 days, because it also takes the last days of the data. Thus all results in dollars use a cost for each 30 days, not a cost for each calendar month.

2b. Columns: which values does the business know at the time of the decision?

A column can be a feature only if the business knows its value when the transaction arrives. Three columns of this data need a decision before Step 4: the customer key, the label and the first day of the card.

The customer key uid is a proxy. The data has no customer ID. Fraud is a behavior of a customer, so the history features need a key for the customer. A common key in the Kaggle discussion joins three values:

• card1: a masked code of the card.
• addr1: a masked code of the billing region.
• D1n: the first day of the card. D1 gives the days from the start of the card to the transaction. Thus D1n = day − D1.

Two real customers can share a uid, and one customer can have two uid values. If D1 or addr1 is empty, the key is empty. The code measures these cases.

The label after a chargeback (as-07). Vesta gives a fraud label to the transaction with a chargeback. It also gives the label to the later transactions of the same account, email address or billing address. Thus a customer with one fraud often has more fraud labels after it. A feature that reads later transactions of the customer, or the labels of earlier transactions, can copy the label.

The data has no chargeback dates. Thus the code cannot know when the business learned about an earlier fraud.

The first day of the card D1n is a date. Each month brings new cards, so the values of D1n move up each month. A model that learns from a date does not repeat its result in later months. Thus D1n is part of the key, but not a feature. The model uses D1, the age of the card.

# Purpose: Make the customer key `uid` and measure its weak points. Measure the effect of as-07.
# Input:   `txn`.
# Output:  `txn` with `D1n` and `uid`. Table `key_checks`. Numbers for the text.

txn["D1n"] = txn["day"] - txn["D1"]  # First day of the card. Only for the customer key.
txn["uid"] = (txn["card1"].astype(str) + "_" + txn["addr1"].astype(str)
              + "_" + txn["D1n"].astype(str))
key_ok = txn["D1"].notna() & txn["addr1"].notna()  # The key has all three parts

# as-07: did the same uid have an earlier fraud? Only for a description of the label. No feature uses it.
g = txn.groupby("uid", sort=False)
earlier_fraud = (g["isFraud"].cumsum() - txn["isFraud"]) > 0
size_uid = g["isFraud"].transform("size")
fraud = txn["isFraud"] == 1

key_checks = pd.Series({
    "Distinct uid values": txn["uid"].nunique(),
    "Transactions with an empty part of the key (D1 or addr1)": int((~key_ok).sum()),
    "Transactions of a uid with more than one transaction": int((size_uid > 1).sum()),
    "Fraud transactions after an earlier fraud of the same uid": int((fraud & earlier_fraud & key_ok).sum()),
}, name="Value")
N.update({
    "n_uid": f"{txn['uid'].nunique():,}",
    "key_empty": f"{(~key_ok).mean():.1%}",
    "uid_repeat": f"{(size_uid > 1).mean():.0%}",
    "fraud_after_fraud": f"{earlier_fraud[fraud & key_ok].mean():.0%}",
    "fr_after_fraud": f"{txn.loc[earlier_fraud & key_ok, 'isFraud'].mean():.0%}",
    "fr_no_earlier": f"{txn.loc[~earlier_fraud & key_ok, 'isFraud'].mean():.1%}",
    "d1n_med_1": f"{txn.loc[txn['month'] == 1, 'D1n'].median():.0f}",
    "d1n_med_6": f"{txn.loc[txn['month'] == 6, 'D1n'].median():.0f}",
})
del g, size_uid, earlier_fraud, fraud
key_checks.map("{:,.0f}".format).to_frame()

	Value
Distinct uid values	199,070
Transactions with an empty part of the key (D1 or addr1)	66,794
Transactions of a uid with more than one transaction	408,233
Fraud transactions after an earlier fraud of the same uid	8,070

The key gives 199,070 customers. 69% of the transactions belong to a uid with more than one transaction, so a history exists for most transactions. For 11.3% of the transactions, a part of the key is empty. These transactions have no uid history, and the model uses only the card1 history for them.

The label after a chargeback has a large effect. 63% of the fraud transactions with a full key come after an earlier fraud of the same uid. After an earlier fraud, the fraud rate of a uid is 66%. Without an earlier fraud, it is 0.9%.

Thus a feature that knows the earlier labels of a customer looks strong in a test. In production, the business learns about a fraud only weeks later, from the chargeback. Step 4a compares the history features with this risk (the decision on history features).

The median of D1n goes from day 9 in month 1 to day 150 in month 6. This change comes from the calendar, not from the risk. Part 2 shows its drift.

2c. Values: missing values and extreme values

The split axis for an empty cell is the cause. Either the source of the value does not exist for this transaction (structural), or Vesta masked the meaning, so the cause is not known (masked). For an extreme value, the axis is reality: the value is impossible, or it is extreme but real.

# Purpose: Measure the empty cells by column group and show that an empty cell carries risk information.
# Input:   `txn`, `V_KEEP`.
# Output:  Table `missing_groups`. Numbers for the text.

ID_LOADED = [c for c in ["DeviceType", "DeviceInfo", "id_30", "id_31", "id_33"] if c in txn.columns]
GROUPS = {
    "Card (card1–card6)": ([f"card{i}" for i in range(1, 7)], "Structural: few cells"),
    "Address (addr1, addr2)": (["addr1", "addr2"], "Structural: no billing address"),
    "Distance (dist1, dist2)": (["dist1", "dist2"], "Structural: no second address"),
    "Email domains (P_, R_)": (["P_emaildomain", "R_emaildomain"], "Structural: no recipient email"),
    "Counts (C1–C14)": ([f"C{i}" for i in range(1, 15)], "Masked"),
    "Time deltas (D1–D15)": ([f"D{i}" for i in range(1, 16)], "Structural: no earlier event"),
    "Match flags (M1–M9)": ([f"M{i}" for i in range(1, 10)], "Masked"),
    f"Vesta features (V, {len(V_KEEP)} kept)": (V_KEEP, "Masked"),
    "Identity (device, browser)": (ID_LOADED, "Structural: no identity row"),
}
missing_groups = pd.DataFrame([
    {"Column group": name, "Columns": len(cols), "Cause": cause,
     "Mean share empty": txn[cols].isna().mean().mean(),
     "Highest share empty": txn[cols].isna().mean().max()}
    for name, (cols, cause) in GROUPS.items()]).set_index("Column group")

has_id = txn["has_identity"].astype(bool)
# The loaded columns, without the columns of Steps 2a and 2b.
loaded = [c for c in txn.columns if c not in {"day", "month", "hour", "D1n", "uid"}]
N.update({
    "miss_id_all": f"{txn['DeviceType'].isna().mean():.0%}",
    "miss_id_in": f"{txn.loc[has_id, 'DeviceType'].isna().mean():.1%}",
    "fr_id": f"{txn.loc[has_id, 'isFraud'].mean():.1%}",
    "fr_no_id": f"{txn.loc[~has_id, 'isFraud'].mean():.1%}",
    "miss_d_max": f"{txn[[f'D{i}' for i in range(1, 16)]].isna().mean().max():.0%}",
    "n_cols_half_empty": f"{int((txn[loaded].isna().mean() > 0.5).sum())}",
    "n_cols_loaded": f"{len(loaded)}",
    "amt_max": f"${txn['TransactionAmt'].max():,.0f}",
    "amt_p99": f"${txn['TransactionAmt'].quantile(0.99):,.0f}",
    "amt_med": f"${txn['TransactionAmt'].median():,.0f}",
    # Raw columns of the two files (header rows only). TransactionID is in both files.
    "n_raw_cols": (f"{len(pd.read_csv(RAW_TXN, nrows=0).columns) + len(pd.read_csv(RAW_ID, nrows=0).columns) - 1}"
                   if RAW_TXN.exists() and RAW_ID.exists() else "434"),
})
del has_id, loaded
fmt(missing_groups, {"Mean share empty": "{:.0%}", "Highest share empty": "{:.0%}"})

	Columns	Cause	Mean share empty	Highest share empty
Column group
Card (card1–card6)	6	Structural: few cells	1%	2%
Address (addr1, addr2)	2	Structural: no billing address	11%	11%
Distance (dist1, dist2)	2	Structural: no second address	77%	94%
Email domains (P_, R_)	2	Structural: no recipient email	46%	77%
Counts (C1–C14)	14	Masked	0%	0%
Time deltas (D1–D15)	15	Structural: no earlier event	58%	93%
Match flags (M1–M9)	9	Masked	50%	59%
Vesta features (V, 30 kept)	30	Masked	45%	86%
Identity (device, browser)	5	Structural: no identity row	81%	88%

34 of the 91 loaded columns have more than 50% empty cells. What we do with each group:

Group	Example in this data	What we do
Structural missing value	DeviceType is empty for 76% of all transactions, but for only 2.4% of the transactions with an identity row. The fraud rate is 7.8% with identity data and 2.1% without it.	Keep the empty cell. The flag has_identity (Step 2d) also records the source.
Masked missing value	A D column is empty for up to 93% of the transactions. Vesta does not say why.	Keep the empty cell. The GBM learns the risk of the empty cells from the data.
Impossible value	The row checks of Step 2a find no amount of $0 or less.	No change is necessary.
Extreme but real value	TransactionAmt goes up to $31,937. The 99th percentile is $1,104, and the median is $69.	Keep the value. Do not cap it. A large amount is the largest fraud loss.

Why the GBM needs no imputation. An empty cell in this data often tells something about the risk. The identity data shows this: the fraud rate with identity data is different from the fraud rate without it. A median or a fixed value in the empty cell puts the transaction into a group of real values, and the model loses this information.

Technical note. The GBM (HistGradientBoostingClassifier) cuts each numeric feature into at most 255 bins at the quantiles of the training data. Empty cells go into a separate bin. At each split, the GBM sends the empty cells to the side that gives the better fit. Thus the GBM learns a risk for “empty” from the data, and it needs no imputation.

The GBM also uses only the order of the values. Thus an extreme amount goes into the top bin and does not change the other splits. A cap at the 99th percentile changes nothing for the GBM. The logistic regression of Step 4b cannot read an empty cell. It uses the median of the training data and adds a flag for each empty cell (SimpleImputer with add_indicator=True).

2d. Feature engineering

Feature engineering makes a better input from the raw columns. The split axis is the rows that a new feature reads. These rows are the same transaction only, the earlier transactions of the same key, or the later transactions of the same key. The table shows what this data needs in each group.

Group	Method	In this data
Same transaction: one column	A change of scale or a part of a value	hour from TransactionDT. log_amt = ln(1 + amount), for the logistic regression. cents: the part of the amount after the decimal point.
Same transaction: two columns	A flag for a match	email_match = 1 if the email domain of the buyer and of the recipient are the same.
Same transaction: two files	A flag for the source	has_identity = 1 if the transaction has a row in the identity file.
Earlier transactions of the same key	History features	Four features for uid and four for card1. They give the count of earlier transactions, the amount against their mean, the hours since the last one and the count today.
Later transactions of the same key	History features over all transactions	Not used. The business does not know the later transactions at the time of the decision, and they carry the label after a chargeback (as-07).

The history features show a change from the normal behavior of a customer. For example, a fraud often has an amount far above the earlier amounts of the card, or many transactions in one day.

Technical note. For transaction t of key k (uid or card1), with the rows in time order:

• _n = the number of transactions of k before t.
• _amt_ratio = amount(t) / mean amount of the transactions of k before t. The value is empty if k has no earlier transaction.
• _hours_since = (TransactionDT(t) − TransactionDT of the last transaction of k before t) / 3,600.
• _n_today = the number of transactions of k before t on the same day.

The code calculates each feature with a cumulative sum or a cumulative count in the sorted data, so each value uses only earlier rows. The argument past_only=False makes the leakage version over all transactions of k. Only the evidence of the decision on history features in Step 4a uses it.

The history starts at the first day of the data, so the counts grow over time. Part 2 shows this drift. In production, a history window of fixed length (for example 90 days) stops it.

# Purpose: Make the features of Step 2d and the feature lists for Steps 3 and 4.
# Input:   `txn` (sorted, with `day` and `uid`).
# Output:  `txn` with the new features. `add_history()`, `HIST`, `EXCLUDE`, `CAT_COLS`, `NUM_COLS`.
#          `train`, `valid`, `test` (the split of Step 3, dec-01).

# Same transaction.
txn["log_amt"] = np.log1p(txn["TransactionAmt"])
txn["cents"] = (txn["TransactionAmt"] % 1).round(3)
txn["email_match"] = (txn["P_emaildomain"] == txn["R_emaildomain"]).astype(float)
txn["has_identity"] = txn["has_identity"].astype(float)


def add_history(frame, key, prefix, past_only=True):
    """Return the frame with four history features for a key. past_only=False reads all transactions (leakage)."""
    g = frame.groupby(key, sort=False)
    amt = frame["TransactionAmt"]
    if past_only:
        # Cumulative values in time order: each row sees only the earlier rows of its key.
        n_prev = g.cumcount()
        sum_prev = g["TransactionAmt"].cumsum() - amt
        mean_prev = np.where(n_prev > 0, sum_prev / n_prev.where(n_prev > 0, 1), np.nan)
        frame[f"{prefix}_n"] = n_prev
        frame[f"{prefix}_amt_ratio"] = amt / mean_prev
        frame[f"{prefix}_hours_since"] = g["TransactionDT"].diff() / 3600
        frame[f"{prefix}_n_today"] = frame.groupby([key, "day"], sort=False).cumcount()
    else:
        # Leakage version for dec-02: each row sees all rows of its key, also the later rows.
        frame[f"{prefix}_n"] = g["TransactionAmt"].transform("count")
        frame[f"{prefix}_amt_ratio"] = amt / g["TransactionAmt"].transform("mean")
        frame[f"{prefix}_hours_since"] = g["TransactionDT"].transform(
            lambda s: (s.max() - s.min()) / 3600)
        frame[f"{prefix}_n_today"] = frame.groupby([key, "day"], sort=False)[
            "TransactionAmt"].transform("count")
    return frame


# Earlier transactions of the same key: the customer proxy and the card.
for key, prefix in [("uid", "uid"), ("card1", "card")]:
    txn = add_history(txn, key, prefix)

# Feature lists. Step 4a explains each excluded column (column_status).
HIST = [c for c in txn.columns if c.startswith(("uid_", "card_"))]
EXCLUDE = {"TransactionID", "isFraud", "TransactionDT", "day", "month", "uid", "D1n"}
CAT_COLS = [c for c in txn.columns
            if c not in EXCLUDE and not pd.api.types.is_numeric_dtype(txn[c])]
NUM_COLS = [c for c in txn.columns
            if c not in EXCLUDE and c not in CAT_COLS and c not in HIST]

N.update({"n_hist": f"{len(HIST)}", "n_num": f"{len(NUM_COLS)}", "n_cat": f"{len(CAT_COLS)}",
          "n_feats": f"{len(NUM_COLS) + len(HIST) + len(CAT_COLS)}",
          "uid_first": f"{(txn['uid_n'] == 0).mean():.0%}"})
fmt(txn[HIST].describe().T[["mean", "50%", "max"]].assign(Empty=txn[HIST].isna().mean()),
    {"mean": "{:,.1f}", "50%": "{:,.1f}", "max": "{:,.0f}", "Empty": "{:.0%}"})

	mean	50%	max	Empty
uid_n	6.1	1.0	1,413	11%
uid_amt_ratio	1.2	1.0	204	45%
uid_hours_since	263.0	73.4	4,300	45%
uid_n_today	1.1	0.0	647	11%
card_n	1,263.9	312.0	14,931	0%
card_amt_ratio	1.1	0.7	146	2%
card_hours_since	47.3	1.6	4,333	2%
card_n_today	10.0	2.0	647	0%

The model gets 100 input columns: 81 numeric columns, 11 text columns and 8 history features. For 34% of the transactions, the uid has no earlier transaction. For these transactions, the amount ratio and the hours since the last transaction are empty: a structural missing value.

2e. Baseline: what does fraud cost if the merchant approves all transactions?

The data is now clean. Before we build a score, we must know the cost that the score must decrease. The simplest policy approves all transactions. It has no false declines and no reviews, so its total cost is the fraud loss.

The fraud loss of a transaction is its amount if it is fraud (as-01: the business pays 100% of the chargeback). The table gives the cost for each 30 days, so that months with a different number of days compare correctly.

# Purpose: Measure the total cost of the approve-all policy by month: the baseline of Step 2e.
# Input:   `txn`, `COST` (as-01).
# Output:  Table `by_month`. Numbers for the text and the business note.

fraud_amt = txn["TransactionAmt"].where(txn["isFraud"] == 1, 0.0)
by_month = txn.assign(fraud_amt=fraud_amt).groupby("month").agg(
    Transactions=("isFraud", "size"), Days=("day", "nunique"), Fraud_rate=("isFraud", "mean"),
    Amount=("TransactionAmt", "sum"), Fraud_amount=("fraud_amt", "sum"))
# Approve-all: each fraud dollar is a loss (as-01). Cost for each 30 days.
by_month["Loss_30"] = by_month["Fraud_amount"] * COST["loss_share"] / by_month["Days"] * 30
by_month["Fraud_share_usd"] = by_month["Fraud_amount"] / by_month["Amount"]
loss_30 = by_month["Loss_30"]
test_loss = (by_month.loc[[5, 6], "Fraud_amount"].sum() * COST["loss_share"]
             / by_month.loc[[5, 6], "Days"].sum() * 30)
N.update({
    "n_txn": f"{len(txn):,}",
    "fraud_rate": f"{txn['isFraud'].mean():.2%}",
    "fraud_amt_share": f"{fraud_amt.sum() / txn['TransactionAmt'].sum():.2%}",
    "txn_per_day": f"{txn.groupby('day').size().mean():,.0f}",
    "base_loss_month": f"${fraud_amt.sum() * COST['loss_share'] / txn['day'].nunique() * 30:,.0f}",
    "base_loss_test": f"${test_loss:,.0f}",
    "base_loss_min": f"${loss_30.min():,.0f}", "base_loss_max": f"${loss_30.max():,.0f}",
    "base_month_max": f"{loss_30.idxmax()}",
    "base_fr_min": f"{by_month['Fraud_rate'].min():.2%}", "base_fr_max": f"{by_month['Fraud_rate'].max():.2%}",
})
del fraud_amt
fmt(by_month.rename(columns={
        "Fraud_rate": "Fraud rate", "Amount": "Amount ($)", "Fraud_amount": "Fraud amount ($)",
        "Loss_30": "Fraud loss, approve-all ($ each 30 days)", "Fraud_share_usd": "Fraud share of $"}),
    {"Transactions": "{:,.0f}", "Fraud rate": "{:.2%}", "Amount ($)": "{:,.0f}",
     "Fraud amount ($)": "{:,.0f}", "Fraud loss, approve-all ($ each 30 days)": "{:,.0f}",
     "Fraud share of $": "{:.2%}"})

	Transactions	Days	Fraud rate	Amount ($)	Fraud amount ($)	Fraud loss, approve-all ($ each 30 days)	Fraud share of $
month
1	130,968	29	2.48%	16,756,862	439,590	454,748	2.62%
2	89,838	30	4.04%	11,990,567	496,493	496,493	4.14%
3	91,768	30	4.03%	12,773,159	515,459	515,459	4.04%
4	98,027	30	3.93%	13,719,550	643,237	643,237	4.69%
5	85,303	30	3.47%	11,520,251	493,821	493,821	4.29%
6	94,636	33	3.47%	12,978,561	495,244	450,222	3.82%

# Purpose: Show the baseline that the champion and the challenger must beat.
# Input:   `by_month`.
# Output:  Two panels: fraud loss for each 30 days (bars) and fraud rate (line).

import chart_style as cs
cs.setup()  # House style (safe to call again)

fig, (ax1, ax2) = plt.subplots(1, 2, figsize=(9, 3.8))
x = by_month.index.to_numpy()
loss_k = by_month["Loss_30"] / 1e3
# Left: fraud loss. The month with the highest loss in blue, the other months in navy.
top_month = int(loss_k.idxmax())
ax1.bar(x, loss_k, color=[cs.BLUE if m == top_month else cs.NAVY for m in x], width=0.6)
for m, v in loss_k.items():
    ax1.annotate(f"${v:,.0f}K", (m, v), xytext=(0, 3), textcoords="offset points", ha="center",
                 fontsize=9, color=cs.BLUE if m == top_month else cs.NAVY)
ax1.set_yticks([])
ax1.spines["left"].set_visible(False)
ax1.set_xticks(x)
ax1.set_xticklabels([f"M{m}" for m in x])
ax1.set_title("Fraud loss, $ thousands each 30 days", fontsize=9.5, color=cs.MUTED,
              fontweight="normal", loc="left")
ax1.margins(y=0.15)
# Right: fraud rate by month, with the mean of all months as context.
fr = by_month["Fraud_rate"]
ax2.plot(x, fr, "o-", color=cs.NAVY)
ax2.axhline(txn["isFraud"].mean(), color=cs.GRAY, lw=1, ls="--")
cs.label_end(ax2, x.max(), txn["isFraud"].mean(), "All months", color=cs.GRAY, dy=-8)
for m, v in fr.items():
    ax2.annotate(f"{v:.1%}", (m, v), xytext=(0, 7), textcoords="offset points", ha="center",
                 fontsize=9, color=cs.NAVY)
cs.percent_axis(ax2, decimals=1)
ax2.set_ylim(0, fr.max() * 1.3)
ax2.set_xticks(x)
ax2.set_xticklabels([f"M{m}" for m in x])
ax2.set_title("Fraud rate (share of transactions)", fontsize=9.5, color=cs.MUTED,
              fontweight="normal", loc="left")
mean_loss_k = by_month["Fraud_amount"].sum() * COST["loss_share"] / by_month["Days"].sum() * 30 / 1e3
cs.action_title(fig, f"Approve-all loses about ${mean_loss_k:,.0f}K to fraud every 30 days",
                "Months 1–6, all transactions of the sample, loss = 100% of the fraud amount (as-01)")
plt.show()

How to read: the left panel gives the fraud loss of each month if the merchant approves all transactions, for each 30 days. The right panel gives the share of fraud transactions in each month; the dashed line is the rate of all months.

So what: the fraud loss stays between $450,222 and $643,237 each 30 days, and the fraud rate stays between 2.48% and 4.04%. Fraud is a constant cost, not one bad month. Month 4 has the highest loss. On the test months 5–6, approve-all costs $470,983 each 30 days.

Business note. The data has 590,540 clean transactions, about 3,245 each day, with a fraud rate of 3.50%. Fraud has 3.87% of the dollars. Approve-all costs about $508,326 each 30 days: this is the ceiling of the fraud loss. Two facts limit what a score can promise: the customer key is a proxy, and the label of a customer can follow a later chargeback. Step 3 now splits the data by time, so that the test looks like production.

Step 3. Split data

Step 2 gave clean transactions in time order, a customer key uid and history features from earlier transactions only. It also showed the cost of a policy that approves all transactions. Before we build a score, we must agree on two things. What does “a better policy” mean, and which months give an honest test of it?

Purpose. This step separates the months that train the score from the months that test it, and it fixes the rules of the test. It supports this decision: approve the challenger policy for a test, with a result that is near to the result in production.

Steps 3 to 7 use the same small set of tools: the GBM settings, the metrics, the cost of each action and the policy rules. The code keeps them in the file case02_helpers.py, so that the story stays short. The appendix shows the full file. The next cell loads the tools and gives them the seed and the cost assumptions.

# Purpose: Load the helper functions that Steps 3–7 use (GBM, metrics, PSI, costs, policies).
# Input:   `SEED`, `COST` (setup cell), `NUM_COLS`, `HIST` (Step 2).
# Output:  The functions and constants of case02_helpers.py in the namespace of the notebook,
#          `FEATS` (the features of the GBM), `RULES` (the rules of the evaluation framework).

import chart_style as cs  # Chart style (deck style guide). A second import is safe.
import case02_helpers
from case02_helpers import *  # noqa: F401,F403

# The GBM uses the seed. The policy functions use the cost assumptions (as-01 to as-05).
configure(seed=SEED, cost=COST)
cs.setup()

# Features of the GBM: the raw and the time features, plus the history features (dec-02).
FEATS = NUM_COLS + HIST

# The rules of the evaluation framework (section 3a). We set them before the evidence.
RULES = {
    "b1_model_gain": 0.10,   # Branch 1: the GBM must have a 10% higher PR-AUC on month 4 (relative)
    "b2_level_gap": 0.005,   # Branch 2: |mean score − fraud rate| less than 0.5 points, months 5–6
    "b3_psi": 0.10,          # Branch 3: PSI of the score less than 0.1
    "b3_pr_keep": 0.80,      # Branch 3: PR-AUC on months 5–6 at 80% or more of month 4
    "b4_fdr": 0.03,          # Branch 4: decline rate of good transactions 3% or less (as-08)
    "b5_leaf_rate": 0.50,    # Branch 5: leaves with a decline rate of 50% or more ...
    "b5_leaf_share": 0.50,   # ... hold 50% or more of all declines
}
N.update({"capacity": f"{capacity():,}", "pr_random": f"{txn['isFraud'].mean():.3f}"})

3a. What the test must measure

A fraud project has many metrics. A reader can ask: “Why these metrics, and why not others?” Thus we start from the question that each metric answers.

The framework splits the risks of a new policy in two groups. The first group is statistical: the score can fail on the order of the transactions, on the level of the score, or over time. The second group is business: the score can be correct but save no money, or save money but give decisions that fraud operations cannot read. Each failure in production belongs to one branch only. A policy that passes all five branches is ready for a test.

We set the rules in this table before the rerun of this notebook, before we saw the evidence. We do not change a rule after a result. If a result fails a rule, the notebook reports the failure and what it means for the test.

Branch	Question	Metric	Rule
1. Order of risk	Does the score put fraud above good transactions better than the champion?	PR-AUC, dollar recall in the top 2% (months 5–6)	Both above the champion. Model rule: the GBM replaces the logistic regression only if its PR-AUC on month 4 is 10% or more higher (relative).
2. Level	Is the mean score equal to the actual fraud rate?	Mean score after the recalibration vs. actual fraud rate, months 5–6	A difference of less than 0.5 points
3. Stability	Does the score keep its power after the training period?	PSI of the score (month 6 vs. months 1–3); PR-AUC on months 5–6 vs. month 4	PSI less than 0.1, and PR-AUC on months 5–6 at 80% or more of month 4
4. Business value	Does the policy lower the total cost at the same review capacity?	Total cost per month vs. champion; decline rate of good transactions	Lower total cost, and a decline rate of good transactions of 3% or less (new assumption as-08: guardrail limit, Marketing to confirm)
5. Explainability	Can fraud operations read why the policy declines?	Surrogate tree, 3 levels, on the decline decision	The leaves with a decline rate of 50% or more hold 50% or more of all declines

The notebook uses each branch in these sections:

• Branch 1: Step 3b (the split), Step 4b (the model type) and Step 6a.
• Branch 2: Step 4c (the low fraud rate), Step 5a (the recalibration) and Step 6b.
• Branch 3: Step 3b and Step 6c. Part 2 adds the PSI of each top feature.
• Branch 4: Step 5 (the action tiers and the value that puts the transactions in order) and Step 6d.
• Branch 5: Step 6e.

Branches 1 and 2 are different. A score can put the transactions in the correct order but give a mean score that is too low. The order sets which transactions go to review or decline. The level sets the expected cost in dollars of each action. Thus the notebook checks both.

A fraud decline needs no adverse action notice, as a credit decline does. Branch 5 thus asks for readable rules for fraud operations, not reason codes for each customer.

Technical note. How each metric works:

• PR-AUC (average precision). Sort the transactions from the highest score. At each fraud transaction in the list, calculate the precision: the share of fraud in all transactions above it. The PR-AUC is the mean of these precisions. A random score gives the fraud rate (0.035 here), and a perfect score gives 1.0. We use the PR-AUC and not only the AUC. With few frauds, the AUC counts many easy good transactions at the bottom of the list. The policy acts only on the top of the list, and the PR-AUC measures the top.
• Dollar recall in the top 2%. Flag the 2% of transactions with the highest score. Dollar recall = fraud amount in the flagged transactions / all fraud amount. The fraud loss is in dollars, so a score that finds the large frauds is worth more. 2% is near the share of transactions that the policy can decline or review.
• Calibration (level). Compare the mean score with the actual fraud rate, for all transactions and for each score decile. The expected cost of each action multiplies the score by the amount. Thus a score that is too high declines good customers, and a score that is too low approves fraud.
• PSI. Cut the score of the reference months into ten bins of equal size. For each bin, compare the share of the reference transactions (e) with the share of the new transactions (a). PSI = Σ (a − e) × ln(a / e). The thresholds 0.1 and 0.25 are the industry convention. The PSI uses no label. Thus it cannot find a change of the fraud patterns when the population stays the same. The PR-AUC rule of branch 3 covers that case.

3b. Which period tests the model?

Which split gives a test result that is near to production?

Why this question matters. The merchant will use the score on the transactions of next month, not on the transactions of last month. If the test uses the wrong transactions, the test result can look good, and the score then fails in production. The policy then promises a decrease of the fraud loss that never comes.

Ways to do it. The split axis is time: where are the test transactions in time, compared with the training transactions? There are four possible answers, and each split method belongs to one of them.

Group	Method	Advantage	Main risk	Status
Same period	Random split, for example 80/20, k-fold cross-validation, or a split by the customer key uid	Simple. Large training set. The two sets have the same fraud rate.	Leakage. The same customer is in both sets, and the label after a chargeback (as-07) copies into the test set. The test cannot show a change over time.	Possible. Test it.
Earlier period	Train on later months, test on earlier months	None for this decision.	The model uses the future to predict the past. Production never has this data.	Screened out
Later period, one cut	Out-of-time split: train on months 1–3, validation on month 4, test on months 5–6	Same condition as production: a score from past months gives a score to new transactions. Month 4 sets the policy. Months 5–6 stay clean for the final result.	Smaller training set. The test result includes drift, so it is often lower.	Possible. Test it.
Later period, many cuts	Rolling time windows: train on months 1 to k, test on month k + 1	Shows how fast the score loses power.	More fits. Six months give few windows.	Possible. Use it as a check. Part 2 retrains on rolling windows (dec-07).

A split by uid removes part of the leakage, but uid is only a proxy for the customer. The data has no customer ID. Thus a split by uid is not a full control.

The out-of-time split needs three sets, not two. The training months fit the score. Month 4 fits all later choices: the recalibration, the thresholds of the policy and the model type. Months 5–6 give only the final result. If the thresholds also used months 5–6, the final result is too good.

Evidence. The code trains the same GBM with the random split and with the out-of-time split. It keeps the GBM of months 1–3 for the next steps. For the check of the decay, it gives the result of that GBM on month 4, month 5 and month 6 separately. These months are one, two and three months after the training period. For each test set, the code measures three branches of the framework:

• Branch 1, order of risk: the PR-AUC, the AUC, and the recall and the dollar recall in the top 2%.
• Branch 2, level: the fraud rate of the training set and of the test set.
• Branch 3, stability: the PSI of the score from the training set to the test set.

# Purpose: Compare a random split with an out-of-time split, and show the decay by month (dec-01).
# Input:   `txn` (Step 2), `FEATS`, `CAT_COLS`, `rng`.
# Output:  `train`, `valid`, `test` (the out-of-time split), `m_b` (GBM on months 1–3),
#          `score_valid` (its score on month 4), table `dec01`.

# Out-of-time split: train on months 1–3, set the policy on month 4, test on months 5–6.
train = txn[txn["month"] <= 3]
valid = txn[txn["month"] == 4]
test = txn[txn["month"] >= 5]

# Random split: 20% of all transactions of the six months go to the test set.
is_test_a = rng.random(len(txn)) < 0.2
m_a = fit_gbm(txn[~is_test_a], FEATS, CAT_COLS)
m_b = fit_gbm(train, FEATS, CAT_COLS)
score_valid = predict(m_b, valid)
score_test_b = predict(m_b, test)

# Scores of the training sets, for the PSI (branch 3).
score_train_a = predict(m_a, txn[~is_test_a])
score_train_b = predict(m_b, train)


def split_row(name, tr, te, s_tr, s_te):
    """Return one row of the table: sizes, fraud rates, metrics of branch 1 and the PSI of the score."""
    return {"Split": name, "Training transactions": len(tr), "Test transactions": len(te),
            "Fraud rate, training": tr["isFraud"].mean(), "Fraud rate, test": te["isFraud"].mean(),
            **score_metrics(te, s_te), "PSI of the score": psi(s_tr, s_te)}


is_m5 = (test["month"] == 5).to_numpy()
rows = [
    split_row("A. Random 80/20 (months 1–6 mixed)", txn[~is_test_a], txn[is_test_a],
              score_train_a, predict(m_a, txn[is_test_a])),
    split_row("B. Out-of-time, validation (month 4)", train, valid, score_train_b, score_valid),
    split_row("B. Out-of-time, test (months 5–6)", train, test, score_train_b, score_test_b),
    split_row("Check: month 5 only", train, test[is_m5], score_train_b, score_test_b[is_m5]),
    split_row("Check: month 6 only", train, test[~is_m5], score_train_b, score_test_b[~is_m5]),
]
dec01 = pd.DataFrame(rows).set_index("Split")

rnd, oot_v, oot_t = dec01.iloc[0], dec01.iloc[1], dec01.iloc[2]
N.update({
    "d1_pr_a": f"{rnd['PR-AUC']:.3f}", "d1_pr_b": f"{oot_t['PR-AUC']:.3f}",
    "d1_pr_v": f"{oot_v['PR-AUC']:.3f}",
    "d1_pr_m5": f"{dec01.iloc[3]['PR-AUC']:.3f}", "d1_pr_m6": f"{dec01.iloc[4]['PR-AUC']:.3f}",
    "d1_rec_a": f"{rnd['Recall, top 2%']:.1%}", "d1_rec_b": f"{oot_t['Recall, top 2%']:.1%}",
    "d1_fr_train": f"{train['isFraud'].mean():.2%}", "d1_fr_valid": f"{valid['isFraud'].mean():.2%}",
    "d1_fr_test": f"{test['isFraud'].mean():.2%}",
    "d1_psi_b": f"{oot_t['PSI of the score']:.3f}",
    "d1_keep": f"{oot_t['PR-AUC'] / oot_v['PR-AUC']:.0%}",
    "n_train": f"{len(train):,}", "n_valid": f"{len(valid):,}", "n_test": f"{len(test):,}",
})
fmt(dec01, {"Training transactions": "{:,.0f}", "Test transactions": "{:,.0f}",
            "Fraud rate, training": "{:.2%}", "Fraud rate, test": "{:.2%}", **METRIC_FMT,
            "PSI of the score": "{:.3f}"})

	Training transactions	Test transactions	Fraud rate, training	Fraud rate, test	PR-AUC	AUC	Recall, top 2%	Dollar recall, top 2%	PSI of the score
Split
A. Random 80/20 (months 1–6 mixed)	472,258	118,282	3.49%	3.54%	0.790	0.961	54.0%	45.5%	0.000
B. Out-of-time, validation (month 4)	312,574	98,027	3.38%	3.93%	0.574	0.907	39.3%	30.9%	0.014
B. Out-of-time, test (months 5–6)	312,574	179,939	3.38%	3.47%	0.486	0.886	37.3%	26.5%	0.004
Check: month 5 only	312,574	85,303	3.38%	3.47%	0.496	0.891	38.2%	29.5%	0.002
Check: month 6 only	312,574	94,636	3.38%	3.47%	0.478	0.882	36.7%	24.6%	0.008

How to read: each row is one test set. The rows “Check” use the GBM of months 1–3 on one later month.

So what: the random split gives a much higher PR-AUC than any later month. The PR-AUC goes down with each month after the training period.

Decision. Out-of-time split: train on months 1–3 (312,574 transactions), set the policy on month 4 (98,027), and test on months 5–6 (179,939). The result by month is a check of the speed of decay.

Why. Three reasons, from the most important:

1. The random split promises too much. It gives a PR-AUC of 0.790. The out-of-time split gives 0.486 on months 5–6. In the top 2% of scores, the random split finds 54.0% of the fraud, and the out-of-time split finds 37.3%. The random split puts later transactions of the same customer in the training set, with the label after a chargeback (as-07). Production never has this information.
2. Production gives a score to the future. The fraud rate is 3.38% in months 1–3, 3.93% in month 4 and 3.47% in months 5–6. The out-of-time split shows this change of level, so Step 5a can correct it.
3. It measures the decay of the score. The PR-AUC is 0.574 on month 4, 0.496 on month 5 and 0.478 on month 6. Months 5–6 keep 85% of the PR-AUC of month 4 (branch 3 asks for 80% or more). The PSI of the score is only 0.004. Thus the fraud patterns change faster than the population. Part 2 uses this result to plan the retrain.

What would change the decision. If the data had a true customer ID, a split by customer inside each month can add a second check. It does not replace the out-of-time split.

3c. The champion on the test months

Step 0 defines the champion policy (as-06). It is a decision tree with 3 levels, fit on month 1 with the raw and the time features. The tree does not change after month 1. The fraud rate of each leaf is the score of the champion. Inside a leaf, the larger amount comes first.

The champion uses the same three tiers and the same review capacity as the challenger: 120 alerts each day. Its thresholds come from month 4, with the same method as the challenger. Thus only the score is different. Step 5b gives the formula of the cost of each action.

Parameters of the tree: 3 levels, at least 200 transactions in each leaf, and −999 for a missing value. A tree needs a value in each cell.

# Purpose: Build the champion (as-06) and give its result on months 5–6: the bar for the challenger.
# Input:   `txn`, `NUM_COLS`, `valid`, `test`, `SEED`.
# Output:  `m1` (month 1), `champ_tree`, `champ_score()`, table `champ_by_month`.

# The champion rules: a decision tree with 3 levels on month 1. A missing value becomes −999.
m1 = txn[txn["month"] == 1]
X_m1 = encode(m1, m1, NUM_COLS, []).fillna(-999)
champ_tree = DecisionTreeClassifier(max_depth=3, min_samples_leaf=200, random_state=SEED)
champ_tree.fit(X_m1, m1["isFraud"])


def champ_score(frame):
    """Return the fraud rate of the leaf. Inside a leaf, the larger amount ranks first (as-06)."""
    leaf_rate = champ_tree.predict_proba(encode(frame, m1, NUM_COLS, []).fillna(-999))[:, 1]
    return leaf_rate + 1e-9 * frame["TransactionAmt"].rank(pct=True).to_numpy()


# Three tiers at the same review capacity. The thresholds come from month 4.
champ_rule = fit_policy("prob3", champ_score(valid), valid)
champ_act_test = apply_policy(champ_rule, champ_score(test), test)
rows = [{"Months": "5–6", **summary(test, champ_act_test)}]
for mo in [5, 6]:
    part = (test["month"] == mo).to_numpy()
    rows.append({"Months": str(mo), **summary(test[part], champ_act_test[part])})
champ_by_month = pd.DataFrame(rows).set_index("Months")

c56 = champ_by_month.iloc[0]
N.update({
    "champ_leaves": str(champ_tree.get_n_leaves()),
    "champ_cost": f"${c56['Total cost ($/month)']:,.0f}",
    "champ_stop": f"{c56['Fraud $ stopped']:.1%}",
    "champ_fdr": f"{c56['Decline rate of good transactions']:.2%}",
    "champ_alerts": f"{c56['Alerts each day (mean)']:.0f}",
    "champ_over": f"{c56['Days over capacity']:.0%}",
})
print(export_text(champ_tree, feature_names=list(X_m1.columns), decimals=1))
fmt(champ_by_month, SUM_FMT)

|--- C7 <= 2.5
|   |--- C14 <= 0.5
|   |   |--- C1 <= 2.5
|   |   |   |--- class: 0
|   |   |--- C1 >  2.5
|   |   |   |--- class: 1
|   |--- C14 >  0.5
|   |   |--- V308 <= 149.5
|   |   |   |--- class: 0
|   |   |--- V308 >  149.5
|   |   |   |--- class: 0
|--- C7 >  2.5
|   |--- V264 <= 187.6
|   |   |--- C13 <= 8.5
|   |   |   |--- class: 0
|   |   |--- C13 >  8.5
|   |   |   |--- class: 0
|   |--- V264 >  187.6
|   |   |--- class: 1

	Fraud loss ($/month)	False decline cost ($/month)	Review friction ($/month)	Total cost ($/month)	Fraud $ stopped	Decline rate of good transactions	Alerts each day (mean)	Days over capacity
Months
5–6	290,176	3,013	6,243	299,432	38.4%	0.28%	119	48%
5	296,354	2,622	5,872	304,848	40.0%	0.24%	114	37%
6	284,560	3,368	6,580	294,508	36.8%	0.31%	124	58%

How to read: the text above the table gives the rules of the champion. Each “class” line is a leaf. The table gives the cost of the champion for each month.

So what: the champion stops only 38.4% of the fraud dollars, and its alerts are more than the capacity on 48% of the days.

The tree has 7 leaves. The size of each leaf changes from day to day, so the alerts of the champion change too. All later comparisons use the same months 5–6 and the same champion.

Business note. We test each policy on months 5–6, as if they are the next two months. On these months, the champion costs $299,432 each month. It stops 38.4% of the fraud dollars and declines 0.28% of the good transactions. A challenger must cost less at the same review capacity, inside the guardrail of branch 4. Step 4 builds the challenger score and judges it with the rules of section 3a.

Step 4. Build models

Step 2 gave each transaction a customer key, a time order and 8 history features. Step 3 fixed the test: train on months 1–3, set the policy on month 4 and measure on months 5–6. This step builds the fraud score that must beat the champion. It first answers which columns the score uses (4a). Then it answers which model gives the score (4b) and how the score handles the low fraud rate (4c).

Purpose. This step builds the fraud score that the policy uses. It supports this decision: the challenger score for the champion/challenger test.

4a. Feature selection

The two files have 434 raw columns. Vesta masked the meaning of most of them. Each column in the model needs a reason that a model risk reviewer accepts. A column that leaks the label makes the test result too good, and the score then fails in production. A column that only repeats another column makes the score slower and harder to monitor.

Why a column leaves the model. The reasons form a funnel with three buckets. The split axes are the time of the information, the stability of the information and the size of the model. Each column gets the first rule that removes it, so the buckets do not overlap. A column that no rule removes is a feature.

Bucket	Question	Rule	Where
1. Not known at the decision	Does the business know the value when the transaction arrives?	1. The label (isFraud). History from later transactions is also in this bucket (see the decision below).	Step 2b, Step 4a
2. No stable risk information	Does the value describe a risk that repeats in a later month?	2. Key or identifier · 3. Time stamp or date	Step 2b
3. Removed to keep the model small	Does the column add information to the other columns?	4. V column with the same missing pattern as two kept V columns · 5. Masked identity column, not loaded	Step 1, Step 4a

Buckets 1 and 2 do not depend on the model. Bucket 3 reduces the size of the data before the model sees it. Its two rules use no label, so they cannot leak the label into the test. The rules of bucket 3 also have a cost: a removed column can carry some extra signal. The funnel accepts this cost, because the GBM must train about 15 times in this notebook.

Rule 4: the V columns. Vesta made 339 V columns from its own data. Many V columns have exactly the same number of empty cells. The columns of one such group often come from the same source of Vesta and carry the same information. The load cell of Step 1 keeps two columns from each group: the two with the most distinct values.

Technical note. For each V column, DuckDB counts the cells that are not empty (COUNT) and the distinct values (approx_count_distinct, a HyperLogLog estimate). The columns with the same count form one group. Inside each group, the code sorts the columns by the number of distinct values and keeps the first two. A column with more distinct values can make more splits in the GBM.

The rule reads no label and no month, so the test months cannot change it. The code saves the counts in a small file, so that later runs do not read the raw CSV again.

# Purpose: Count the V groups and the V columns that rule 4 removes (Step 4a funnel).
# Input:   `RAW_TXN`, `V_KEEP`, `DATA_DIR`. The raw CSV only on the first run.
# Output:  `vstat` (one row for each V column: non-empty count, distinct values, group, kept). Table by group size.

VSTAT_FILE = DATA_DIR / "interim" / "fraud_vstat.csv"
if not VSTAT_FILE.exists():
    # One pass over the raw file. The same counts as the load cell of Step 1.
    con = duckdb.connect()
    con.sql(f"""CREATE VIEW t AS SELECT * FROM read_csv('{RAW_TXN.as_posix()}',
                header = true, sample_size = -1)""")
    v_cols = [c for c in con.sql("DESCRIBE t").df()["column_name"] if c.startswith("V")]
    stats = con.sql("SELECT " + ", ".join(
        f"COUNT({c}) AS n_{c}, approx_count_distinct({c}) AS d_{c}" for c in v_cols
    ) + " FROM t").df().iloc[0]
    pd.DataFrame({"col": v_cols, "non_null": [int(stats[f"n_{c}"]) for c in v_cols],
                  "distinct": [int(stats[f"d_{c}"]) for c in v_cols]}).to_csv(VSTAT_FILE, index=False)
    con.close()

vstat = pd.read_csv(VSTAT_FILE)
# Group number: 1 = the group with the most non-empty cells.
order = sorted(vstat["non_null"].unique(), reverse=True)
vstat["group"] = vstat["non_null"].map({v: i + 1 for i, v in enumerate(order)})
vstat["group_size"] = vstat.groupby("group")["col"].transform("size")
vstat["kept"] = vstat["col"].isin(V_KEEP)
vstat["share_empty"] = 1 - vstat["non_null"] / len(txn)
# Check: the kept columns of Step 1 must be at most two in each group.
assert vstat.groupby("group")["kept"].sum().le(2).all(), "Rule 4 keeps at most two columns in each group."

v_by_size = (vstat.groupby("group").agg(size=("col", "size"), kept=("kept", "sum"))
             .groupby("size").agg(Groups=("kept", "size"), Kept=("kept", "sum")))
v_by_size["V columns"] = v_by_size.index * v_by_size["Groups"]
v_by_size["Removed"] = v_by_size["V columns"] - v_by_size["Kept"]
v_by_size.index.name = "Columns in the group"
N.update({"n_v": f"{len(vstat)}", "n_vgroups": f"{vstat['group'].nunique()}",
          "n_vkept": f"{int(vstat['kept'].sum())}", "n_vremoved": f"{int((~vstat['kept']).sum())}",
          "v_big": f"{int(vstat['group_size'].max())}",
          "v_small": f"{int(vstat['group_size'].min())}"})
v_by_size[["Groups", "V columns", "Kept", "Removed"]]

	Groups	V columns	Kept	Removed
Columns in the group
11	3	33	6	27
16	1	16	2	14
18	3	54	6	48
19	1	19	2	17
20	1	20	2	18
22	1	22	2	20
23	1	23	2	21
31	1	31	2	29
32	1	32	2	30
43	1	43	2	41
46	1	46	2	44

The 339 V columns form 15 groups. The groups have from 11 to 46 columns. In total, rule 4 keeps 30 V columns and removes 309.

Rule 5: the identity columns. Only about one quarter of the transactions have a row in the identity file. Step 1 loads the five identity columns that people can read. They give the device type, the device name, the system (id_30), the browser (id_31) and the screen size (id_33). The other id_ columns hold masked codes of the network and the device. The flag has_identity keeps the main signal of the identity file for all transactions.

History features. The last group of columns comes from Step 2d: the history features of each customer. They are the only columns that read other transactions, so they are the main risk of leakage. The decision below examines them.

Which customer history can the model use?

Why this question matters. A fraud often looks normal alone, but not against the earlier behavior of the card. For example, a fraud can have an amount ten times the usual amount, or many transactions in one hour. But the label after a chargeback (as-07) puts many fraud labels on the later transactions of the same customer. A history feature that reads these later transactions copies the label. The test then promises a fraud loss decrease that production does not give.

Ways to do it. The split axis is the set of transactions of the same key that a feature reads at the time of the decision:

Group	Advantage	Main risk	Screen
A. No transactions of the key: no history	No risk of leakage. Simple.	The score does not see a change from the normal behavior of the customer.	Test
B. Earlier transactions only	The business has the same information at the time of the decision. It shows a change of behavior.	More code. A new customer has no history. The counts grow over time (Part 2).	Test
C. All transactions, also the later ones	The highest test result. Many Kaggle solutions use it.	Leakage. It reads the future and the label after a chargeback (as-07).	Test, to measure the leakage
B2. Earlier transactions plus their labels (“earlier fraud of this customer”)	A strong signal: Step 2b shows the high fraud rate after an earlier fraud.	The business learns a label only from the chargeback, weeks later.	Screened out: the data has no chargeback dates, so the code cannot know when a label became known.

A history window of fixed length (for example 90 days) is a variant of group B. It changes the drift of the features, not the leakage, so Part 2 discusses it.

Evidence. The code trains the same GBM on months 1–3 with groups A, B and C, and scores month 4. It uses the metrics of branch 1 (Step 3a): PR-AUC and the dollar recall in the top 2% of the scores. For group C, the code makes the history features again over all transactions of each key (add_history(..., past_only=False)).

# Purpose: Compare no history, history from earlier transactions and history from all transactions (dec-02).
# Input:   `txn`, `train`, `valid`, `HIST`, `NUM_COLS`, `CAT_COLS`, `FEATS`, `score_valid`, `fit_gbm()`.
# Output:  Table `dec02` (metrics on month 4 for each group).

# Group C: the leakage version of the history features.
leak = txn[["uid", "card1", "day", "TransactionAmt", "TransactionDT"]].copy()
for key, prefix in [("uid", "uid"), ("card1", "card")]:
    leak = add_history(leak, key, prefix, past_only=False)
txn_leak = txn.drop(columns=HIST).join(leak[HIST])
del leak

m_02a = fit_gbm(train, NUM_COLS, CAT_COLS)                       # A. No history
m_02c = fit_gbm(txn_leak[txn_leak["month"] <= 3], FEATS, CAT_COLS)  # C. All transactions
# B is the GBM of Step 3 (`score_valid`): the same features as the challenger.
dec02 = pd.DataFrame([
    {"Option": "A. No history", **score_metrics(valid, predict(m_02a, valid))},
    {"Option": "B. Earlier transactions only", **score_metrics(valid, score_valid)},
    {"Option": "C. All transactions (leakage)",
     **score_metrics(valid, predict(m_02c, txn_leak[txn_leak["month"] == 4]))},
]).set_index("Option")
del txn_leak, m_02c
N.update({f"d2_pr_{k}": f"{v:.3f}" for k, v in zip("abc", dec02["PR-AUC"])})
N.update({f"d2_usd_{k}": f"{v:.1%}" for k, v in zip("abc", dec02["Dollar recall, top 2%"])})
N.update({"d2_gain_b": f"{dec02['PR-AUC'].iloc[1] / dec02['PR-AUC'].iloc[0] - 1:+.0%}",
          "d2_gain_c": f"{dec02['PR-AUC'].iloc[2] / dec02['PR-AUC'].iloc[1] - 1:+.0%}"})
fmt(dec02, METRIC_FMT)

	PR-AUC	AUC	Recall, top 2%	Dollar recall, top 2%
Option
A. No history	0.570	0.903	39.3%	32.3%
B. Earlier transactions only	0.574	0.907	39.3%	30.9%
C. All transactions (leakage)	0.599	0.916	41.4%	32.0%

Decision. Use history features from earlier transactions only (group B). Do not use history from later transactions or earlier labels.

Why. On month 4, the PR-AUC is 0.570 with no history, 0.574 with earlier transactions and 0.599 with all transactions. In the top 2% of the scores, the three groups find 32.3%, 30.9% and 32.0% of the fraud dollars.

The earlier history changes the PR-AUC by +1% against no history, and it does not increase the dollar recall. Thus the history adds little power to the score on month 4. We keep it for two other uses. They are readable, so the rules of the surrogate tree (Step 6) use them. They are also the base for a feature “earlier fraud of this customer” when the chargeback dates exist.

Group C gives a further +4% on the PR-AUC. Production cannot give these features, so this gain is leakage. The gain is small on month 4, because the model trains only on months 1–3. A random split puts the same customers in the training data and in the test, and the leakage then grows (Step 3).

What would change the decision. If the surrogate tree and the fraud analysts do not use the history features, then group A is simpler and as strong. Then remove the history features.

# Purpose: Give each raw column of the two files and each engineered feature its bucket, rule and detail.
# Input:   `RAW_TXN`, `RAW_ID`, `txn`, `vstat`, `V_KEEP`, `NUM_COLS`, `CAT_COLS`, `HIST`.
# Output:  `column_status` (columns: column, group, bucket, rule, detail; contract C4), funnel table.

RULE_BUCKET = {
    "1. Label": "1. Not known at the decision",
    "2. Key or identifier": "2. No stable risk information",
    "3. Time stamp or date": "2. No stable risk information",
    "4. V column with the same missing pattern": "3. Removed to keep the model small",
    "5. Masked identity column, not loaded": "3. Removed to keep the model small",
    "Feature": "Feature",
}
# Raw column names: the header rows of the two files only.
if RAW_TXN.exists() and RAW_ID.exists():
    raw_txn_cols = list(pd.read_csv(RAW_TXN, nrows=0).columns)
    raw_id_cols = [c for c in pd.read_csv(RAW_ID, nrows=0).columns if c != "TransactionID"]
else:  # Only the Parquet file exists: the names follow the Vesta pattern.
    raw_txn_cols = (["TransactionID", "isFraud", "TransactionDT", "TransactionAmt", "ProductCD"]
                    + [f"card{i}" for i in range(1, 7)] + ["addr1", "addr2", "dist1", "dist2"]
                    + ["P_emaildomain", "R_emaildomain"] + [f"C{i}" for i in range(1, 15)]
                    + [f"D{i}" for i in range(1, 16)] + [f"M{i}" for i in range(1, 10)]
                    + [f"V{i}" for i in range(1, 340)])
    raw_id_cols = [f"id_{i:02d}" for i in range(1, 39)] + ["DeviceType", "DeviceInfo"]
ENGINEERED = ["day", "month", "hour", "D1n", "uid", "log_amt", "cents", "email_match", "has_identity"] + HIST
FEATURE_SET = set(NUM_COLS) | set(CAT_COLS) | set(HIST)


def col_group(c):
    """Column group of one column (the groups of Step 1b and Step 2d)."""
    if c in ("TransactionID", "isFraud"):
        return "Key and label"
    if c in ("TransactionDT", "day", "month", "hour"):
        return "Time"
    if c in HIST:
        return "Engineered: history"
    if c in ("D1n", "uid"):
        return "Engineered: customer key"
    if c in ("log_amt", "cents", "email_match", "has_identity"):
        return "Engineered: same transaction"
    for prefix, name in [("card", "Card"), ("addr", "Address"), ("dist", "Distance"), ("C", "C: counts"),
                         ("D", "D: time deltas"), ("M", "M: match flags"), ("V", "V: Vesta features"),
                         ("id_", "Identity: id_"), ("Device", "Identity: device")]:
        if c.startswith(prefix) and (prefix not in ("C", "D", "M", "V") or c[1:].isdigit()):
            return name
    if c.endswith("emaildomain"):
        return "Email domain"
    return {"TransactionAmt": "Amount", "ProductCD": "Product"}.get(c, "Other")


def col_rule(c):
    """The first rule of the funnel that removes the column, or Feature."""
    if c == "isFraud":
        return "1. Label"
    if c in ("TransactionID", "uid"):
        return "2. Key or identifier"
    if c in ("TransactionDT", "day", "month", "D1n"):
        return "3. Time stamp or date"
    if c in FEATURE_SET:
        return "Feature"
    if c.startswith("V"):
        return "4. V column with the same missing pattern"
    if c in raw_id_cols:
        return "5. Masked identity column, not loaded"
    raise ValueError(f"No rule for column {c}")


# Empty share of the identity columns that Step 1 does not load: from the identity file (144,000 rows).
id_not_loaded = [c for c in raw_id_cols if c not in txn.columns]
id_nonempty = (pd.read_csv(RAW_ID, usecols=id_not_loaded).notna().sum()
               if RAW_ID.exists() else pd.Series(dtype=float))
v_info = vstat.set_index("col")
v_kept_in = vstat[vstat["kept"]].groupby("group")["col"].agg(", ".join)
DETAIL = {
    "isFraud": "Target. Includes later transactions after a chargeback (as-07)",
    "TransactionID": "Key of both files",
    "uid": "Customer proxy (card1 + addr1 + D1n); key of the history features",
    "TransactionDT": "Time order of the rows and the split (Step 3)",
    "day": "Day number; history and capacity by day", "month": "Split and results by month (Step 3)",
    "D1n": "First day of the card; a date that moves each month (Step 2b, Part 2)",
}


def col_detail(c, rule):
    """Short text with the evidence for the status of one column."""
    if c in DETAIL:
        return DETAIL[c]
    if c in v_info.index:
        r = v_info.loc[c]
        text = (f"Group {r['group']} ({r['group_size']} columns, {r['share_empty']:.0%} empty); "
                f"{r['distinct']:,} distinct values")
        return text if r["kept"] else text + f"; group keeps {v_kept_in.get(r['group'], 'none')}"
    if rule.startswith("5."):
        return (f"Not empty for {id_nonempty[c] / len(txn):.1%} of the transactions"
                if c in id_nonempty.index else "Masked code")
    kind = "text" if c in CAT_COLS else "history" if c in HIST else "numeric"
    return f"{kind}; {txn[c].isna().mean():.0%} empty"


cols = list(dict.fromkeys(raw_txn_cols + raw_id_cols + ENGINEERED))
column_status = pd.DataFrame({"column": cols})
column_status["group"] = column_status["column"].map(col_group)
column_status["rule"] = column_status["column"].map(col_rule)
column_status["bucket"] = column_status["rule"].map(RULE_BUCKET)
column_status["detail"] = [col_detail(c, r) for c, r in zip(column_status["column"], column_status["rule"])]
column_status = column_status[["column", "group", "bucket", "rule", "detail"]]
# Checks: each model input has the status Feature, and each feature is an input of the model.
assert set(column_status.loc[column_status["rule"] == "Feature", "column"]) == FEATURE_SET
assert column_status["column"].is_unique

feat = column_status[column_status["rule"] == "Feature"]
N.update({"n_features": f"{len(feat)}",
          "n_feat_raw": f"{int((~feat['column'].isin(ENGINEERED)).sum())}",
          "n_feat_eng": f"{int(feat['column'].isin(ENGINEERED).sum())}",
          "n_id_out": f"{len(id_not_loaded)}"})
# Funnel table: the columns that each rule removes, by bucket.
funnel = (column_status.groupby(["bucket", "rule"])["column"]
          .agg(Columns="size", Examples=lambda c: ", ".join(list(c)[:5])).reset_index()
          .rename(columns={"bucket": "Bucket", "rule": "Rule"}).set_index(["Bucket", "Rule"]))
funnel

		Columns	Examples
Bucket	Rule
1. Not known at the decision	1. Label	1	isFraud
2. No stable risk information	2. Key or identifier	2	TransactionID, uid
	3. Time stamp or date	4	TransactionDT, day, month, D1n
3. Removed to keep the model small	4. V column with the same missing pattern	309	V1, V2, V3, V4, V5
	5. Masked identity column, not loaded	35	id_01, id_02, id_03, id_04, id_05
Feature	Feature	100	TransactionAmt, ProductCD, card1, card2, card3

The funnel reads 434 raw columns of the two files and the engineered columns of Step 2. It keeps 100 features: 87 raw columns and 13 engineered features. Rule 5 removes 35 masked identity columns. Appendix A gives the status of each column group and of each column that the model uses.

4b. Model type

Section 4a fixed the columns of the score, with history features from earlier transactions only. The champion uses the same raw columns in a tree with 3 levels. The next question is: which type of model gives the score, and is the extra power worth the cost?

Which model gives the fraud score?

Why this question matters. The model type sets how much fraud the score finds in the top of the list, where the policy acts. It also sets how fast the score runs in the payment flow and how easily fraud operations can read the decisions. A model that finds more fraud but cannot score a payment in milliseconds cannot go into production.

Ways to do it. The first split axis is the label: does the model learn from the fraud label? The models that learn from the label then split by how they make a decision. These five groups cover the common models, and each model belongs to one group. Each group then goes through three filters:

• Predictive power on this table. The data has 312,574 training transactions, text features with many levels, and many missing values.
• Readable for fraud operations. A fraud decline needs no adverse action notice. But fraud operations must find new fraud patterns and answer a partner who asks “why this decline?” (branch 5).
• Deployment and control. The model must give a score in real time, the same score each day, and the team must retrain it each month (Part 2).

Group: how the model decides	Models	Predictive power	Readable	Deployment	Result
No label: a transaction is fraud if it is far from normal transactions	Isolation forest, autoencoder	Low. Many frauds look like normal purchases. The data has the label, so a method without it throws away information.	Poor. “Far from normal” is not a reason.	Simple.	Screened out
Distance or geometry: a transaction is fraud if it is near past frauds or on the wrong side of a boundary	KNN, SVM	Medium. They need scaled numbers and no missing values.	Poor.	KNN keeps all training transactions in the system. It is too slow for real time.	Screened out
Connectionist: many layers of weights learn new features	Neural networks	Strong on sequences of transactions. On this table, usually not better than tree ensembles.	Poor. A black box.	More effort to fit, to monitor and to retrain each month.	Screened out
Trees and rules: the model splits the transactions by rules	Decision tree, random forest, GBM	GBM: usually the best on tables with missing values. One tree: weak, but it is the champion (as-06). Random forest: strong, but usually below the GBM.	One tree: strong. GBM: medium. Step 6e adds readable rules with a surrogate tree.	Standard in fraud systems. Fast.	Keep the GBM. Keep the tree as the champion.
Parametric: one equation with one weight for each feature	Naive Bayes, logistic regression	Naive Bayes assumes that the features are independent. That is not true here (the count features move together). Logistic regression: good on a straight line in each feature.	Strong in principle. But Vesta masked most features, so a weight has little meaning.	Simple and fast.	Keep the logistic regression

The shortlist has three models: the champion tree, the logistic regression and the GBM. The logistic regression is the simple challenger: easy to deploy, but it adds the effect of each feature without interactions. The GBM finds interactions, for example “a new card and a large amount at night”. It also uses a missing value as information.

The rule before the evidence (branch 1). The challenger score is the logistic regression, unless the GBM has a PR-AUC on month 4 that is 10% or more higher (relative). We set this rule in section 3a, before the rerun. A more complex model must earn its place with a clear gain.

Technical note. Settings of the two challenger models. Both use the training months 1–3 and the same features: the raw and the time features plus the history features.

• Logistic regression. For each number: fill a missing value with the median and add a flag for “missing”. Then change the number to a normal distribution with a quantile transform (100 quantiles). For each text feature: one column for each level with 200 or more transactions. Regularization C = 0.1.
• GBM. The settings are in GBM_PARAMS (case02_helpers.py). The learning rate is 0.08, with at most 400 trees. Each tree has at most 63 leaves, and each leaf has at least 100 transactions. The L2 regularization is 1.0. An early stop uses 10% of the training rows. A text feature with 200 levels or fewer becomes a category. A text feature with more levels becomes its frequency in the training data. The GBM has no imputation: it learns the best side of each split for a missing value.
• Champion. The tree of Step 3c (as-06), on month 1 only.

Evidence. The code trains the logistic regression and compares the three models on month 4, the validation month. The months 5–6 stay clean for Step 6. The table gives branch 1: the PR-AUC, the AUC, and the recall and the dollar recall in the top 2%.

# Purpose: Compare the champion, the logistic regression and the GBM on month 4, and apply the
#          model rule of branch 1 (dec-03).
# Input:   `train`, `valid`, `champ_score()`, `score_valid` (GBM, Step 3b), `NUM_COLS`, `HIST`, `CAT_COLS`.
# Output:  `lr`, `lr_cols`, `score_valid_lr`, table `dec03`, `d3_gain` (relative PR-AUC gain of the GBM).

# Logistic regression: quantile transform for numbers, one column for each frequent level of a text feature.
num_lr = NUM_COLS + HIST
lr = make_pipeline(
    ColumnTransformer([
        ("num", make_pipeline(SimpleImputer(strategy="median", add_indicator=True),
                              QuantileTransformer(n_quantiles=100, output_distribution="normal",
                                                  subsample=100_000, random_state=SEED)), num_lr),
        ("cat", make_pipeline(SimpleImputer(strategy="constant", fill_value="missing"),
                              OneHotEncoder(handle_unknown="infrequent_if_exist",
                                            min_frequency=200)), CAT_COLS),
    ]),
    LogisticRegression(C=0.1, max_iter=1000),
)
lr_cols = num_lr + CAT_COLS
lr.fit(train[lr_cols].astype({c: object for c in CAT_COLS}), train["isFraud"])
score_valid_lr = lr.predict_proba(valid[lr_cols].astype({c: object for c in CAT_COLS}))[:, 1]
score_valid_champ = champ_score(valid)

dec03 = pd.DataFrame([
    {"Model": "Champion: decision tree, 3 levels (month 1)", **score_metrics(valid, score_valid_champ)},
    {"Model": "Logistic regression", **score_metrics(valid, score_valid_lr)},
    {"Model": "GBM", **score_metrics(valid, score_valid)},
]).set_index("Model")

# The model rule of branch 1, set before the evidence: the GBM needs a 10% higher PR-AUC (relative).
d3_gain = dec03.iloc[2]["PR-AUC"] / dec03.iloc[1]["PR-AUC"] - 1
d3_pass = d3_gain >= RULES["b1_model_gain"]
N.update({f"d3_pr_{k}": f"{v:.3f}" for k, v in zip("abc", dec03["PR-AUC"])})
N.update({f"d3_usd_{k}": f"{v:.1%}" for k, v in zip("abc", dec03["Dollar recall, top 2%"])})
N.update({"d3_gain": f"{d3_gain:.0%}", "d3_rule": "passes" if d3_pass else "fails",
          "d3_vs_rule": "more than" if d3_pass else "less than",
          "d4_one_in": f"{round(1 / train['isFraud'].mean())}"})
fmt(dec03, METRIC_FMT)

	PR-AUC	AUC	Recall, top 2%	Dollar recall, top 2%
Model
Champion: decision tree, 3 levels (month 1)	0.268	0.733	24.1%	17.2%
Logistic regression	0.329	0.850	26.1%	16.1%
GBM	0.574	0.907	39.3%	30.9%

The chart shows the same comparison for each share of transactions that the policy can flag, from 0% to 10%. A policy declines or reviews only a small share of the transactions, so the left part of the chart matters most.

# Purpose: Show the dollar recall of each model for each share of transactions flagged (dec-03).
# Input:   `valid`, `score_valid_champ`, `score_valid_lr`, `score_valid`, `dollar_recall_curve()`.
# Output:  Chart of the dollar recall on month 4.

shares = np.linspace(0.001, 0.10, 100)
curves = {"Champion": (score_valid_champ, cs.GRAY), "Logistic regression": (score_valid_lr, cs.CYAN),
          "GBM": (score_valid, cs.NAVY)}
at2 = {name: dollar_recall_curve(valid, s, [0.02])[0] for name, (s, _) in curves.items()}

fig, ax = plt.subplots(figsize=(8, 4))
for name, (s, color) in curves.items():
    y = dollar_recall_curve(valid, s, shares)
    ax.plot(shares, y, color=color, lw=2.2 if name == "GBM" else 1.6)
    # The logistic regression and the champion end near the same point. A vertical offset keeps the labels apart.
    dy = {"Logistic regression": 7, "Champion": -7}.get(name, 0)
    cs.label_end(ax, shares[-1], y[-1], name, color=color, dy=dy, bold=(name == "GBM"))
# The point that the framework uses: the top 2% of scores.
ax.axvline(0.02, color=cs.LIGHT_GRAY, lw=1, ls="--")
ax.plot([0.02], [at2["GBM"]], "o", color=cs.BLUE)
ax.annotate(f"Top 2%: {at2['GBM']:.0%}", (0.02, at2["GBM"]), xytext=(6, 8), textcoords="offset points",
            color=cs.BLUE, fontsize=9)
ax.set_xlim(0, 0.125)
ax.set_ylim(0, None)
ax.set_xlabel("Share of transactions flagged (highest scores first)")
ax.set_ylabel("Share of the fraud dollars found")
cs.percent_axis(ax, "x")
cs.percent_axis(ax, "y")
cs.action_title(fig, f"In the top 2%, the GBM finds {at2['GBM']:.0%} of fraud dollars, "
                     f"the champion {at2['Champion']:.0%}",
                "Dollar recall against the share of transactions flagged, month 4 (validation)")
plt.show()

How to read: each line starts at the transaction with the highest score and adds transactions in the order of the score. The height is the share of all fraud dollars of month 4 that the flagged transactions hold.

So what: at the top 2%, the GBM finds 30.9% of the fraud dollars, against 17.2% for the champion. Thus the GBM finds more fraud dollars for the same review and decline work.

Decision. The challenger score is the GBM. The GBM passes the model rule of branch 1 that we set before the evidence. The logistic regression stays as a check.

Why. On month 4, the PR-AUC is 0.268 for the champion, 0.329 for the logistic regression and 0.574 for the GBM. Thus the GBM is 75% higher than the logistic regression, more than the limit of 10% in the rule.

The business result agrees. In the top 2% of scores, the GBM finds 30.9% of the fraud dollars. The champion finds 17.2% and the logistic regression finds 16.1%.

The GBM is harder to read than a tree or a weight table. That cost is smaller here than in a credit decision:

• A fraud decline needs no adverse action notice. Fraud operations need readable patterns, not a reason for each customer.
• Vesta masked most features, so the weights of the logistic regression are not readable either.
• Step 6e gives readable rules with a surrogate tree on the decline decision (branch 5).

What would change the decision. A regulator or a partner can require a reason for each decline. Then use the logistic regression for the decline tier and the GBM for the review tier.

4c. Low fraud rate: class imbalance and the level of the score

The GBM puts the fraud transactions at the top of the list better than the other models. But Step 5 multiplies the score by the amount to get the expected cost in dollars. Thus the level of the score must also be correct (branch 2). Two different problems can make the level wrong, and teams often mix them.

1. Class imbalance. Only about one transaction in 30 is fraud. Some teams then give more weight to fraud, with a class weight, with fewer good transactions in the training data, or with synthetic fraud (SMOTE). These methods make the model behave as if fraud is more common than it is. Thus they push the score up, far above the real fraud rate.
2. A change of level over time. The fraud rate is 3.38% in months 1–3 and 3.93% in month 4 (Step 3b). A model that learns from months 1–3 gives a score at the level of months 1–3.

The second problem is the more dangerous one, because it hides in the money. The chain is:

A level that is too low → expected fraud loss too low → too many approvals of fraud. A level that is too high → expected cost of an approval too high → too many declines of good customers.

How do we handle the low fraud rate?

Why this question matters. The score enters the expected cost of each action in dollars (Step 5b). A score that is too high declines good customers. A score that is too low approves fraud.

Ways to do it. The split axis is where the correction happens: in the training data, in the model, or after the model. These three places cover all methods.

Where	Method	Advantage	Main risk	Status
Training data	A random sample of the good transactions: 4 good transactions for each fraud	Faster training. Balanced classes.	It removes data. The score becomes too high.	Test it
Training data	SMOTE: synthetic fraud from the fraud transactions	Balanced classes.	Synthetic rows from text features and missing values have no clear meaning. The score becomes too high.	Screened out: the same effect as the sample, with more risk
Model	Class weight “balanced”: each fraud counts more in the fit	More weight to fraud.	The score becomes too high. It needs a recalibration after the fit.	Test it
After the model	No change in the fit. Then move the level of the score to the fraud rate of the most recent month (recalibration).	The score stays near the real rate. One constant to explain.	It needs a recent month with known labels.	Test it

Evidence. The code trains the same GBM again on months 1–3, one time with the class weight and one time on the sample. It gives the PR-AUC (branch 1), the mean score and the actual fraud rate on month 4 (branch 2). The chart then compares the mean score with the actual fraud rate for each score decile.

# Purpose: Compare three ways to handle the low fraud rate on month 4 (dec-04).
# Input:   `train`, `valid`, `score_valid` (no change), `FEATS`, `CAT_COLS`, `rng`.
# Output:  `m_04b`, `m_04c`, `score_valid_w` (class weight), table `dec04`.

# The sample: all fraud of months 1–3 and 4 random good transactions for each fraud.
fraud_idx = train.index[train["isFraud"] == 1]
good_idx = train.index[train["isFraud"] == 0]
under = train.loc[np.r_[fraud_idx, rng.choice(good_idx, 4 * len(fraud_idx), replace=False)]]
m_04b = fit_gbm(train, FEATS, CAT_COLS, class_weight="balanced")
m_04c = fit_gbm(train, FEATS, CAT_COLS, sample=under)
score_valid_w = predict(m_04b, valid)

rows = []
for name, s in [("A. No change", score_valid), ("B. Class weight", score_valid_w),
                ("C. Sample, 1 fraud for 4 good", predict(m_04c, valid))]:
    rows.append({"Option": name, "PR-AUC": average_precision_score(valid["isFraud"], s),
                 "Mean score": s.mean(), "Actual fraud rate": valid["isFraud"].mean()})
dec04 = pd.DataFrame(rows).set_index("Option")
N.update({"d4_mean_a": f"{dec04.iloc[0]['Mean score']:.2%}",
          "d4_mean_b": f"{dec04.iloc[1]['Mean score']:.1%}",
          "d4_mean_c": f"{dec04.iloc[2]['Mean score']:.1%}",
          "d4_actual": f"{valid['isFraud'].mean():.2%}",
          "d4_pr": " / ".join(f"{v:.3f}" for v in dec04["PR-AUC"])})
fmt(dec04, {"PR-AUC": "{:.3f}", "Mean score": "{:.2%}", "Actual fraud rate": "{:.2%}"})

	PR-AUC	Mean score	Actual fraud rate
Option
A. No change	0.574	2.68%	3.93%
B. Class weight	0.542	11.82%	3.93%
C. Sample, 1 fraud for 4 good	0.557	8.13%	3.93%

# Purpose: Show the calibration by score decile, with no change and with the class weight (dec-04).
# Input:   `valid`, `score_valid`, `score_valid_w`.
# Output:  Calibration chart, table `calib04`.

# Deciles of the score with no change. Decile 10 holds the 10% of transactions with the highest score.
calib04 = pd.DataFrame({
    "decile": pd.qcut(pd.Series(score_valid).rank(method="first"), 10, labels=False) + 1,
    "Actual": valid["isFraud"].to_numpy(),
    "No change": score_valid,
    "Class weight": score_valid_w,
}).groupby("decile").mean()
top = calib04.loc[10]

fig, ax = plt.subplots(figsize=(8, 4))
styles = {"Actual": (cs.GRAY, "o", "Actual fraud rate"), "No change": (cs.NAVY, "s", "GBM, no change"),
          "Class weight": (cs.BLUE, "^", "GBM, class weight")}
for col, (color, marker, label) in styles.items():
    ax.plot(calib04.index, calib04[col], marker=marker, color=color, lw=1.5 if col == "Actual" else 2)
    # "Actual" and "No change" end near the same point. A vertical offset keeps the labels apart.
    dy = {"Actual": 7, "No change": -7, "Class weight": 0}[col]
    cs.label_end(ax, calib04.index[-1], calib04[col].iloc[-1], label, color=color, dy=dy,
                 bold=(col == "Class weight"))
ax.set_xlabel("Score decile of the GBM with no change (10 = highest score)")
ax.set_ylabel("Fraud rate or mean score")
ax.set_xticks(calib04.index)
ax.set_xlim(0.5, 12.9)
cs.percent_axis(ax)
cs.action_title(fig, f"A class weight puts the mean score at {score_valid_w.mean():.0%} against "
                     f"{valid['isFraud'].mean():.1%} actual fraud",
                "Mean score and actual fraud rate by score decile, month 4 (validation)")
plt.show()

N.update({"d4_top_actual": f"{top['Actual']:.1%}", "d4_top_none": f"{top['No change']:.1%}",
          "d4_top_w": f"{top['Class weight']:.1%}"})
fmt(calib04, {c: "{:.2%}" for c in calib04.columns})

	Actual	No change	Class weight
decile
1	0.29%	0.04%	0.56%
2	0.35%	0.09%	1.27%
3	0.36%	0.15%	2.02%
4	0.42%	0.23%	2.99%
5	0.74%	0.33%	4.27%
6	0.83%	0.48%	6.15%
7	1.42%	0.72%	8.98%
8	2.10%	1.14%	13.91%
9	4.72%	2.25%	23.51%
10	28.04%	21.42%	54.56%

How to read: each point is one tenth of the transactions of month 4, in the order of the GBM score. The gray line is the actual fraud rate. The navy line is the GBM with no change. The blue line is the GBM with the class weight.

So what: the class weight puts the score above the actual fraud rate in each decile. With no change, the score is below the actual fraud rate in each decile.

Decision. No class weight, no sample and no SMOTE. In Step 5a, move the level of the score to the fraud rate of month 4 (recalibration).

Why. The PR-AUC of options A, B and C is 0.574 / 0.542 / 0.557. Thus the order of the transactions changes little (branch 1). But the mean score is 11.8% with the class weight and 8.1% with the sample. The actual fraud rate is 3.93% (branch 2 fails for both).

In the top decile, the class weight gives a mean score of 54.6% against an actual fraud rate of 28.0%. The policy in Step 5 then declines too many good customers.

With no change, the mean score is 2.68%, below the actual fraud rate in each decile. The order is correct, but the level is too low. One constant on the log-odds scale corrects the level and keeps the order. Step 5a applies it.

What would change the decision. If the policy uses only the order of the transactions, for example a fixed number of alerts each day, the level has no effect. Then a class weight or a sample can be useful if it increases the PR-AUC.

Business note. The challenger score is a GBM on the raw, time and history features. On month 4, its PR-AUC is 0.574, against 0.268 for the champion rules. In the top 2%, it finds 30.9% of the fraud dollars. Its level is too low for month 4, and the policy needs a correct level. Step 5 corrects the level and changes the score into an action for each transaction.

Step 5. Make predictions and actions

Step 4 selected the GBM as the challenger score. It left two open points. The level of the score is too low for month 4, and a score alone does not stop fraud. This step closes both points: what score does each transaction get, and what action does the policy take on it?

Purpose. This step gives each transaction a fraud score at the correct level and one action: approve, review or decline. It supports this decision: the challenger policy for the champion/challenger test.

5a. Recalibration of the score

Section 4c found that the mean score of the GBM is 2.68% on month 4, but the actual fraud rate is 3.93%. If we do not correct this, the expected fraud loss of an approval is too low. The policy then approves fraud that a review or a decline can stop.

The correction uses one constant on the log-odds scale. The code adds the constant to the log-odds of each score. The constant makes the mean score of month 4 equal to the fraud rate of month 4. Months 5–6 then check the result. The recalibration does not change the order of the transactions. Thus the PR-AUC and the dollar recall do not change.

Technical note. Score after the recalibration = 1 / (1 + exp(−(ln(p / (1 − p)) + δ))), with the function shift_logodds(). The code finds the constant δ (DELTA) with a root search (Brent’s method) between −5 and 5. On the log-odds scale, one constant keeps the order of the transactions and keeps each score between 0 and 1.

In production, the team uses the most recent month with known labels. Here month 4 has this role. Months 5–6 are an honest check, because the constant never sees them.

# Purpose: Move the score to the fraud rate of month 4 with one constant on the log-odds scale (dec-04).
# Input:   `score_valid` (GBM on month 4), `m_b`, `valid`, `test`, `shift_logodds()` (case02_helpers.py).
# Output:  `DELTA` (the constant), `p_valid` (score of month 4), `p_test` (score of months 5–6), table `recal`.

# Find the constant that makes the mean score of month 4 equal to its fraud rate.
DELTA = brentq(lambda d: shift_logodds(score_valid, d).mean() - valid["isFraud"].mean(), -5, 5)
p_valid = shift_logodds(score_valid, DELTA)
score_test = predict(m_b, test)  # Score of months 5–6 before the recalibration
p_test = shift_logodds(score_test, DELTA)

recal = pd.DataFrame({
    "Sample": ["Month 4 (calibration)", "Months 5–6 (check)"],
    "Actual fraud rate": [valid["isFraud"].mean(), test["isFraud"].mean()],
    "Mean score before": [score_valid.mean(), score_test.mean()],
    "Mean score after": [p_valid.mean(), p_test.mean()],
}).set_index("Sample")
gap_test = p_test.mean() - test["isFraud"].mean()
N.update({"delta": f"{DELTA:+.2f}", "calib_before": f"{score_test.mean():.2%}",
          "calib_test": f"{p_test.mean():.2%}", "fraud_test": f"{test['isFraud'].mean():.2%}",
          "calib_gap": f"{abs(gap_test) * 100:.2f}",
          "calib_vs_rule": "less than" if abs(gap_test) < RULES["b2_level_gap"] else "more than"})
fmt(recal, {c: "{:.2%}" for c in recal.columns})

	Actual fraud rate	Mean score before	Mean score after
Sample
Month 4 (calibration)	3.93%	2.68%	3.93%
Months 5–6 (check)	3.47%	2.31%	3.41%

The constant is +0.69 on the log-odds scale. On months 5–6, the mean score goes from 2.31% before the recalibration to 3.41% after it. The actual fraud rate is 3.47%. The difference is 0.06 points, less than the limit of 0.5 points of branch 2. Step 6b gives the full check by score decile.

5b. Cost of each action

The score tells how likely a fraud is. The business decision is about money. Thus each action gets an expected cost in dollars, from the score and the amount. Each action has a different type of cost:

• Approve. If the transaction is fraud, the merchant pays the chargeback (as-01).
• Decline. If the transaction is good, the merchant loses the margin and the customer has friction (as-02).
• Review. An analyst examines the transaction before the approval. The analyst stops most of the fraud, and a good customer waits a short time (as-05).

The analysts have a fixed cost for each day (as-04). Thus a review has no extra analyst cost inside the review capacity. The next cell lists the cost assumptions and gives an example for two amounts with the same score.

# Purpose: Show the cost assumptions (as-01 to as-05) and the expected cost of each action for an example.
# Input:   `COST`, `expected_costs()`, `capacity()` (case02_helpers.py).
# Output:  Table `cost_example`. N values for the text of Step 5b.

EX_P = 0.10  # The fraud score of the example
EX_AMT = [50.0, 1000.0]  # A small and a large amount
cost_example = pd.DataFrame([
    dict(zip(["Amount ($)", "Approve ($)", "Decline ($)", "Review ($)"],
             [amt, *[float(v) for v in expected_costs(EX_P, amt)]]))
    for amt in EX_AMT
]).set_index("Amount ($)")
cost_example["Cheapest without review"] = np.where(
    cost_example["Approve ($)"] <= cost_example["Decline ($)"], "Approve", "Decline")
cost_example["Decrease from a review ($)"] = (
    cost_example[["Approve ($)", "Decline ($)"]].min(axis=1) - cost_example["Review ($)"])

N.update({
    "loss_share": f"{COST['loss_share']:.0%}", "fd_margin": f"{COST['fd_margin']:.0%}",
    "fd_friction": f"${COST['fd_friction']:.0f}", "review_catch": f"{COST['review_catch']:.0%}",
    "review_friction": f"${COST['review_friction']:.0f}", "analysts": str(COST["analysts"]),
    "alerts_per_analyst": str(COST["alerts_per_analyst"]),
    "ex_p": f"{EX_P:.0%}", "ex_s": f"${EX_AMT[0]:,.0f}", "ex_l": f"${EX_AMT[1]:,.0f}",
    "ex_gain_50": f"${cost_example.iloc[0]['Decrease from a review ($)']:,.2f}",
    "ex_gain_1000": f"${cost_example.iloc[1]['Decrease from a review ($)']:,.2f}",
    "ex_act_50": cost_example.iloc[0]["Cheapest without review"].lower(),
    "ex_act_1000": cost_example.iloc[1]["Cheapest without review"].lower(),
})
fmt(cost_example.reset_index(), {"Amount ($)": "{:,.0f}", "Approve ($)": "{:,.2f}", "Decline ($)": "{:,.2f}",
                                 "Review ($)": "{:,.2f}", "Decrease from a review ($)": "{:,.2f}"})

	Amount ($)	Approve ($)	Decline ($)	Review ($)	Cheapest without review	Decrease from a review ($)
0	50	5.00	10.35	2.30	Approve	2.70
1	1,000	100.00	36.00	11.80	Decline	24.20

How to read: each row is one transaction with a fraud score of 10%. The columns give the expected cost of each action.

So what: with the same score, the cheapest action is different for the two amounts. A review saves more on the large transaction.

Without a review, the cheapest action is to approve the $50 transaction and to decline the $1,000 transaction. A review decreases the expected cost by $24.20 for the large transaction and by $2.70 for the small one. Thus the amount changes the action, not only the score. Section 5d uses this result.

Technical note. For a transaction with amount \(a\) and fraud score \(p\), the expected cost of each action is:

Action	Expected cost	Actual cost with the label \(y\)
Approve	\(p \cdot a \cdot L\)	\(y \cdot a \cdot L\)
Decline	\((1-p)(m \cdot a + f)\)	\((1-y)(m \cdot a + f)\)
Review	\(p \cdot a \cdot L (1-q) + (1-p) f_r\)	\(y \cdot a \cdot L (1-q) + (1-y) f_r\)

The symbols and their values are:

• \(L\): the loss share, 100% of the amount (as-01).
• \(m\) and \(f\): the costs of a false decline, 3% of the amount and $10 (as-02).
• \(q\): the catch rate of a review, 90% (as-05).
• \(f_r\): the friction cost of a review for a good customer, $2 (as-05).

The functions expected_costs() and actual_cost() in case02_helpers.py apply these formulas. The total cost of a policy is the sum of the actual costs. It has three parts: fraud loss, false decline cost and review friction. The function summary() gives each part for each month.

All policies in this step use the same review capacity: 3 analysts × 40 alerts = 120 alerts each day (as-03). A real-time decision cannot wait for the end of the day. Thus the policy uses a threshold, not a daily list of the alerts. The code sets the threshold on month 4, so that the mean number of alerts each day equals the capacity (threshold_for_capacity()).

5c. Action tiers

How many actions does the policy have?

Why this question matters. Each extra action costs money to run, but it can decrease the total cost. The number of actions sets if the merchant needs an analyst team, and how large.

Ways to do it. The split axis is the number of actions that the policy can take on a transaction. The answers are one, two, three, or more than three actions.

Actions	Method	Advantage	Main risk	Status
One	Approve all transactions	No model and no team.	The merchant pays all fraud.	Reference: the baseline of Step 2e
Two	Approve or decline, with one threshold on the score	Simple. No analyst team.	Each transaction with a medium score gets a bad choice: approve and accept the fraud, or decline and lose a good customer.	Test it
Three	Approve, review or decline, with the review capacity as a limit	The analysts stop the fraud in the medium scores and release the good customers.	It needs an analyst team. The capacity limits the review tier.	Test it
More than three	Add a step-up check, for example a one-time code to the card holder (3-D Secure)	A cheap check for a medium score.	The data has no result of a step-up check. We cannot measure its catch rate or its friction.	Screened out: no data. A next step.

Evidence. All options use the GBM score after the recalibration. The code sets the thresholds of each option on month 4 to minimize the total cost, and gives the result on months 5–6. Two tiers use one threshold on the score. Three tiers use a decline threshold on the score, and the next highest scores fill the review capacity.

# Purpose: Compare one, two and three actions on months 5–6 (dec-05).
# Input:   `p_valid`, `p_test`, `valid`, `test`, `fit_policy()`, `apply_policy()`, `summary()`.
# Output:  `pol_2`, `pol_3` (policy rules), table `dec05`.

pol_2 = fit_policy("prob2", p_valid, valid)
pol_3 = fit_policy("prob3", p_valid, valid)
dec05 = pd.DataFrame([
    {"Option": "Reference: approve all", **summary(test, np.zeros(len(test), int))},
    {"Option": "A. Two tiers", **summary(test, apply_policy(pol_2, p_test, test))},
    {"Option": "B. Three tiers", **summary(test, apply_policy(pol_3, p_test, test))},
]).set_index("Option")
saving = dec05.iloc[1]["Total cost ($/month)"] - dec05.iloc[2]["Total cost ($/month)"]
team = COST["analysts"] * COST["analyst_day"] * 30
N.update({"d5_all": f"${dec05.iloc[0]['Total cost ($/month)']:,.0f}",
          "d5_two": f"${dec05.iloc[1]['Total cost ($/month)']:,.0f}",
          "d5_saving": f"${saving:,.0f}", "d5_team": f"${team:,.0f}",
          "d5_vs_team": "more than" if saving > team else "less than",
          "d5_fdr_a": f"{dec05.iloc[1]['Decline rate of good transactions']:.2%}",
          "d5_fdr_b": f"{dec05.iloc[2]['Decline rate of good transactions']:.2%}"})
fmt(dec05, SUM_FMT)

	Fraud loss ($/month)	False decline cost ($/month)	Review friction ($/month)	Total cost ($/month)	Fraud $ stopped	Decline rate of good transactions	Alerts each day (mean)	Days over capacity
Option
Reference: approve all	470,983	0	0	470,983	0.0%	0.00%	0	0%
A. Two tiers	199,068	69,970	0	269,039	57.7%	5.47%	0	0%
B. Three tiers	168,320	63,440	5,126	236,886	64.3%	4.95%	92	5%

How to read: each row is one policy on months 5–6. The cost columns are dollars each month. The last two columns show the use of the review capacity.

So what: the review tier decreases the total cost and the decline rate of good transactions at the same time.

Decision. Three tiers: approve, review or decline, with the review capacity as a limit.

Why. Approve all costs $470,983 each month, and two tiers cost $269,039. The three tiers decrease the total cost by a further $32,153 each month. This is more than the cost of the analyst team, $27,000 each month (as-03, as-04).

The decline rate of good transactions goes from 5.47% to 4.95%. The review tier does the work that a single threshold cannot do: it separates fraud from good customers in the medium scores.

What would change the decision. If the decrease of the total cost is less than the cost of the team, use two tiers. This happens if the catch rate of a review (as-05) is low or if the false decline cost (as-02) is low.

5d. The value that puts the transactions in order

Section 5c fixed three tiers. Section 5b showed that the same score can give a different cheapest action for a different amount. Thus the last question of this step is: which value puts the transactions in order for the decline tier and the review tier?

Which value puts the transactions in order for the policy?

Why this question matters. The review capacity does not change. Each alert on a small transaction takes the place of an alert on a large one. The value that puts the transactions in order decides which fraud dollars the analysts can stop.

Ways to do it. The split axis is what the value measures: how likely the fraud is, how much money is at stake, or both.

What the value measures	Method	Advantage	Main risk	Status
How likely the fraud is	A. The fraud score (probability). Thresholds on the score.	Simple. The same threshold for all transactions. Easy to explain to fraud operations.	A $5 transaction and a $2,000 transaction with the same score get the same action.	Test it
How much money is at stake	The amount only. Review the largest amounts first.	Simple. It protects the large amounts.	It ignores the score. Most large transactions are good, so the analysts examine good customers.	Screened out: it does not use the score
Both	B. The expected cost in dollars: the cheapest action for each transaction. Review the transactions where a review decreases the expected cost most.	It minimizes the total cost directly. It sends the large risky transactions to review first.	It needs a correct level of the score (dec-04). The threshold changes with the amount, so a customer can ask “why me?”.	Test it

Evidence. Both options use three tiers, the same score and the same review capacity. The code sets them on month 4 and gives the result on months 5–6.

# Purpose: Compare the fraud score and the expected cost as the value of the policy (dec-06).
# Input:   `pol_3`, `p_valid`, `p_test`, `valid`, `test`.
# Output:  `pol_usd` (the challenger policy rule), `act_test` (its action on months 5–6), table `dec06`.

pol_usd = fit_policy("dollar", p_valid, valid)
act_test = apply_policy(pol_usd, p_test, test)
dec06 = pd.DataFrame([
    {"Option": "A. Fraud score", **summary(test, apply_policy(pol_3, p_test, test))},
    {"Option": "B. Expected cost", **summary(test, act_test)},
]).set_index("Option")
saving6 = dec06.iloc[0]["Total cost ($/month)"] - dec06.iloc[1]["Total cost ($/month)"]
N.update({"d6_saving": f"${saving6:,.0f}",
          "d6_stop_a": f"{dec06.iloc[0]['Fraud $ stopped']:.1%}",
          "d6_stop_b": f"{dec06.iloc[1]['Fraud $ stopped']:.1%}",
          "d6_fdr_b": f"{dec06.iloc[1]['Decline rate of good transactions']:.2%}",
          "d6_cost_b": f"${dec06.iloc[1]['Total cost ($/month)']:,.0f}",
          "d6_over": f"{dec06.iloc[1]['Days over capacity']:.0%}"})
fmt(dec06, SUM_FMT)

	Fraud loss ($/month)	False decline cost ($/month)	Review friction ($/month)	Total cost ($/month)	Fraud $ stopped	Decline rate of good transactions	Alerts each day (mean)	Days over capacity
Option
A. Fraud score	168,320	63,440	5,126	236,886	64.3%	4.95%	92	5%
B. Expected cost	135,768	31,933	4,911	172,612	71.2%	2.46%	90	5%

How to read: each row is one policy with three tiers on months 5–6. Both rows use the same score and the same review capacity.

So what: the expected cost stops more fraud dollars and costs less each month than thresholds on the score.

Decision. The expected cost in dollars. This policy is the challenger.

Why. The expected cost decreases the total cost by $64,274 each month against the fraud score. It stops 71.2% of the fraud dollars, against 64.3%.

A fraud score shows how likely a fraud is. The expected cost shows how much money is at risk. The business decision is about money, so the expected cost is the correct value. On 5% of the test days, the alerts are more than the capacity. Fraud operations can move these alerts to the next day or decline the lowest ones.

What would change the decision. Assume that the false decline cost does not change with the amount. Also assume that fraud amounts are like good amounts. Then the two options give the same result, and option A is simpler.

Business note. Each transaction now has a score at the level of the recent fraud rate and one action. The challenger policy approves, reviews or declines by the expected cost in dollars, inside the capacity of 120 alerts each day. On months 5–6, it costs $172,612 each month and stops 71.2% of the fraud dollars. It declines 2.46% of the good transactions. Step 6 checks the challenger against the champion with the five branches of the evaluation framework.

Step 6. Evaluate

Step 5 changed the fraud score into a policy. Each transaction gets one action from its expected cost, inside the review capacity. Now we must judge this challenger policy as a whole. This step uses the five branches of the evaluation framework of Step 3a, one at a time. We keep the rules of Step 3a, also when a result fails a rule.

Purpose. This step answers: is the challenger good enough for a test, in model metrics and in dollars? It supports this decision: approve the challenger policy for the champion/challenger test.

All results in this step use the test months 5–6. The model, the recalibration and the thresholds never saw these months. The table below repeats the five rules of Step 3a. We set them before the evidence, and we do not change them here.

Branch	Question	Rule
1. Order of risk	Does the score put fraud above good transactions better than the champion?	The PR-AUC and the dollar recall in the top 2% are both above the champion. The GBM is 10% or more above the logistic regression on month 4.
2. Level	Is the mean score equal to the actual fraud rate?	A difference of less than 0.5 points.
3. Stability	Does the score keep its power after the training months?	A PSI of less than 0.1, and a PR-AUC on months 5–6 of 80% or more of month 4.
4. Business value	Does the policy decrease the total cost at the same review capacity?	A lower total cost, and a decline rate of good transactions of 3% or less (as-08).
5. Explainability	Can fraud operations read why the policy declines?	The leaves with a decline rate of 50% or more hold 50% or more of all declines.

6a. Branch 1: order of risk

The question: does the challenger score put fraud above good transactions better than the champion? The champion score is the fraud rate of its rule segment (as-06). The challenger score is the GBM score after the recalibration. The rule has two parts:

• On months 5–6, the PR-AUC and the dollar recall in the top 2% of scores are both above the champion.
• The model rule of Step 4b: the GBM replaces the logistic regression only if its PR-AUC on month 4 is 10% or more higher (relative).

# Purpose: Measure branch 1 (order of risk) on months 5–6, and the model rule of dec-03 on month 4.
# Input:   `test`, `valid`, `champ_score()`, `p_test`, `score_valid`, `score_valid_lr`, `score_metrics()`.
# Output:  Table `eval_order`, `champ_test` (champion score, months 5–6), `lr_gain`.

# The setup cell imports chart_style as `cs`. This guard keeps the part usable with an older setup cell.
if "cs" not in globals():
    import chart_style as cs
    cs.setup()

# Rules of the evaluation framework (Step 3a, set before the evidence). Fallback: the same values from the brief.
RULES = globals().get("RULES", {"b1_model_gain": 0.10, "b2_level_gap": 0.005, "b3_psi": 0.10, "b3_pr_keep": 0.80,
                                 "b4_fdr": 0.03, "b5_leaf_rate": 0.50, "b5_leaf_share": 0.50})

champ_test = champ_score(test)
eval_order = pd.DataFrame([
    {"Score": "Champion: static rules (as-06)", **score_metrics(test, champ_test)},
    {"Score": "Challenger: GBM after the recalibration", **score_metrics(test, p_test)},
]).set_index("Score")
m_champ, m_chal = eval_order.iloc[0], eval_order.iloc[1]

# Model rule (dec-03): relative PR-AUC gain of the GBM over the logistic regression on month 4.
pr4_gbm = average_precision_score(valid["isFraud"], score_valid)
pr4_lr = average_precision_score(valid["isFraud"], score_valid_lr)
lr_gain = pr4_gbm / pr4_lr - 1

b1_pass = (m_chal["PR-AUC"] > m_champ["PR-AUC"]
           and m_chal["Dollar recall, top 2%"] > m_champ["Dollar recall, top 2%"] and lr_gain >= RULES["b1_model_gain"])
N.update({
    "e1_pr_champ": f"{m_champ['PR-AUC']:.3f}", "e1_pr_chal": f"{m_chal['PR-AUC']:.3f}",
    "e1_usd_champ": f"{m_champ['Dollar recall, top 2%']:.1%}",
    "e1_usd_chal": f"{m_chal['Dollar recall, top 2%']:.1%}",
    "e1_pr4_gbm": f"{pr4_gbm:.3f}", "e1_pr4_lr": f"{pr4_lr:.3f}", "e1_lr_gain": f"{lr_gain:+.0%}",
    "b1_verdict": "The challenger passes branch 1." if b1_pass else "The challenger fails branch 1.",
})
fmt(eval_order, METRIC_FMT)

	PR-AUC	AUC	Recall, top 2%	Dollar recall, top 2%
Score
Champion: static rules (as-06)	0.269	0.736	25.4%	17.0%
Challenger: GBM after the recalibration	0.486	0.886	37.3%	26.5%

On months 5–6, the challenger has a PR-AUC of 0.486, against 0.269 for the champion. In the top 2% of scores, the challenger finds 26.5% of the fraud dollars. The champion finds 17.0%.

On month 4, the GBM has a PR-AUC of 0.574, and the logistic regression has 0.329. The relative gain is +75%, against the limit of +10%. The challenger passes branch 1.

6b. Branch 2: level

The question: is the mean score equal to the actual fraud rate, on months that the recalibration did not use? The policy multiplies the score by the amount (Step 5b). Thus a score that is too high declines too many good customers, and a score that is too low approves too much fraud. The recalibration of Step 5a used month 4. The table checks months 5 and 6.

# Purpose: Measure branch 2 (level): mean score against the actual fraud rate on the test months.
# Input:   `test`, `p_test` (score after the recalibration of dec-04).
# Output:  Table `level`, `level_gap` (mean score minus the actual fraud rate, months 5–6).

level = (pd.DataFrame({"Month": test["month"].to_numpy(), "Actual fraud rate": test["isFraud"].to_numpy(),
                       "Mean score": p_test})
         .groupby("Month").mean())
level.index = [f"Month {m}" for m in level.index]
level.loc["Months 5–6"] = [test["isFraud"].mean(), p_test.mean()]
level["Difference (points)"] = (level["Mean score"] - level["Actual fraud rate"]) * 100
level_gap = p_test.mean() - test["isFraud"].mean()

b2_pass = abs(level_gap) < RULES["b2_level_gap"]
N.update({"e2_mean": f"{p_test.mean():.2%}", "e2_actual": f"{test['isFraud'].mean():.2%}",
          "e2_gap": f"{level_gap * 100:+.2f}",
          "b2_verdict": "The challenger passes branch 2." if b2_pass else "The challenger fails branch 2."})
fmt(level, {"Actual fraud rate": "{:.2%}", "Mean score": "{:.2%}", "Difference (points)": "{:+.2f}"})

	Actual fraud rate	Mean score	Difference (points)
Month 5	3.47%	3.32%	-0.15
Month 6	3.47%	3.50%	+0.03
Months 5–6	3.47%	3.41%	-0.06

On months 5–6, the mean score is 3.41%, and the actual fraud rate is 3.47%. The difference is -0.06 points, against the limit of 0.5 points. The challenger passes branch 2. The expected cost of Step 5b can use this score.

6c. Branch 3: stability

The question: does the score keep its power after the training months? Two metrics answer it, because they find different types of drift:

• PSI of the score. It compares the score distribution of month 6 with months 1–3. It finds a change of the population. It needs no label.
• PR-AUC over time. It compares the PR-AUC on months 5–6 with the PR-AUC on month 4. It finds a change of the fraud patterns. It needs the label.

The code uses the same model m_b as Step 3c. It does not train a new model. The recalibration does not change the order of the scores, so it does not change the PSI or the PR-AUC.

# Purpose: Measure branch 3 (stability): PSI of the score and the PR-AUC of each month after training.
# Input:   `m_b`, `train`, `valid`, `test`, `p_valid`, `p_test`, `DELTA`, `psi()`, `pr4_gbm`.
# Output:  `score_train` (GBM score of months 1–3, Part 2 uses it too), table `stab`, `psi_score6`, `pr_ratio`.

# Score of the training months, with the same recalibration as the test months.
score_train = predict(m_b, train)
p_train = shift_logodds(score_train, DELTA)
m6 = (test["month"] == 6).to_numpy()
m5 = (test["month"] == 5).to_numpy()
psi_score6 = psi(p_train, p_test[m6])

# PR-AUC of each month. Month 4 is the reference: the model never saw it, but it is next to training.
pr_month = {4: pr4_gbm,
            5: average_precision_score(test["isFraud"][m5], p_test[m5]),
            6: average_precision_score(test["isFraud"][m6], p_test[m6])}
pr56 = average_precision_score(test["isFraud"], p_test)
pr_ratio = pr56 / pr4_gbm

stab = pd.DataFrame([
    ("PSI of the score, month 6 against months 1–3", f"{psi_score6:.3f}", "Less than 0.1"),
    ("PR-AUC, month 4 (reference)", f"{pr_month[4]:.3f}", ""),
    ("PR-AUC, month 5", f"{pr_month[5]:.3f}", ""),
    ("PR-AUC, month 6", f"{pr_month[6]:.3f}", ""),
    ("PR-AUC, months 5–6", f"{pr56:.3f}", ""),
    ("PR-AUC of months 5–6 as a share of month 4", f"{pr_ratio:.0%}", "80% or more"),
], columns=["Metric", "Result", "Rule"]).set_index("Metric")

b3_pass = psi_score6 < RULES["b3_psi"] and pr_ratio >= RULES["b3_pr_keep"]
N.update({"e3_psi": f"{psi_score6:.3f}", "e3_pr4": f"{pr_month[4]:.3f}", "e3_pr56": f"{pr56:.3f}",
          "e3_ratio": f"{pr_ratio:.0%}", "e3_drop": f"{1 - pr_ratio:.0%}",
          "b3_verdict": "The challenger passes branch 3." if b3_pass else "The challenger fails branch 3."})
stab

	Result	Rule
Metric
PSI of the score, month 6 against months 1–3	0.008	Less than 0.1
PR-AUC, month 4 (reference)	0.574
PR-AUC, month 5	0.496
PR-AUC, month 6	0.478
PR-AUC, months 5–6	0.486
PR-AUC of months 5–6 as a share of month 4	85%	80% or more

The PSI of the score in month 6 is 0.008, far below the limit of 0.1. The population of transactions stays the same. But the PR-AUC goes from 0.574 on month 4 to 0.486 on months 5–6. Thus the score keeps 85% of its power, against the limit of 80%. The challenger passes branch 3.

The two metrics tell different stories. The PSI says “no drift”, but the score loses 15% of its PR-AUC. The fraud patterns change, and the PSI cannot see this change. Thus a PSI alone is not a safe check in production. Part 2 uses this result for the retrain decision.

6d. Branch 4: business value

The question: at the same review capacity, does the challenger decrease the total cost against the champion? This is the main business test. Both policies have the same three tiers and the same capacity of 120 alerts each day (as-03). Only the score and the value that ranks the transactions change:

• Champion (as-06). The rules of a decision tree with 3 levels, fit on month 1. The code sets its decline and review thresholds on month 4, with the method of Step 5c.
• Challenger. The GBM score after the recalibration, and the expected cost in dollars (Step 5d).

The guardrail is the decline rate of good transactions. A decline of a good customer has a cost in the model (as-02), but it also hurts the brand. Thus the rule adds a hard limit of 3% (as-08). Marketing must confirm this limit before the test.

# Purpose: Compare the total cost of the champion and the challenger at the same review capacity (branch 4).
# Input:   `champ_score()`, `champ_test`, `valid`, `test`, `act_test`, `fit_policy()`, `apply_policy()`, `summary()`.
# Output:  `champ_valid`, `pol_champ`, `act_champ`, table `result`, `GUARDRAIL_FDR`.

# as-08: limit of the decline rate of good transactions (Marketing to confirm). The setup cell holds it in COST.
GUARDRAIL_FDR = RULES["b4_fdr"]  # Same value as COST["fdr_limit"] in the setup cell

# Champion: thresholds on the leaf fraud rate, set on month 4 (same method as dec-05).
champ_valid = champ_score(valid)
pol_champ = fit_policy("prob3", champ_valid, valid)
act_champ = apply_policy(pol_champ, champ_test, test)
result = pd.DataFrame([
    {"Policy": "Champion: static rules, three tiers", **summary(test, act_champ)},
    {"Policy": "Challenger: GBM, expected cost, three tiers", **summary(test, act_test)},
]).set_index("Policy")
r_champ, r_chal = result.iloc[0], result.iloc[1]
delta_month = r_champ["Total cost ($/month)"] - r_chal["Total cost ($/month)"]

b4_pass = delta_month > 0 and r_chal["Decline rate of good transactions"] <= GUARDRAIL_FDR
N.update({
    "cc_saving": f"${delta_month:,.0f}",
    "cc_saving_pct": f"{delta_month / r_champ['Total cost ($/month)']:.0%}",
    "cc_cost_champ": f"${r_champ['Total cost ($/month)']:,.0f}",
    "cc_cost_chal": f"${r_chal['Total cost ($/month)']:,.0f}",
    "cc_stop_champ": f"{r_champ['Fraud $ stopped']:.1%}",
    "cc_stop_chal": f"{r_chal['Fraud $ stopped']:.1%}",
    "cc_fdr_champ": f"{r_champ['Decline rate of good transactions']:.2%}",
    "cc_fdr_chal": f"{r_chal['Decline rate of good transactions']:.2%}",
    "cc_over_champ": f"{r_champ['Days over capacity']:.0%}",
    "cc_over_chal": f"{r_chal['Days over capacity']:.0%}",
    "guardrail": f"{GUARDRAIL_FDR:.0%}",
    "b4_verdict": "The challenger passes branch 4." if b4_pass else "The challenger fails branch 4.",
})
N["headline"] = (
    f"On the test months, the challenger decreases the total cost by {N['cc_saving']} each month "
    f"({N['cc_saving_pct']}) against the champion, with the same review capacity. "
    f"It stops {N['cc_stop_chal']} of the fraud dollars. The champion stops {N['cc_stop_champ']}.")
fmt(result, SUM_FMT)

	Fraud loss ($/month)	False decline cost ($/month)	Review friction ($/month)	Total cost ($/month)	Fraud $ stopped	Decline rate of good transactions	Alerts each day (mean)	Days over capacity
Policy
Champion: static rules, three tiers	290,176	3,013	6,243	299,432	38.4%	0.28%	119	48%
Challenger: GBM, expected cost, three tiers	135,768	31,933	4,911	172,612	71.2%	2.46%	90	5%

On the test months, the challenger decreases the total cost by $126,820 each month (42%) against the champion, with the same review capacity. It stops 71.2% of the fraud dollars. The champion stops 38.4%.

The challenger declines more good transactions: 2.46% against 0.28% for the champion. This rate is inside the guardrail limit of 3%. The challenger passes branch 4.

The champion approves almost all transactions and accepts the fraud loss. With the cost of a false decline in as-02, this choice is expensive. The champion also has a capacity problem. Its rule segments change in size from day to day. Thus its alerts are more than the capacity on 48% of the test days, against 5% for the challenger.

The chart splits the total cost of each test month into its three parts.

# Purpose: Show where the saving of branch 4 comes from, for each test month.
# Input:   `test`, `act_champ`, `act_test`, `summary()`, `N`.
# Output:  Table `cc_month` (cost by month, policy and component), stacked bar chart.

COMP = {"Fraud loss ($/month)": ("Fraud loss", cs.NEGATIVE),
        "False decline cost ($/month)": ("False declines", cs.NAVY),
        "Review friction ($/month)": ("Review friction", cs.GRAY)}
rows = []
for mo in sorted(test["month"].unique()):
    in_mo = (test["month"] == mo).to_numpy()
    for name, act in (("Champion", act_champ), ("Challenger", act_test)):
        s = summary(test[in_mo], act[in_mo])
        rows.append({"Month": mo, "Policy": name, **{c: s[c] for c in COMP}, "Total": s["Total cost ($/month)"]})
cc_month = pd.DataFrame(rows)

fig, ax = plt.subplots(figsize=(8.5, 4.4))
# Two bars for each month: champion left, challenger right. A gap separates the months.
xpos = [i // 2 * 2.6 + i % 2 for i in range(len(cc_month))]
bottom = np.zeros(len(cc_month))
for col, (lab, color) in COMP.items():
    vals = cc_month[col].to_numpy() / 1e3
    ax.bar(xpos, vals, bottom=bottom, color=color, width=0.8, edgecolor="white", lw=0.5)
    # Direct label at the right of the last bar, in the middle of its segment.
    cs.label_end(ax, xpos[-1] + 0.4, bottom[-1] + vals[-1] / 2, lab, color=color, dx=6)
    bottom += vals
for x, tot in zip(xpos, cc_month["Total"] / 1e3):
    ax.annotate(f"${tot:,.0f}k", (x, tot), xytext=(0, 4), textcoords="offset points", ha="center",
                fontsize=9, color=cs.INK)
ax.set_xticks(xpos)
ax.set_xticklabels([f"{p}\nMonth {m}" for p, m in zip(cc_month["Policy"], cc_month["Month"])])
ax.set_ylabel("$ thousands each month")
ax.set_xlim(-0.6, xpos[-1] + 2.2)
cs.action_title(fig, f"The challenger costs {N['cc_saving']} less each month than the champion",
                "Total cost per month by component, test months 5–6, same review capacity")
plt.show()
fmt(cc_month.set_index(["Month", "Policy"]), {c: "{:,.0f}" for c in [*COMP, "Total"]})

		Fraud loss ($/month)	False decline cost ($/month)	Review friction ($/month)	Total
Month	Policy
5	Champion	296,354	2,622	5,872	304,848
	Challenger	141,778	27,463	4,674	173,914
6	Champion	284,560	3,368	6,580	294,508
	Challenger	130,305	35,996	5,127	171,428

How to read: each bar is one policy in one test month. The red part is the fraud loss, the navy part is the cost of false declines, and the gray part is the review friction. The number on top is the total cost.

So what: the challenger moves cost from fraud loss to false declines, and the sum goes down in each month. Thus the result does not depend on one good month.

The challenger in this test uses the model from months 1–3 with no retrain. Part 2 shows that a monthly retrain improves the score, so this decrease of the total cost is a low estimate. The dollar values are for this data. The data is a sample of the payment flow of one merchant, so the values for the full flow are larger.

How much of the gain depends on the champion?

The champion is our assumption (as-06, Step 0b). A weak champion makes any challenger look good. Thus we repeat branch 4 against four champions, from weak to strong. Each champion uses the same three tiers, the same review capacity and the same method to set its thresholds on month 4 (Step 5c). Only its score changes:

Champion	Score	Why it is in the ladder
1. Static rules (as-06)	Decision tree, 3 levels, fit on month 1 only	The base champion of this notebook: a small rule set that does not change.
2. Rules from more data	Decision tree, 5 levels, fit on months 1–3	A rule set that a strategy team updates with the same data as the challenger.
3. Simple model	Logistic regression of Step 4b, fit on months 1–3	A team with a model, but with a simple model.
4. Same model, score thresholds	The GBM of the challenger, ranked by the fraud score (Step 5d, option A)	The strongest champion: the same score as the challenger. Only the dollar order is missing.

# Purpose: Show how much of the saving of branch 4 depends on the champion assumption (as-06).
# Input:   `train`, `valid`, `test`, `act_champ`, `act_test`, `lr`, `lr_cols`, `pol_3`, `p_test`,
#          `encode()`, `fit_policy()`, `apply_policy()`, `summary()`.
# Output:  Table `ladder`, N values `lad_*`, a bar chart of the saving against each champion.

def tree_champion(fit_frame, depth):
    """A rule champion: a decision tree on the numeric features. Its leaf fraud rate is the score."""
    X_fit = encode(fit_frame, fit_frame, NUM_COLS, []).fillna(-999)
    tree = DecisionTreeClassifier(max_depth=depth, min_samples_leaf=200, random_state=SEED)
    tree.fit(X_fit, fit_frame["isFraud"])
    score = lambda f: tree.predict_proba(encode(f, fit_frame, NUM_COLS, []).fillna(-999))[:, 1]
    return score(valid), score(test)


def champion_actions(s_valid, s_test):
    """Thresholds on the score, set on month 4 with the capacity limit (Step 5c), then applied to months 5-6."""
    return apply_policy(fit_policy("prob3", s_valid, valid), s_test, test)


as_obj = {c: object for c in CAT_COLS}
s5_valid, s5_test = tree_champion(train, depth=5)
lr_test = lr.predict_proba(test[lr_cols].astype(as_obj))[:, 1]
lr_valid = lr.predict_proba(valid[lr_cols].astype(as_obj))[:, 1]
champions = {
    "1. Static rules (as-06)": act_champ,
    "2. Rules from more data": champion_actions(s5_valid, s5_test),
    "3. Simple model": champion_actions(lr_valid, lr_test),
    "4. Same model, score thresholds": apply_policy(pol_3, p_test, test),
}
cost_chal = summary(test, act_test)["Total cost ($/month)"]
rows = []
for name, act in champions.items():
    s = summary(test, act)
    rows.append({"Champion": name, "Total cost ($/month)": s["Total cost ($/month)"],
                 "Fraud $ stopped": s["Fraud $ stopped"],
                 "Decline rate of good transactions": s["Decline rate of good transactions"],
                 "Saving of the challenger ($/month)": s["Total cost ($/month)"] - cost_chal})
ladder = pd.DataFrame(rows).set_index("Champion")
save = ladder["Saving of the challenger ($/month)"]
N.update({
    "lad_save_1": f"${save.iloc[0]:,.0f}", "lad_save_2": f"${save.iloc[1]:,.0f}",
    "lad_save_3": f"${save.iloc[2]:,.0f}", "lad_save_4": f"${save.iloc[3]:,.0f}",
    "lad_save_min": f"${save.min():,.0f}", "lad_save_max": f"${save.max():,.0f}",
    "lad_min_name": save.idxmin().split(". ", 1)[1].lower(),
    "lad_all_pos": "every" if (save > 0).all() else "not every",
    "lad_stop_2": f"{ladder['Fraud $ stopped'].iloc[1]:.1%}",
    "lad_stop_3": f"{ladder['Fraud $ stopped'].iloc[2]:.1%}",
})
# Share of the base gain that depends on the weak base champion (the part above the strongest champion).
N["lad_share_txt"] = (f"Thus {1 - save.min() / save.iloc[0]:.0%} of the headline gain depends on the weak base "
                      f"champion, and {save.min() / save.iloc[0]:.0%} stays against the strongest champion.")

fig, ax = plt.subplots(figsize=(8, 3.4))
ypos = np.arange(len(save))[::-1]
colors = [cs.GRAY] * (len(save) - 1) + [cs.BLUE]  # The strongest champion is the point of the chart
ax.barh(ypos, save / 1e3, color=colors, height=0.6)
for y, v in zip(ypos, save / 1e3):
    ax.annotate(f"${v:,.0f}k", (v, y), xytext=(4, 0), textcoords="offset points", va="center", fontsize=9,
                color=cs.INK)
ax.set_yticks(ypos)
ax.set_yticklabels(save.index)
ax.set_xlabel("Saving of the challenger, $ thousands each month")
ax.set_xlim(0, save.max() / 1e3 * 1.18)
cs.action_title(fig, f"Against the strongest champion, the challenger still saves {N['lad_save_min']} each month",
                "Branch 4 repeated against four champions, test months 5–6, same review capacity")
plt.show()
fmt(ladder, {"Total cost ($/month)": "{:,.0f}", "Fraud $ stopped": "{:.1%}",
             "Decline rate of good transactions": "{:.2%}", "Saving of the challenger ($/month)": "{:,.0f}"})

	Total cost ($/month)	Fraud $ stopped	Decline rate of good transactions	Saving of the challenger ($/month)
Champion
1. Static rules (as-06)	299,432	38.4%	0.28%	126,820
2. Rules from more data	276,518	65.4%	8.10%	103,906
3. Simple model	295,739	58.6%	7.90%	123,127
4. Same model, score thresholds	236,886	64.3%	4.95%	64,274

How to read: each bar is the gain of the challenger against one champion: the decrease of the total cost each month. The top bar is the base champion of this notebook. The bottom bar is the strongest champion: the same GBM score, with thresholds on the score instead of the expected cost.

So what: the gain is $126,820 against the static rules, but only $64,274 against the strongest champion. Thus 49% of the headline gain depends on the weak base champion, and 51% stays against the strongest champion. The part that does not depend on the champion is the dollar order of Step 5d. The challenger saves money against every champion in the ladder. A real test (Step 7c) measures the gain against the real rules of the merchant.

6e. Branch 5: explainability

The question: can fraud operations read why the policy declines a transaction? The GBM gives no rules. Fraud operations and the partners ask “why did you decline this transaction?”. A fraud decline needs no adverse action notice, unlike a credit decline. Thus the law does not ask for a reason for each decline. But the rule of Step 3a still applies, because fraud operations must understand and check the declines.

The method is a surrogate tree. It is a small decision tree that copies the decline decisions of the challenger. The tree has these parameters:

• 3 levels, so that each rule has three conditions or fewer.
• 500 transactions or more in each leaf, so that each rule covers a real segment.
• Only features that people can read. These are the amount, the hour, the distance, the email match, the identity flag and the history features. Four fields of the card, the product and the device complete the list.

The rule of branch 5: the leaves with a decline rate of 50% or more must hold 50% or more of all declines. If this is true, a few readable rules explain most declines.

# Purpose: Fit a surrogate tree with 3 levels on the decline decision of the challenger (branch 5).
# Input:   `test`, `act_test`, `CAT_COLS`, `SEED`.
# Output:  `sur` (the tree), table `leaves`, `leaf_rules()`, `sur_share` (share of declines in leaves of 50%+).

READ_COLS = ["TransactionAmt", "hour", "dist1", "email_match", "has_identity",
             "uid_n", "uid_amt_ratio", "uid_n_today", "card_n_today"]
READ_CAT = [c for c in ["ProductCD", "card4", "card6", "DeviceType"] if c in CAT_COLS]
X_read = pd.get_dummies(test[READ_COLS + READ_CAT], columns=READ_CAT, dtype=float)
X_read["uid_hours_since"] = test["uid_hours_since"].fillna(10_000)  # 10,000 = no earlier transaction
X_read = X_read.fillna(-1)
is_decline = (act_test == 2).astype(int)
sur = DecisionTreeClassifier(max_depth=3, min_samples_leaf=500, random_state=SEED)
sur.fit(X_read, is_decline)


def leaf_rules(tree, names, binary):
    """Return the rule text of each leaf of a fitted tree."""
    t, rules = tree.tree_, {}

    def walk(node, conds):
        if t.children_left[node] == -1:
            rules[node] = " and ".join(conds)
            return
        f, thr = names[t.feature[node]], t.threshold[node]
        left, right = ((f"{f} = 0", f"{f} = 1") if f in binary
                       else (f"{f} ≤ {thr:,.1f}", f"{f} > {thr:,.1f}"))
        walk(t.children_left[node], conds + [left])
        walk(t.children_right[node], conds + [right])

    walk(0, [])
    return rules


binary = {c for c in X_read.columns if set(X_read[c].unique()) <= {0.0, 1.0}}
rules = leaf_rules(sur, list(X_read.columns), binary)
leaves = (pd.DataFrame({"leaf": sur.apply(X_read), "decline": is_decline})
          .groupby("leaf")["decline"].agg(["size", "mean", "sum"]))
leaves.columns = ["Transactions", "Decline rate", "Declines"]
leaves["Share of all declines"] = leaves["Declines"] / is_decline.sum()
leaves.insert(0, "Rule", leaves.index.map(rules))
leaves = leaves.sort_values("Decline rate", ascending=False).reset_index(drop=True)

# Rule of branch 5: share of all declines in the leaves with a decline rate of 50% or more.
sur_share = leaves.loc[leaves["Decline rate"] >= RULES["b5_leaf_rate"], "Share of all declines"].sum()
top = leaves.iloc[0]
b5_pass = sur_share >= RULES["b5_leaf_share"]
N.update({"sur_rule": top["Rule"], "sur_rate": f"{top['Decline rate']:.0%}",
          "sur_share": f"{top['Share of all declines']:.0%}",
          "sur_all": f"{is_decline.mean():.1%}",
          "sur_n_big": f"{(leaves['Decline rate'] >= RULES['b5_leaf_rate']).sum()}",
          "sur_big_share": f"{sur_share:.0%}",
          "b5_verdict": "The challenger passes branch 5." if b5_pass else "The challenger fails branch 5."})
fmt(leaves.drop(columns="Declines"),
    {"Transactions": "{:,.0f}", "Decline rate": "{:.1%}", "Share of all declines": "{:.0%}"})

	Rule	Transactions	Decline rate	Share of all declines
0	email_match = 1 and TransactionAmt > 53.2 and ...	5,320	26.4%	19%
1	email_match = 1 and TransactionAmt ≤ 53.2 and ...	6,051	14.5%	12%
2	email_match = 0 and uid_hours_since ≤ 48.7 and...	15,352	10.8%	23%
3	email_match = 1 and TransactionAmt > 53.2 and ...	3,879	8.2%	4%
4	email_match = 1 and TransactionAmt ≤ 53.2 and ...	10,100	6.6%	9%
5	email_match = 0 and uid_hours_since > 48.7 and...	20,396	4.4%	12%
6	email_match = 0 and uid_hours_since ≤ 48.7 and...	26,788	3.1%	11%
7	email_match = 0 and uid_hours_since > 48.7 and...	92,053	0.8%	10%

The challenger declines 4.1% of the test transactions. The segment with the highest decline rate is: email_match = 1 and TransactionAmt > 53.2 and ProductCD_R = 0. In this segment, the challenger declines 26% of the transactions. The segment has 19% of all declines. email_match = 1 means that the email domain of the buyer and the email domain of the recipient are the same.

0 leaves have a decline rate of 50% or more. These leaves hold 0% of all declines, against the limit of 50%. The challenger fails branch 5.

Three levels of readable rules cannot copy the score. The GBM declines a transaction because of many small signals together, and a rule with three conditions cannot show them. The rules still show the main patterns, and fraud operations can use them to find new fraud patterns. They cannot replace the score. Step 7 adds a control for this gap.

6f. Summary of the evaluation

The table collects one result for each branch, with the rule of Step 3a.

# Purpose: Collect the result of the challenger for each branch of the evaluation framework.
# Input:   Results of sections 6a–6e.
# Output:  Table `evaluation`, `N["n_pass"]`, `N["fail_list"]`.

evaluation = pd.DataFrame([
    ("1. Order of risk",
     "PR-AUC and dollar recall in the top 2%, months 5–6 (challenger / champion); "
     "PR-AUC gain of the GBM over the logistic regression, month 4",
     f"{m_chal['PR-AUC']:.3f} / {m_champ['PR-AUC']:.3f}; "
     f"{m_chal['Dollar recall, top 2%']:.1%} / {m_champ['Dollar recall, top 2%']:.1%}; {lr_gain:+.0%}",
     "Both above the champion; gain of 10% or more", b1_pass),
    ("2. Level", "Mean score minus actual fraud rate, months 5–6", f"{level_gap * 100:+.2f} points",
     "Less than 0.5 points", b2_pass),
    ("3. Stability", "PSI of the score (month 6 against months 1–3); PR-AUC of months 5–6 as a share of month 4",
     f"{psi_score6:.3f}; {pr_ratio:.0%}", "PSI less than 0.1; 80% or more", b3_pass),
    ("4. Business value", "Total cost per month (challenger / champion); decline rate of good transactions",
     f"${r_chal['Total cost ($/month)']:,.0f} / ${r_champ['Total cost ($/month)']:,.0f}; "
     f"{r_chal['Decline rate of good transactions']:.2%}",
     f"Lower total cost; {GUARDRAIL_FDR:.0%} or less (as-08)", b4_pass),
    ("5. Explainability", "Share of declines in surrogate leaves with a decline rate of 50% or more",
     f"{sur_share:.0%}", "50% or more", b5_pass),
], columns=["Branch", "Metric", "Result", "Rule", "Pass"]).set_index("Branch")
evaluation["Pass"] = evaluation["Pass"].map({True: "Yes", False: "No"})
failed = [b.split(". ", 1)[1].lower() for b in evaluation.index[evaluation["Pass"] == "No"]]
N.update({"n_pass": f"{(evaluation['Pass'] == 'Yes').sum()}",
          "fail_list": ", ".join(failed) if failed else "none"})
evaluation

	Metric	Result	Rule	Pass
Branch
1. Order of risk	PR-AUC and dollar recall in the top 2%, months...	0.486 / 0.269; 26.5% / 17.0%; +75%	Both above the champion; gain of 10% or more	Yes
2. Level	Mean score minus actual fraud rate, months 5–6	-0.06 points	Less than 0.5 points	Yes
3. Stability	PSI of the score (month 6 against months 1–3);...	0.008; 85%	PSI less than 0.1; 80% or more	Yes
4. Business value	Total cost per month (challenger / champion); ...	$172,612 / $299,432; 2.46%	Lower total cost; 3% or less (as-08)	Yes
5. Explainability	Share of declines in surrogate leaves with a d...	0%	50% or more	No

The challenger passes 4 of the 5 branches. The branches that fail are: explainability.

What a failure means for the test:

• Explainability (branch 5). The policy cannot give a short readable rule for most declines. Fraud operations can still use the rules to find patterns, and a decline needs no adverse action notice. Thus this failure does not stop a test. But a partner or a regulator can ask for a reason for each decline. Then a decline reason must come from another method, and Step 4b gives the alternative.
• Any other branch. A failure of branches 1 to 4 goes back to the owner of the decision before the test. The test plan of Step 7 assumes that branches 1 to 4 pass.

Business note. The challenger passes 4 of the 5 branches of the evaluation framework. At the same review capacity, it decreases the total cost by $126,820 each month (42%). It declines 2.46% of good transactions, inside the guardrail of 3%.

Two results need attention. The score loses 15% of its PR-AUC after month 4, and the PSI does not show this loss. Three levels of readable rules do not explain most declines. Step 7 changes the result into a test plan, and Part 2 sets the retrain schedule.

Step 7. Implementation

Step 6 showed that the challenger passes 4 of the 5 branches. At the same review capacity, it decreases the total cost by $126,820 each month on the test months. A result on past data is not yet a decision.

This step answers three questions. Is the analyst team the correct size? Does the result hold for other costs? How do we test the policy safely?

Purpose. This step changes the evaluation into a team size, a range of dollar results and a test plan. It supports this decision: go or no-go for a champion/challenger test on 10% of transactions.

7a. Value of one more analyst

The review capacity is a limit of the policy (as-03). Each analyst examines 40 alerts each day and costs $9,000 each month (as-04). One more analyst lets the policy send more transactions to review. The question: does the decrease of the total cost pay for the analyst?

The code repeats the challenger policy for 0 to 6 analysts. For each team size, it sets the thresholds again on month 4 and measures the total cost on months 5–6. The team cost is separate from the total cost (glossary), so the chart adds them.

# Purpose: Find the value of one more analyst (as-03, as-04): the shadow price of the review capacity.
# Input:   `COST`, `fit_policy()`, `apply_policy()`, `summary()`, `p_valid`, `valid`, `p_test`, `test`.
# Output:  Table `cap`, `best` (team size with the lowest total cost + team cost).

rows = []
for n_an in range(0, 7):
    c = {**COST, "analysts": n_an}
    # Set the thresholds again on month 4 for this capacity. Then measure on months 5–6.
    rule = fit_policy("dollar", p_valid, valid, c)
    s = summary(test, apply_policy(rule, p_test, test), c)
    rows.append({"Analysts": n_an, "Alerts each day": capacity(c),
                 "Total cost ($/month)": s["Total cost ($/month)"],
                 "Team cost ($/month)": n_an * COST["analyst_day"] * 30})
cap = pd.DataFrame(rows).set_index("Analysts")
cap["Cost + team ($/month)"] = cap["Total cost ($/month)"] + cap["Team cost ($/month)"]
cap["Saving from the last analyst ($/month)"] = -cap["Total cost ($/month)"].diff()
best = int(cap["Cost + team ($/month)"].idxmin())
N.update({"cap_best": str(best), "cap_now": str(COST["analysts"]),
          "cap_next": f"${cap.loc[COST['analysts'] + 1, 'Saving from the last analyst ($/month)']:,.0f}",
          "analyst_month": f"${COST['analyst_day'] * 30:,.0f}",
          "cap_gap": f"${cap.loc[COST['analysts'], 'Cost + team ($/month)'] - cap.loc[best, 'Cost + team ($/month)']:,.0f}"})
fmt(cap, {c: "{:,.0f}" for c in cap.columns if "$" in c})

	Alerts each day	Total cost ($/month)	Team cost ($/month)	Cost + team ($/month)	Saving from the last analyst ($/month)
Analysts
0	0	238,693	0	238,693	nan
1	40	202,986	9,000	211,986	35,706
2	80	187,051	18,000	205,051	15,936
3	120	172,612	27,000	199,612	14,439
4	160	162,078	36,000	198,078	10,534
5	200	153,197	45,000	198,197	8,881
6	240	146,916	54,000	200,916	6,281

# Purpose: Show the shadow price of the review capacity (7a).
# Input:   `cap`, `best`, `COST`, `N`.
# Output:  Line chart.

fig, ax = plt.subplots(figsize=(8, 4.2))
x = cap.index.to_numpy()
ax.plot(x, cap["Total cost ($/month)"] / 1e3, color=cs.GRAY, lw=1.6, marker="o", ms=4)
ax.plot(x, cap["Cost + team ($/month)"] / 1e3, color=cs.NAVY, lw=2, marker="o", ms=4)
cs.label_end(ax, x[-1], cap["Total cost ($/month)"].iloc[-1] / 1e3, "Total cost", color=cs.GRAY)
cs.label_end(ax, x[-1], cap["Cost + team ($/month)"].iloc[-1] / 1e3, "Total cost + team", color=cs.NAVY, bold=True)
# Gray: the current team (as-03). Blue: the team size with the lowest cost + team, the one point of the chart.
now = COST["analysts"]
ax.axvline(now, color=cs.LIGHT_GRAY, lw=1, ls="--")
ax.annotate(f"Today: {now} analysts", (now, ax.get_ylim()[1]), xytext=(4, -12), textcoords="offset points",
            fontsize=8.5, color=cs.MUTED)
ax.plot(best, cap.loc[best, "Cost + team ($/month)"] / 1e3, "o", color=cs.BLUE, ms=9, zorder=5)
ax.annotate(f"Lowest: {best} analysts", (best, cap.loc[best, "Cost + team ($/month)"] / 1e3), xytext=(0, -18),
            textcoords="offset points", ha="center", fontsize=9, color=cs.BLUE, fontweight="bold")
ax.set_xlabel("Analysts")
ax.set_ylabel("$ thousands each month")
ax.set_xlim(-0.3, x[-1] + 1.6)
cs.action_title(fig, f"One more analyst saves {N['cap_next']} each month and costs {N['analyst_month']}",
                f"Challenger policy, test months 5–6. Team cost: ${COST['analyst_day']:,.0f} for each analyst and day (as-04).")
plt.show()

How to read: the gray line is the total cost of fraud, false declines and review friction. The navy line adds the team cost. The blue point is the team size with the lowest sum.

So what: with 3 analysts, one more analyst decreases the total cost by $10,534 each month and costs $9,000. The lowest sum is at 4 analysts. The difference from today is only $1,534 each month, so the team size is not a critical decision at this volume. For the full payment flow, repeat the calculation with the actual volume and cost.

7b. Sensitivity of the result

The costs in the policy are assumptions (as-01 to as-05). If an assumption is wrong, the result can change. The code changes one assumption at a time and repeats the full comparison of Step 6d. For each case, it sets the thresholds of both policies again on month 4. The cases are:

Assumption	Base	Other values
Customer friction of a false decline (as-02)	$10	$5 and $20
Loss share of approved fraud (as-01)	100%	50%
Catch rate of a review (as-05)	90%	70%
Analysts (as-03)	3	2 and 4

# Purpose: Repeat the champion/challenger comparison for other cost assumptions (as-01 to as-05).
# Input:   `COST`, `champ_valid`, `champ_test`, `p_valid`, `p_test`, `valid`, `test`, `GUARDRAIL_FDR`.
# Output:  Table `sens`.

cases = [("Base", {}), ("False decline friction $5", {"fd_friction": 5.0}),
         ("False decline friction $20", {"fd_friction": 20.0}),
         ("Loss share 50%", {"loss_share": 0.5}), ("Review catch rate 70%", {"review_catch": 0.7}),
         ("2 analysts", {"analysts": 2}), ("4 analysts", {"analysts": 4})]
rows = []
for name, change in cases:
    c = {**COST, **change}
    # Both policies get new thresholds on month 4 for the changed cost.
    a_ch = apply_policy(fit_policy("prob3", champ_valid, valid, c), champ_test, test)
    a_cl = apply_policy(fit_policy("dollar", p_valid, valid, c), p_test, test)
    s_ch, s_cl = summary(test, a_ch, c), summary(test, a_cl, c)
    rows.append({"Case": name,
                 "Champion total cost ($/month)": s_ch["Total cost ($/month)"],
                 "Challenger total cost ($/month)": s_cl["Total cost ($/month)"],
                 "Saving ($/month)": s_ch["Total cost ($/month)"] - s_cl["Total cost ($/month)"],
                 "Challenger decline rate of good transactions": s_cl["Decline rate of good transactions"]})
sens = pd.DataFrame(rows).set_index("Case")
over = sens["Challenger decline rate of good transactions"] > GUARDRAIL_FDR
N.update({"sens_min": f"${sens['Saving ($/month)'].min():,.0f}",
          "sens_min_case": sens["Saving ($/month)"].idxmin(),
          "sens_max": f"${sens['Saving ($/month)'].max():,.0f}",
          "sens_fdr_max": f"{sens['Challenger decline rate of good transactions'].max():.2%}",
          "sens_fdr_case": sens["Challenger decline rate of good transactions"].idxmax(),
          "sens_n_over": f"{over.sum()}", "sens_n": f"{len(sens)}"})
fmt(sens, {"Champion total cost ($/month)": "{:,.0f}", "Challenger total cost ($/month)": "{:,.0f}",
           "Saving ($/month)": "{:,.0f}", "Challenger decline rate of good transactions": "{:.2%}"})

	Champion total cost ($/month)	Challenger total cost ($/month)	Saving ($/month)	Challenger decline rate of good transactions
Case
Base	299,432	172,612	126,820	2.46%
False decline friction $5	297,487	156,733	140,754	4.38%
False decline friction $20	301,746	191,124	110,622	1.07%
Loss share 50%	154,344	100,799	53,544	0.79%
Review catch rate 70%	330,887	194,777	136,110	2.84%
2 analysts	314,989	187,051	127,938	2.99%
4 analysts	291,439	162,078	129,361	1.97%

# Purpose: Show the range of the saving across the cost assumptions (7b).
# Input:   `sens`, `GUARDRAIL_FDR`, `N`.
# Output:  Horizontal bar chart.

s_plot = sens.iloc[::-1]  # Base at the top
fig, ax = plt.subplots(figsize=(8, 4.2))
colors = [cs.BLUE if c == "Base" else cs.NAVY for c in s_plot.index]
ax.barh(s_plot.index, s_plot["Saving ($/month)"] / 1e3, color=colors, height=0.6)
for i, (case, row) in enumerate(s_plot.iterrows()):
    fdr = row["Challenger decline rate of good transactions"]
    # Value label, and the decline rate of good transactions. Above the guardrail: red text.
    ax.annotate(f"${row['Saving ($/month)'] / 1e3:,.0f}k   declines {fdr:.1%} of good",
                (row["Saving ($/month)"] / 1e3, i), xytext=(4, 0), textcoords="offset points", va="center",
                fontsize=8.5, color=cs.NEGATIVE if fdr > GUARDRAIL_FDR else cs.MUTED)
ax.set_xlabel("Saving of the challenger against the champion ($ thousands each month)")
ax.set_xlim(0, sens["Saving ($/month)"].max() / 1e3 * 1.55)
cs.action_title(fig, f"The challenger saves at least {N['sens_min']} each month in all {N['sens_n']} cases",
                f"Test months 5–6. Text: decline rate of good transactions (red if above {GUARDRAIL_FDR:.0%}, as-08).")
plt.show()

How to read: each bar is one case. The blue bar is the base case. The text at the end of the bar gives the decline rate of good transactions of the challenger in that case.

So what: in all 7 cases, the challenger costs less than the champion, by at least $53,544 each month (Loss share 50%). The guardrail is more sensitive. The highest decline rate of good transactions is 4.38% (False decline friction $5). The number of cases above the limit of 3% is 1.

The cost of a false decline (as-02) has the largest effect on the decline rate of good transactions. If this cost is low, the policy declines more. Finance and Marketing must confirm it before the test, together with the guardrail limit (as-08).

7c. Rollout plan

The result of Step 6 comes from past transactions. Fraud patterns, customers and costs change. A controlled test gives the proof before a full rollout. The plan has five parts.

1. Test design. Send a random 10% of transactions to the challenger. The other 90% stay with the champion. A random allocation makes the two arms equal in all other ways, so the difference comes from the policy.

Each arm gets the same share of the review capacity as of the transactions. Thus the challenger arm gets 10% of the alerts each day, and the test keeps the condition “same review capacity”.

The test must run long enough to find the difference in cost. The code estimates the number of days from the cost of each transaction on months 5–6. It uses a significance level of 5% (two-sided) and a power of 80%.

Technical note. Let d be the difference of the mean cost for each transaction (champion minus challenger). Let s₁² and s₂² be the variances of the cost for each transaction under the champion and the challenger. Let v be the transactions each day, and w the share of the challenger arm. Then the test needs D days: D = (1.96 + 0.84)² × (s₁² / (1 − w) + s₂² / w) / (v × d²).

This formula uses a normal approximation. The cost of a fraud transaction has a long tail, so we double D for safety. The fraud label also arrives late (as-07), so the result needs time after the last day of the test.

# Purpose: Size the champion/challenger test on 10% of transactions (7c).
# Input:   `test`, `act_champ`, `act_test`, `actual_cost()`, `COST`, `capacity()`.
# Output:  Table `rollout`, `test_days`.

TEST_SHARE = globals().get("TEST_SHARE", 0.10)  # Share of transactions in the challenger arm (setup cell)
Z = 1.96 + 0.84     # Significance 5% (two-sided) and power 80%

cost_ch = actual_cost(test, act_champ)
cost_cl = actual_cost(test, act_test)
per_day = len(test) / test["day"].nunique()
diff = cost_ch.mean() - cost_cl.mean()
days_raw = Z ** 2 * (cost_ch.var() / (1 - TEST_SHARE) + cost_cl.var() / TEST_SHARE) / (per_day * diff ** 2)
test_days = int(np.ceil(2 * days_raw))  # Double the estimate: the cost has a long tail.

rollout = pd.DataFrame({"Value": [
    f"{per_day:,.0f}", f"{per_day * TEST_SHARE:,.0f}", f"{per_day * TEST_SHARE * test['isFraud'].mean():,.0f}",
    f"{capacity() * TEST_SHARE:,.0f}", f"${diff:,.2f}", f"{days_raw:,.1f}", f"{test_days}",
]}, index=["Transactions each day (months 5–6)", "Transactions each day, challenger arm",
           "Fraud transactions each day, challenger arm", "Alerts each day, challenger arm",
           "Difference of the mean cost for each transaction", "Days to find the difference (formula)",
           "Days to find the difference (doubled)"])
N.update({"ro_arm_day": f"{per_day * TEST_SHARE:,.0f}", "ro_alerts": f"{capacity() * TEST_SHARE:,.0f}",
          "ro_days": f"{test_days}", "ro_diff": f"${diff:,.2f}"})
rollout

	Value
Transactions each day (months 5–6)	2,856
Transactions each day, challenger arm	286
Fraud transactions each day, challenger arm	10
Alerts each day, challenger arm	12
Difference of the mean cost for each transaction	$1.48
Days to find the difference (formula)	7.7
Days to find the difference (doubled)	16

The challenger arm gets about 286 transactions and 12 alerts each day. The challenger saves $1.48 for each transaction. To find this difference, the test needs about 16 days of transactions. Then it needs time for the chargebacks. The data has no chargeback dates, so Fraud Operations must give this delay.

2. Main KPI. The total cost for each 1,000 transactions in each arm, after the chargeback delay. The total cost includes the fraud loss, the false declines and the review friction (Step 5b). The cost for each 1,000 transactions makes the two arms of different size comparable.

3. Guardrails. Check these values each week:

• The decline rate of good transactions of the challenger arm: 3% or less (as-08). The label arrives late, so the decline rate of all transactions is the early signal.
• The alerts each day against the capacity of the arm.
• The PSI of the score against months 1–3, and the PR-AUC of each month with labels (Part 2).

4. Kill criteria. Stop the test in any of these cases:

• The decline rate of all transactions in the challenger arm is above 3% for two weeks in a row.
• The fraud loss for each 1,000 transactions of the challenger arm is above the champion arm for one full month with labels.
• The PSI of the score is more than 0.25.

5. Top risks.

Risk	Effect	Control
The cost of a false decline (as-02) is higher than the assumption.	The challenger declines too many good customers. The decrease of the total cost is too high.	Finance and Marketing confirm as-02 and as-08 before the test. The guardrail and the kill criteria stop the test.
The fraud patterns change after the training months.	The score loses power, but the PSI stays low (Step 6c).	Retrain each month (Part 2). Check the PR-AUC of each month with labels.
Fraud operations cannot read why the policy declines (branch 5).	A partner complaint has no short answer. Fraud operations cannot check the declines.	Give the surrogate rules to fraud operations as patterns, not as reasons. If a partner asks for reasons, use the alternative of Step 4b.
The champion is an assumption (as-06).	The real champion can be stronger or weaker. The decrease of the total cost changes.	The test measures the real champion, because the 90% arm uses it.
The labels arrive late (as-07).	The test result and each retrain use labels that are not complete.	Read the KPI only after the chargeback delay. Train on all months except the last one when the labels are late.

Decision at the end. If the challenger arm has a lower total cost for each 1,000 transactions and stays inside the guardrail, roll out the challenger. Increase the share step by step and keep the champion arm as a control.

Business note. At the current capacity, the challenger decreases the total cost by $126,820 each month. In each cost case that we tested, the decrease is at least $53,544. The best team size changes the cost by only $1,534 each month.

The recommendation is a test on 10% of transactions for about 16 days, with the guardrail of 3% (as-08). The test uses one score with no retrain. Part 2 examines how fast this score loses power, and when to retrain it.

Part 2: Drift and retrain

Step 7 gave a test plan for one score, the GBM from months 1–3 with no retrain. But Step 6c showed a problem. The PSI of the score stays low, but the score keeps only 85% of its month 4 PR-AUC on months 5–6. Fraud patterns change, and a static score loses value. Part 2 asks two questions: what changes after the training months, and how often must we retrain the score?

Purpose. This part shows how the data and the score change over time. It supports this decision: the retrain schedule and the checks of the model in production.

Drift: what changes after the training months?

A score can lose value in two ways. The population can change: new customers, new products or new devices. Or the relation between the features and fraud can change: the fraudsters find a new method. The PSI finds only the first type. The PR-AUC of each month with labels finds the second type.

The table gives the PSI of the score and of nine important features for months 4, 5 and 6, against months 1–3. A PSI of less than 0.1 is stable. A PSI of more than 0.25 is a large change.

# Purpose: Measure the drift of the score and of the top features after the training months.
# Input:   `txn`, `train`, `p_train`, `p_valid`, `p_test` (Step 6c), `psi()`.
# Output:  Table `drift` (one row for each month, one column for the score and each feature).

imp_feats = [c for c in ["TransactionAmt", "C1", "C13", "card1", "D1", "D1n", "uid_n",
                         "uid_amt_ratio", "card_n_today"] if c in txn.columns]
# Scores after the recalibration of each month. The recalibration keeps the order, so the PSI does not change.
month_score = {4: p_valid, 5: p_test[(test["month"] == 5).to_numpy()], 6: p_test[(test["month"] == 6).to_numpy()]}
rows = []
for mo in [4, 5, 6]:
    part = txn[txn["month"] == mo]
    row = {"Month": mo, "Score": psi(p_train, month_score[mo])}
    row.update({f: psi(train[f], part[f]) for f in imp_feats})
    rows.append(row)
drift = pd.DataFrame(rows).set_index("Month")
feat6 = drift.drop(columns="Score").loc[6]
N.update({"psi_score_6": f"{drift.loc[6, 'Score']:.3f}",
          "psi_max_feat": feat6.idxmax(), "psi_max_val": f"{feat6.max():.2f}",
          "psi_n_large": f"{(feat6 > 0.25).sum()}",
          "psi_uid_n": f"{drift.loc[6, 'uid_n']:.2f}" if "uid_n" in drift else "n/a"})
drift.round(3)

	Score	TransactionAmt	C1	C13	card1	D1	D1n	uid_n	uid_amt_ratio	card_n_today
Month
4	0.014	0.012	0.003	0.008	0.001	0.004	1.678	0.095	0.004	0.004
5	0.002	0.013	0.011	0.040	0.001	0.036	1.777	0.193	0.013	0.016
6	0.008	0.009	0.016	0.032	0.004	0.028	2.123	0.219	0.011	0.006

# Purpose: Show which variables drift in month 6, against the PSI limits 0.1 and 0.25.
# Input:   `drift`, `N`.
# Output:  Horizontal bar chart.

d6 = drift.loc[6].sort_values()
XMAX = 0.6  # Bars above this value end at the edge. The label gives the true value.
fig, ax = plt.subplots(figsize=(8, 4.4))
colors = [cs.BLUE if v == "Score" else cs.NAVY for v in d6.index]
ax.barh(d6.index, d6.clip(upper=XMAX), color=colors, height=0.6)
for i, v in enumerate(d6):
    ax.annotate(f"{v:.3f}" + (" →" if v > XMAX else ""), (min(v, XMAX), i), xytext=(4, 0),
                textcoords="offset points", va="center", fontsize=8.5, color=cs.MUTED)
for lim, text in ((0.1, "0.1 stable"), (0.25, "0.25 large change")):
    ax.axvline(lim, color=cs.GRAY, lw=0.8, ls="--")
    ax.annotate(text, (lim, len(d6) - 0.4), xytext=(3, 0), textcoords="offset points", fontsize=8,
                color=cs.MUTED)
ax.set_xlim(0, XMAX * 1.2)
ax.set_xlabel("PSI, month 6 against months 1–3")
cs.action_title(fig, f"The score PSI is {N['psi_score_6']}; {N['psi_max_feat']} drifts most at {N['psi_max_val']}",
                "PSI of the score (blue) and of nine important features (navy), month 6 against months 1–3")
plt.show()

How to read: each bar is the PSI of one variable in month 6. The dashed lines are the limits 0.1 and 0.25. The blue bar is the score.

So what: the score PSI is 0.008, so the population of scores does not move. But Step 6c showed that the PR-AUC goes down. Thus the PSI cannot find this type of drift, and the checks in production must also use the PR-AUC of each month with labels.

The feature with the largest PSI in month 6 is D1n (2.12). D1n is the first day of the card, a date. Its value moves each month, because new cards come into the data. For this reason, the model does not use it (Step 2).

The PSI of uid_n is 0.22. It goes up, because the history of each customer gets longer. In production, a history window of fixed length (for example 90 days) stops this drift.

Retrain schedule

How often do we retrain the score?

Why this question matters. The score loses 15% of its PR-AUC from month 4 to months 5–6 (Step 6c). A weaker score stops less fraud at the same review capacity. But each new model needs a model risk review, a recalibration and new thresholds. Thus a retrain has a cost, and the schedule must pay for it.

Ways to do it. A schedule has two axes: when the retrain happens, and which months the new model learns from.

When	Which months	Method	Advantage	Main risk
Never	Months 1–3	A. No retrain	One model risk review. Stable decisions.	The score loses value when fraud patterns change.
Each month	All months with labels	B. Retrain each month on all labeled months	The score learns new patterns. It keeps rare old patterns.	A review for each new model. The last month can have labels that are not complete.
Each month	The last three months	C. Retrain each month on a window of three months	It forgets old patterns faster.	A smaller training set. Old patterns can come back.
On a trigger	All months with labels	Retrain when the PSI is above a limit	Few retrains.	Step 6c showed that the PSI stays low when the score loses power. Thus the trigger does not fire. We screen it out.
Each day or each week	All transactions	Online learning	It follows the newest patterns.	The fraud label arrives weeks later, and each change needs a review. We screen it out.

Evidence. The code scores months 5 and 6 with each of the options A, B and C. Each retrain uses only the months before the scored month. Thus each option sees the same data that production has on that date. The metrics are the PR-AUC and the dollar recall in the top 2%. For month 4, all three options are the same model m_b, because all three train on months 1–3.

# Purpose: Compare three retrain schedules on months 5 and 6 (dec-07).
# Input:   `txn`, `m_b`, `fit_gbm()`, `predict()`, `score_metrics()`, `FEATS`, `CAT_COLS`, `score_valid`.
# Output:  Table `dec07` (option × metric × month, months 5–6), `pr_by_month` (PR-AUC of months 4–6 for the chart).

rows = []
# Month 4: all options use m_b (trained on months 1–3).
r4 = score_metrics(valid, score_valid)
for name in ["A. No retrain", "B. Monthly, all months", "C. Monthly, last 3 months"]:
    rows.append({"Month": 4, "Option": name, "PR-AUC": r4["PR-AUC"],
                 "Dollar recall, top 2%": r4["Dollar recall, top 2%"]})
for mo in [5, 6]:
    part = txn[txn["month"] == mo]
    # Each retrain uses only the months before the scored month.
    m_all = fit_gbm(txn[txn["month"] < mo], FEATS, CAT_COLS)
    m_win = fit_gbm(txn[txn["month"].between(mo - 3, mo - 1)], FEATS, CAT_COLS)
    for name, m in [("A. No retrain", m_b), ("B. Monthly, all months", m_all),
                    ("C. Monthly, last 3 months", m_win)]:
        r = score_metrics(part, predict(m, part))
        rows.append({"Month": mo, "Option": name, "PR-AUC": r["PR-AUC"],
                     "Dollar recall, top 2%": r["Dollar recall, top 2%"]})
long = pd.DataFrame(rows)
pr_by_month = long.pivot(index="Month", columns="Option", values="PR-AUC")
dec07 = long[long["Month"] >= 5].pivot(index="Option", columns="Month")
mean_pr = dec07["PR-AUC"].mean(axis=1)
mean_usd = dec07["Dollar recall, top 2%"].mean(axis=1)
N.update({"d7_a": f"{mean_pr.iloc[0]:.3f}", "d7_b": f"{mean_pr.iloc[1]:.3f}",
          "d7_c": f"{mean_pr.iloc[2]:.3f}",
          "d7_usd_a": f"{mean_usd.iloc[0]:.1%}", "d7_usd_b": f"{mean_usd.iloc[1]:.1%}",
          "d7_gain": f"{mean_pr.iloc[1] / mean_pr.iloc[0] - 1:+.0%}",
          "d7_bc": f"{abs(mean_pr.iloc[2] - mean_pr.iloc[1]):.3f}",
          "d7_ab": f"{mean_pr.iloc[1] - mean_pr.iloc[0]:.3f}"})
dec07.round(3)

	PR-AUC		Dollar recall, top 2%
Month	5	6	5	6
Option
A. No retrain	0.496	0.478	0.295	0.246
B. Monthly, all months	0.580	0.571	0.396	0.346
C. Monthly, last 3 months	0.597	0.564	0.399	0.345

# Purpose: Show the PR-AUC of each retrain option over months 4–6 (dec-07).
# Input:   `pr_by_month`, `N`.
# Output:  Line chart.

fig, ax = plt.subplots(figsize=(8, 4.2))
style = {"A. No retrain": (cs.GRAY, 1.6), "B. Monthly, all months": (cs.NAVY, 2.2),
         "C. Monthly, last 3 months": (cs.CYAN, 1.6)}
for opt, (color, lw) in style.items():
    y = pr_by_month[opt]
    ax.plot(y.index, y, color=color, lw=lw, marker="o", ms=4)
    # Direct labels at the line end. B and C end near each other, so C moves down.
    dy = {"A. No retrain": 0, "B. Monthly, all months": 7, "C. Monthly, last 3 months": -7}[opt]
    cs.label_end(ax, y.index[-1], y.iloc[-1], opt, color=color, dy=dy, bold=opt.startswith("B"))
# Blue: the month 4 start point, the same model for all options.
ax.plot(4, pr_by_month.iloc[0, 0], "o", color=cs.BLUE, ms=8, zorder=5)
ax.set_xticks([4, 5, 6])
ax.set_xticklabels(["Month 4\n(validation)", "Month 5", "Month 6"])
ax.set_xlim(3.8, 7.4)
ax.set_ylabel("PR-AUC")
cs.action_title(fig, f"A monthly retrain lifts the PR-AUC from {N['d7_a']} to {N['d7_b']}",
                "Each retrain uses only the months before the scored month. In month 4, all three options are the same model.")
plt.show()

How to read: each line is one retrain option. All lines start at the same point in month 4, because all options use the same model there. A higher line finds fraud better.

So what: with no retrain, the PR-AUC goes down after month 4. A monthly retrain keeps more of the power of the score.

Decision. Option B: retrain each month on all months with labels.

Why. The mean PR-AUC on months 5 and 6 is:

• A. No retrain: 0.487.
• B. Monthly retrain on all months: 0.576.
• C. Monthly retrain on the last three months: 0.580.

A retrain adds 0.089 of PR-AUC (+18%). In the top 2% of scores, the dollar recall goes from 27.0% to 37.1%. Options B and C differ by only 0.005. We select option B, because it has more training data and keeps rare fraud patterns from older months. The recalibration (Step 5a) and the thresholds (Step 5c) must follow each retrain.

What would change the decision. If the PR-AUC stays stable for three months, a retrain each quarter is enough. It costs less review work. If the chargebacks come late, the last month has labels that are not complete. Then train on all months except the last one.

The policy result of Step 6d uses option A, the score with no retrain. Thus the decrease of the total cost in Step 6d is a low estimate.

Checks in production

The checks follow from the drift results. The PSI is cheap and fast, but it does not see a change of the fraud patterns. The PR-AUC sees this change, but it needs the labels. Thus the plan uses both.

Frequency	Check	Limit	Action
Each day	Alerts each day against the capacity; decline rate of all transactions	Capacity of 120 alerts; 3% (as-08)	Fraud operations move the extra alerts to the next day. Above the limit for two weeks: stop the challenger.
Each week	PSI of the score and of the top features, against the training months	0.1 examine; 0.25 act	Above 0.25: examine the change before the next retrain.
Each month	PR-AUC and dollar recall of the last month with labels	80% or more of the PR-AUC of the validation month (branch 3)	Retrain (dec-07), recalibrate (Step 5a) and set the thresholds again on the last month with labels.
Each month	Decline rate of good transactions, with labels	3% or less (as-08)	Above the limit: stop the challenger (Step 7c).

Business note. The population of transactions stays stable, but the fraud patterns change fast. With no retrain, the score loses 15% of its PR-AUC in two months. A monthly retrain on all months with labels gives a mean PR-AUC of 0.576, against 0.487 with no retrain. The checks in production must use the PR-AUC of each month, not only the PSI. The conclusion collects the results of all steps.

Conclusion

The notebook started with one question: which approve, review and decline policy gives the lowest total cost of fraud, false declines and reviews? Steps 0 to 7 answered it for the test months. Part 2 then showed how fast the answer gets old.

The answer

The challenger policy gives a lower total cost than the champion. It uses a GBM fraud score and the expected cost in dollars, with the same review capacity. On the test months, it decreases the total cost by $126,820 each month (42%) against our base champion.

Against the strongest champion of the ladder, the decrease is $64,274 each month. The champion is our assumption, so the lower value is the result that we can defend. The decrease is at least $53,544 each month for each cost assumption that we tested. It stops 71.2% of the fraud dollars, against 38.4% for the champion.

The price is more declines of good customers: 2.46% against 0.28%. This rate is inside the guardrail of 3% (as-08). The challenger passes 4 of the 5 branches of the evaluation framework. The branches that fail are: explainability.

Recommendation

1. Test the challenger policy. Send a random 10% of transactions and 10% of the review capacity to the challenger. Run the test for about 16 days (Step 7c). Stop the test if a kill criterion applies.
2. Confirm two values before the test. Finance and Marketing confirm the cost of a false decline (as-02) and the guardrail limit of 3% (as-08). These values decide the size of the decline tier (Step 7b).
3. Retrain the score each month on all months with labels. A monthly retrain lifts the mean PR-AUC from 0.487 to 0.576 (Part 2).
4. Check the PR-AUC of each month, not only the PSI. The PSI of the score stays at 0.008, but the score loses 15% of its PR-AUC (Step 6c).
5. Do not change the team size before the test. The lowest sum of the total cost and the team cost is at 4 analysts. It is only $1,534 each month below the current team (Step 7a).

What a reviewer should challenge

These limits can change the result most. Each one has a control.

Limitation	Effect on the result	Control, or what to do next
The costs are assumptions (as-01 to as-05).	The dollar values and the size of the decline tier change.	The sensitivity check of Step 7b. Finance and Fraud Operations confirm the values.
The guardrail limit of 3% is an assumption (as-08).	A lower limit forces fewer declines and a smaller decrease of the total cost.	Marketing confirms the limit before the test. The kill criteria use it.
The champion is an assumption (as-06). We built it because the data has no real rules.	The gain of the challenger depends on the champion: $64,274 to $126,820 each month.	The ladder of four champions in Step 6d. Both policies use the same review capacity and cost method. The test of Step 7c measures the gain against the real rules.
The surrogate tree does not explain most declines (branch 5).	Fraud operations cannot give a short reason for each decline.	A fraud decline needs no adverse action notice. If a partner asks for reasons, use the alternative of Step 4b.
The PSI does not find a change of the fraud patterns.	A score can lose value with a stable PSI.	Check the PR-AUC of each month with labels. Retrain each month (dec-07).
The label includes later transactions of a customer after a chargeback (as-07).	The history features can look stronger than in production.	History from earlier transactions only (dec-02).
The data does not give the date of the chargeback.	The retrain test assumes that all labels are complete.	Train on all months except the last one when the labels are late.
The policy result uses the score with no retrain.	The decrease of the total cost is a low estimate.	Part 2 shows the gain of a monthly retrain.

Other limits have a small effect. The data has no customer ID, so uid is a proxy. Vesta masked most features, so reason codes from the raw features have little meaning. The data is a sample of the payment flow of one merchant, so the dollar values are small. Scale the result with the actual volume. Real-time latency is a task for engineering.

Next steps

1. Business case deck (6–8 slides): the challenger policy for a champion/challenger test on 10% of transactions.
2. Confirm the cost assumptions with Finance (as-01, as-02), Fraud Operations (as-03, as-05) and Marketing (as-08).
3. A one-pager for fraud operations: the review capacity, the value of one more analyst and the surrogate rules as patterns.
4. Add a feature “earlier chargebacks of the customer” when the chargeback dates are available (dec-02).
5. Set up the checks in production of Part 2 before the start of the test.

Appendix A: data dictionary

The table gives each column that the notebook uses, then each group of masked columns. Vesta masked the meaning of most columns. The meaning comes from the data description of Vesta on the Kaggle competition page. It also uses the answers of Vesta in the Kaggle discussion. The other parts of the table come from this notebook:

• Bucket. “Feature” means that the fraud score uses the column. Any other value is the bucket of the funnel in Step 4a: 1. Not known at the decision, 2. No stable risk information, 3. Removed to keep the model small.
• Rule. The first rule of the funnel that removes the column. A group row counts the columns of the group for each rule.
• Detail. The evidence for the rule, or the type and the share of empty cells of a feature.
• Other use. The use of the column outside the fraud score, for example the time order or the customer key.
• Empty. The share of empty cells in all transactions. For a group row, it is the mean of the loaded columns of the group. A dash means that the code does not load the column.

The first table counts the columns in each bucket. The second table shows the named columns and the engineered features first, then the groups of masked columns.

# Purpose: Count the columns of Appendix A in each bucket.
# Input:   `column_status`.
# Output:  Small table `dd_counts`.

dd_order = ["Feature", "1. Not known at the decision", "2. No stable risk information",
            "3. Removed to keep the model small"]
dd_counts = column_status.groupby("bucket")["column"].size().reindex(dd_order).fillna(0).astype(int)
dd_counts.rename("Columns").rename_axis("Bucket").to_frame()

	Columns
Bucket
Feature	100
1. Not known at the decision	1
2. No stable risk information	6
3. Removed to keep the model small	344

# Purpose: Build Appendix A: one row for each named column and engineered feature, one row for each masked group.
# Input:   `column_status`, `txn`.
# Output:  Table `dictionary`.

# Meaning of the named columns: the data description of Vesta (Kaggle), without the masked details.
MEANING = {
    "TransactionID": "ID of the transaction. Key of the transaction file and of the identity file.",
    "isFraud": "Fraud label: 1 = fraud. Vesta labels a transaction with a chargeback as fraud. It also labels the later "
               "transactions of the same account, email address or billing address (as-07).",
    "TransactionDT": "Time difference in seconds from a reference date and time. Vesta did not publish the reference.",
    "TransactionAmt": "Amount of the payment in US dollars.",
    "ProductCD": "Product code: the product of the transaction (masked: W, C, R, H, S).",
    "card1": "Payment card information (masked), for example the card type, the card category, the bank or the country.",
    "card2": "Payment card information (masked).", "card3": "Payment card information (masked).",
    "card4": "Card network (for example visa or mastercard).", "card5": "Payment card information (masked).",
    "card6": "Card type (for example debit or credit).",
    "addr1": "Billing region (masked).", "addr2": "Billing country (masked).",
    "dist1": "Distance between two addresses of the transaction, for example the billing address and the IP address (masked).",
    "dist2": "A second distance between two addresses (masked).",
    "P_emaildomain": "Email domain of the purchaser.", "R_emaildomain": "Email domain of the recipient.",
    "DeviceType": "Device type: desktop or mobile (identity file).",
    "DeviceInfo": "Device name or system version (identity file).",
    "id_30": "Operating system and version (identity file).", "id_31": "Browser and version (identity file).",
    "id_33": "Screen resolution (identity file).",
    "day": "Day number: TransactionDT divided by 86,400 seconds (Step 2a).",
    "month": "Block of 30 days, 1 to 6. The last days of the data go into month 6 (Step 2a).",
    "hour": "Hour of the reference clock, 0 to 23. It is not local time (Step 2a).",
    "D1n": "First day of the card: day − D1 (Step 2b).",
    "uid": "Customer key (a proxy): card1 + addr1 + D1n (Step 2b).",
    "log_amt": "ln(1 + TransactionAmt) (Step 2d).",
    "cents": "The part of the amount after the decimal point (Step 2d).",
    "email_match": "1 if the purchaser and the recipient have the same email domain (Step 2d).",
    "has_identity": "1 if the transaction has a row in the identity file (Step 2d).",
}
HIST_TEXT = {"_n": "Number of earlier transactions of the {k}.",
             "_amt_ratio": "Amount divided by the mean amount of the earlier transactions of the {k}.",
             "_hours_since": "Hours since the last earlier transaction of the {k}.",
             "_n_today": "Number of earlier transactions of the {k} on the same day."}
for c in HIST:
    prefix, rest = c.split("_", 1)
    k = "customer key uid" if prefix == "uid" else "card (card1)"
    MEANING[c] = HIST_TEXT["_" + rest].format(k=k) + " History feature (Step 2d)."
# Masked groups: one row each.
GROUPS_DD = {
    "C: counts": ("C1–C14", "Counts, for example the number of addresses that Vesta links to the card. Vesta masked the meaning."),
    "D: time deltas": ("D1–D15", "Time differences in days, for example the days since the last transaction. "
                       "D1 is the age of the card."),
    "M: match flags": ("M1–M9", "Match flags, for example a match of the name on the card and the address."),
    "V: Vesta features": ("V1–V339", "Features that Vesta made from its own data: ranks, counts and other links "
                          "between entities."),
    "Identity: id_": ("id_01–id_38", "Identity data: network connection (IP, ISP, proxy) and digital signature "
                      "(browser, system, version). Vesta and its security partners collect it. Vesta masked "
                      "the names. The model uses only id_30, id_31 and id_33 of this group."),
}
OTHER_USE = {
    "TransactionID": "Join of the two files", "isFraud": "Target",
    "TransactionDT": "Time order, history features", "TransactionAmt": "Fraud loss, expected cost, history features",
    "day": "History features, review capacity by day", "month": "Split (Step 3), results by month",
    "D1n": "Part of uid", "uid": "Key of the history features", "card1": "Part of uid, key of the history features",
    "addr1": "Part of uid", "D: time deltas": "D1 is a part of uid",
}


def empty_share(cols):
    """Mean share of empty cells of the loaded columns, or a dash."""
    cols = [c for c in cols if c in txn.columns]
    return f"{txn[cols].isna().mean().mean():.1%}" if cols else "—"


named = column_status[~column_status["group"].isin(GROUPS_DD) | column_status["column"].isin(MEANING)]
rows = [{"Column": c, "Meaning": MEANING.get(c, ""), "Bucket": b, "Rule": r, "Detail": d,
         "Other use": OTHER_USE.get(c, ""), "Empty": empty_share([c])}
        for c, b, r, d in named[["column", "bucket", "rule", "detail"]].itertuples(index=False)]
for grp, (span, text) in GROUPS_DD.items():
    part = column_status[column_status["group"] == grp]
    counts = part.groupby("rule")["column"].size()
    rows.append({"Column": f"{span} ({len(part)} columns)", "Meaning": text,
                 "Bucket": "; ".join(f"{b}: {n}" for b, n in part.groupby("bucket")["column"].size().items()),
                 "Rule": "; ".join(f"{r}: {n}" for r, n in counts.items()),
                 "Detail": ("Features: " + ", ".join(part.loc[part["rule"] == "Feature", "column"]))
                           if (part["rule"] == "Feature").any() else "",
                 "Other use": OTHER_USE.get(grp, ""), "Empty": empty_share(part["column"])})
dictionary = pd.DataFrame(rows)
# Sort: features first, then the buckets in order, then the masked groups.
dictionary["_g"] = dictionary["Column"].str.contains(r"\(\d+ columns\)").astype(int)
dictionary["_b"] = dictionary["Bucket"].map({b: i for i, b in enumerate(dd_order)}).fillna(len(dd_order))
dictionary = (dictionary.sort_values(["_g", "_b"], kind="stable")
              .drop(columns=["_g", "_b"]).set_index("Column"))
assert (dictionary["Meaning"] != "").all(), "Each row needs a meaning."
N["n_dict_rows"] = f"{len(dictionary)}"
(dictionary.style.set_properties(**{"text-align": "left", "vertical-align": "top"})
 .set_properties(subset=["Meaning"], **{"min-width": "320px"}))

	Meaning	Bucket	Rule	Detail	Other use	Empty
Column
TransactionAmt	Amount of the payment in US dollars.	Feature	Feature	numeric; 0% empty	Fraud loss, expected cost, history features	0.0%
ProductCD	Product code: the product of the transaction (masked: W, C, R, H, S).	Feature	Feature	text; 0% empty		0.0%
card1	Payment card information (masked), for example the card type, the card category, the bank or the country.	Feature	Feature	numeric; 0% empty	Part of uid, key of the history features	0.0%
card2	Payment card information (masked).	Feature	Feature	numeric; 2% empty		1.5%
card3	Payment card information (masked).	Feature	Feature	numeric; 0% empty		0.3%
card4	Card network (for example visa or mastercard).	Feature	Feature	text; 0% empty		0.3%
card5	Payment card information (masked).	Feature	Feature	numeric; 1% empty		0.7%
card6	Card type (for example debit or credit).	Feature	Feature	text; 0% empty		0.3%
addr1	Billing region (masked).	Feature	Feature	numeric; 11% empty	Part of uid	11.1%
addr2	Billing country (masked).	Feature	Feature	numeric; 11% empty		11.1%
dist1	Distance between two addresses of the transaction, for example the billing address and the IP address (masked).	Feature	Feature	numeric; 60% empty		59.7%
dist2	A second distance between two addresses (masked).	Feature	Feature	numeric; 94% empty		93.6%
P_emaildomain	Email domain of the purchaser.	Feature	Feature	text; 16% empty		16.0%
R_emaildomain	Email domain of the recipient.	Feature	Feature	text; 77% empty		76.8%
id_30	Operating system and version (identity file).	Feature	Feature	text; 87% empty		86.9%
id_31	Browser and version (identity file).	Feature	Feature	text; 76% empty		76.2%
id_33	Screen resolution (identity file).	Feature	Feature	text; 88% empty		87.6%
DeviceType	Device type: desktop or mobile (identity file).	Feature	Feature	text; 76% empty		76.2%
DeviceInfo	Device name or system version (identity file).	Feature	Feature	text; 80% empty		79.9%
hour	Hour of the reference clock, 0 to 23. It is not local time (Step 2a).	Feature	Feature	numeric; 0% empty		0.0%
log_amt	ln(1 + TransactionAmt) (Step 2d).	Feature	Feature	numeric; 0% empty		0.0%
cents	The part of the amount after the decimal point (Step 2d).	Feature	Feature	numeric; 0% empty		0.0%
email_match	1 if the purchaser and the recipient have the same email domain (Step 2d).	Feature	Feature	numeric; 0% empty		0.0%
has_identity	1 if the transaction has a row in the identity file (Step 2d).	Feature	Feature	numeric; 0% empty		0.0%
uid_n	Number of earlier transactions of the customer key uid. History feature (Step 2d).	Feature	Feature	history; 11% empty		11.3%
uid_amt_ratio	Amount divided by the mean amount of the earlier transactions of the customer key uid. History feature (Step 2d).	Feature	Feature	history; 45% empty		45.0%
uid_hours_since	Hours since the last earlier transaction of the customer key uid. History feature (Step 2d).	Feature	Feature	history; 45% empty		45.0%
uid_n_today	Number of earlier transactions of the customer key uid on the same day. History feature (Step 2d).	Feature	Feature	history; 11% empty		11.3%
card_n	Number of earlier transactions of the card (card1). History feature (Step 2d).	Feature	Feature	history; 0% empty		0.0%
card_amt_ratio	Amount divided by the mean amount of the earlier transactions of the card (card1). History feature (Step 2d).	Feature	Feature	history; 2% empty		2.3%
card_hours_since	Hours since the last earlier transaction of the card (card1). History feature (Step 2d).	Feature	Feature	history; 2% empty		2.3%
card_n_today	Number of earlier transactions of the card (card1) on the same day. History feature (Step 2d).	Feature	Feature	history; 0% empty		0.0%
isFraud	Fraud label: 1 = fraud. Vesta labels a transaction with a chargeback as fraud. It also labels the later transactions of the same account, email address or billing address (as-07).	1. Not known at the decision	1. Label	Target. Includes later transactions after a chargeback (as-07)	Target	0.0%
TransactionID	ID of the transaction. Key of the transaction file and of the identity file.	2. No stable risk information	2. Key or identifier	Key of both files	Join of the two files	0.0%
TransactionDT	Time difference in seconds from a reference date and time. Vesta did not publish the reference.	2. No stable risk information	3. Time stamp or date	Time order of the rows and the split (Step 3)	Time order, history features	0.0%
day	Day number: TransactionDT divided by 86,400 seconds (Step 2a).	2. No stable risk information	3. Time stamp or date	Day number; history and capacity by day	History features, review capacity by day	0.0%
month	Block of 30 days, 1 to 6. The last days of the data go into month 6 (Step 2a).	2. No stable risk information	3. Time stamp or date	Split and results by month (Step 3)	Split (Step 3), results by month	0.0%
D1n	First day of the card: day − D1 (Step 2b).	2. No stable risk information	3. Time stamp or date	First day of the card; a date that moves each month (Step 2b, Part 2)	Part of uid	0.2%
uid	Customer key (a proxy): card1 + addr1 + D1n (Step 2b).	2. No stable risk information	2. Key or identifier	Customer proxy (card1 + addr1 + D1n); key of the history features	Key of the history features	11.3%
C1–C14 (14 columns)	Counts, for example the number of addresses that Vesta links to the card. Vesta masked the meaning.	Feature: 14	Feature: 14	Features: C1, C2, C3, C4, C5, C6, C7, C8, C9, C10, C11, C12, C13, C14		0.0%
D1–D15 (15 columns)	Time differences in days, for example the days since the last transaction. D1 is the age of the card.	Feature: 15	Feature: 15	Features: D1, D2, D3, D4, D5, D6, D7, D8, D9, D10, D11, D12, D13, D14, D15	D1 is a part of uid	58.2%
M1–M9 (9 columns)	Match flags, for example a match of the name on the card and the address.	Feature: 9	Feature: 9	Features: M1, M2, M3, M4, M5, M6, M7, M8, M9		49.9%
V1–V339 (339 columns)	Features that Vesta made from its own data: ranks, counts and other links between entities.	3. Removed to keep the model small: 309; Feature: 30	4. V column with the same missing pattern: 309; Feature: 30	Features: V6, V7, V17, V18, V37, V38, V55, V56, V77, V78, V127, V128, V159, V160, V162, V163, V203, V204, V209, V210, V264, V265, V271, V272, V307, V308, V314, V315, V332, V333		45.5%
id_01–id_38 (38 columns)	Identity data: network connection (IP, ISP, proxy) and digital signature (browser, system, version). Vesta and its security partners collect it. Vesta masked the names. The model uses only id_30, id_31 and id_33 of this group.	3. Removed to keep the model small: 35; Feature: 3	5. Masked identity column, not loaded: 35; Feature: 3	Features: id_30, id_31, id_33		83.6%

Reader guides: decisions, assumptions and glossary

This appendix holds the reference material of the notebook. Click a title to open it.

NoteHow to read this notebook

• Read the text to follow the logic. Click Show code to see the code for a step.
• The notebook follows the steps of a model project, from Step 0 to Step 7. Part 2 uses the same data for the drift and the retrain schedule.
• Each step starts with a gray block. This block says what the step before showed, and the question of this step.
• A blue block records a decision. It starts with a question. Then it gives the possible methods, the evidence, the decision and the reason.
• A gray-blue block (Technical note) gives the method in detail for a data scientist. A business reader can skip it.
• A green block is a business note. It gives the result of a step for a manager.
• Appendix A gives the meaning of each column in the data.

Each team can read only the part that it needs.

Part	Question	Team
Step 0	What is the business problem?	All
Steps 1–3	Which data can answer it? How do we test the answer?	All
Steps 4–5	Which transactions are fraud? Which action does each transaction get?	Fraud strategy
Step 6	Is the challenger good enough for a test?	Fraud strategy, model risk
Step 7	What is the value of one more analyst? How do we test the policy?	Fraud strategy, fraud operations, Finance
Part 2	Does the score stay good over time? When do we retrain it?	Fraud strategy, model risk

NoteDecision log

The decision blocks in the text show only the question. This log gives each decision an ID for a model risk review. Click the question to go to the decision.

ID	Question	Decision	Step
dec-01	Which split gives a test result that is near to production?	Out-of-time split: train months 1–3, validation month 4, test months 5–6	3b
dec-02	Which customer history can the model use?	Features from earlier transactions only	4a
dec-03	Which model gives the fraud score?	GBM. The rule before the evidence: the GBM replaces the logistic regression only if its PR-AUC on month 4 is 10% or more higher.	4b
dec-04	How do we handle the low fraud rate?	No class weight and no resampling. Recalibrate the score to the fraud rate of month 4.	4c
dec-05	How many actions does the policy have?	Three tiers: approve, review and decline, with the review capacity as a limit	5c
dec-06	Which value puts the transactions in order for the policy?	The expected cost in dollars, not the probability of fraud	5d
dec-07	How often do we retrain the score?	Retrain each month on all months with labels	Part 2

NoteAssumption register

ID	Assumption	Value	Source	Effect if the value is wrong
as-01	Loss for an approved fraud transaction	100% of the amount	Assumption: card-not-present fraud, the business pays the chargeback	Large. The sensitivity check (Step 7b) uses 50%.
as-02	Cost of a false decline	3% of the amount (lost margin) + $10 (customer friction and attrition)	Assumption. Confirm with Finance and Marketing.	Large. It sets the decline tier. The sensitivity check uses $5 and $20.
as-03	Review capacity	3 analysts × 40 alerts each day = 120 alerts each day	Assumption: about 10 minutes for each alert. Confirm with Fraud Operations.	Large. Step 7a gives the value of one more analyst.
as-04	Cost of one analyst	$300 each day	Assumption: about $75,000 each year with overhead	It changes only the value of one more analyst.
as-05	Result of a review	The analyst stops 90% of the fraud alerts. A good customer waits and has a friction cost of $2.	Assumption	Medium. A lower catch rate makes the review tier smaller.
as-06	Champion policy	Rules from a decision tree with 3 levels. The tree uses month 1 data and does not change after that. Inside each rule segment, the analysts review the largest amounts first.	Assumption. The data does not give the rules of the merchant.	The gain of the challenger changes. The method stays the same.
as-07	Fraud label	The isFraud label of the dataset. It includes later transactions of a customer after a chargeback.	Dataset documentation (Vesta, Kaggle discussion)	The fraud rate can be too high for some customers. The history features become stronger. See dec-02.
as-08	Guardrail limit for the decline rate of good transactions	3% of the good transactions	Assumption, set before the rerun (evaluation framework, branch 4). Marketing to confirm.	Medium. A lower limit can stop the challenger in the test. Step 6d and Step 7 compare the decline rate of good transactions with this limit.

NoteGlossary

The words in this table are technical names. The notebook uses each name only with the meaning below.

Technical name	Meaning in this notebook
Fraud	A transaction with isFraud = 1.
Fraud rate	The number of fraud transactions divided by the number of transactions.
Fraud score	The probability of fraud that the model gives to a transaction.
Champion	The current policy: static rules.
Challenger	The new policy that we test against the champion.
Champion/challenger test	A test that sends a random share of the transactions to the challenger and the rest to the champion.
Approve, review, decline	The three actions of the policy. A review sends the transaction to an analyst before the approval.
Alert	A transaction in the review tier.
Review capacity	The maximum number of alerts that the analysts can examine in one day.
False decline	A decline of a good transaction.
Decline rate of good transactions	The number of declined good transactions divided by the number of good transactions.
Chargeback	A refund to the card holder after a report of fraud.
Total cost	Fraud loss + false decline cost + review friction cost. The cost of the analysts is separate.
Expected cost	The total cost that we expect for one transaction and one action, from the fraud score and the amount.
Customer key (uid)	A proxy for the customer: card + billing address + the first day of the card. The data has no customer ID.
History features	Features from the earlier transactions of the same customer key or card.
Feature engineering	The work that makes new features from the raw columns.
Out-of-time split	A split where the test data comes from a later period than the training data.
Leakage	Information in a feature that is not available at the time of the decision.
GBM (gradient boosting machine)	A model that adds many small decision trees. Each tree corrects the errors of the previous trees.
Class weight	A larger weight for the fraud transactions in the training.
Undersampling	A removal of a random part of the good transactions from the training data.
SMOTE	A method that adds synthetic fraud transactions to the training data. It makes new rows between near fraud transactions.
PR-AUC	Area under the precision-recall curve (average precision). A random score gives the fraud rate.
AUC	Area under the ROC curve. 0.5 is random. 1.0 is perfect.
Precision	The share of fraud in the flagged transactions.
Recall	The share of all fraud that the policy flags.
Dollar recall	The share of the fraud amount that the policy flags.
Calibration	The match between the mean fraud score and the actual fraud rate.
Recalibration	A change of the level of the fraud score that keeps the order of the transactions.
Log-odds	ln(p / (1 − p)).
Evaluation framework	The five branches of Step 3a: order of risk, level, stability, business value and explainability. Each branch has a rule that we set before the evidence.
PSI (population stability index)	A measure of the change in a distribution between two samples. Less than 0.1 is stable. More than 0.25 is a large change.
Drift	A change of the data or of the relation between the features and fraud over time.
Online learning	A model that learns from each new transaction or each new day of data.
Shadow price	The decrease of the total cost from one more unit of a limited resource.
Surrogate tree	A small decision tree that copies the decisions of a model, so that people can read the rules.
Adverse action notice	A letter that gives the main reasons for a decline of credit. A fraud decline needs no adverse action notice.
Guardrail	A metric that the test checks each week. It must stay inside a limit.
Kill criteria	The conditions that stop a test before its end.
Significance level	The probability that a test finds a difference when no difference exists.
Power	The probability that a test finds a difference when the difference exists.
Business note	A short result of a step for a manager.
Technical note	Detail of a method for a data scientist. A business reader can skip it.

Appendix B: reproducibility

# Purpose: Print the versions of the packages, for reproducibility.
# Input:   None.
# Output:  Text.

import sys
import sklearn
print("Python", sys.version.split()[0])
print("numpy", np.__version__, "| pandas", pd.__version__,
      "| scikit-learn", sklearn.__version__, "| duckdb", duckdb.__version__)
print("Seed:", SEED)

Python 3.13.16
numpy 2.5.3 | pandas 3.0.5 | scikit-learn 1.9.1 | duckdb 1.5.6
Seed: 42

Appendix C: functions used in the analysis

Steps 3 to 7 and Part 2 use the functions in the file case02_helpers.py. They give the feature matrix and the GBM, the metrics of the evaluation framework, the recalibration, the cost of each action and the policies. A reviewer can read the full source below, without a separate file.

# Purpose: Show the source of the helper module in the notebook.
# Input:   File case02_helpers.py (next to this notebook).
# Output:  The source as a Python code block, or a note if the file does not exist.

from pathlib import Path

helpers_file = Path("case02_helpers.py")
if helpers_file.exists():
    fence = "`" * 3  # A literal fence here would end the cell in the .qmd file
    print(fence + "python\n" + helpers_file.read_text(encoding="utf-8").rstrip("\n") + "\n" + fence)
else:
    print("*The file `case02_helpers.py` does not exist. The helper functions are in the cells of Step 3.*")

"""Helper functions for case 02, Steps 3 to 7 and Part 2.

# Purpose: One place for the tools that Steps 3-7 use: the feature matrix and the GBM, the metrics of the
#          evaluation framework (PR-AUC, dollar recall, PSI), the recalibration, the cost of each action
#          and the approve / review / decline policies.
# Input:   configure(seed=..., cost=...) one time. The notebook gives the seed and the cost assumptions.
# Output:  Functions and constants. The notebook loads them with `from case02_helpers import *`.

The configuration objects (GBM_PARAMS, COST_REF) are dicts that configure() changes in place.
Thus the names that the notebook imports always show the current configuration.
A function with a cost argument `c=None` uses COST_REF, the cost assumptions of the notebook (as-01 to as-05).
"""
import numpy as np
import pandas as pd
from sklearn.ensemble import HistGradientBoostingClassifier
from sklearn.metrics import average_precision_score, roc_auc_score

__all__ = [
    "configure", "GBM_PARAMS", "COST_REF",
    "encode", "fit_gbm", "predict",
    "top_k", "score_metrics", "dollar_recall_curve", "psi", "METRIC_FMT",
    "shift_logodds",
    "expected_costs", "actual_cost", "capacity", "threshold_for_capacity",
    "fit_policy", "apply_policy", "summary", "SUM_FMT",
]

# --- Configuration (set by configure()) ---
# One set of GBM settings for all GBMs of the notebook. Early stop on 10% of the training rows.
GBM_PARAMS = dict(learning_rate=0.08, max_iter=400, max_leaf_nodes=63, min_samples_leaf=100,
                  l2_regularization=1.0, early_stopping=True, validation_fraction=0.1,
                  n_iter_no_change=30, random_state=None)
COST_REF = {}  # The cost assumptions of the notebook (COST). configure() fills it.


def configure(seed=None, cost=None):
    """Set the seed of the GBM and the cost assumptions that the policy functions use by default."""
    if seed is not None:
        GBM_PARAMS["random_state"] = seed
    if cost is not None:
        COST_REF.clear()
        COST_REF.update(cost)


def _cost(c):
    """Return the cost assumptions: the argument if given, else the assumptions of the notebook."""
    if c is not None:
        return c
    if not COST_REF:
        raise RuntimeError("Call configure(cost=COST) before the policy functions.")
    return COST_REF


# --- Feature matrix and GBM ---
def encode(frame, ref, cols_num, cols_cat):
    """Return the feature matrix. A text feature with 200 levels or fewer becomes a category with the
    levels of `ref` (the training data). A text feature with more levels becomes its frequency in `ref`."""
    X = frame[cols_num].astype(float).copy()
    for c in cols_cat:
        counts = ref[c].value_counts()
        if len(counts) <= 200:
            # A level that is not in the training data becomes missing.
            X[c] = pd.Categorical(frame[c].where(frame[c].isin(counts.index)), categories=counts.index)
        else:
            X[c + "_freq"] = frame[c].map(counts).astype(float)
    return X


def fit_gbm(frame, cols_num, cols_cat, class_weight=None, sample=None):
    """Return a GBM with GBM_PARAMS, fitted on `frame` (or on `sample`, a subset of rows of `frame`).
    The category levels always come from `frame`. The GBM uses missing values directly: no imputation."""
    fit_frame = frame if sample is None else sample
    X = encode(fit_frame, frame, cols_num, cols_cat)
    model = HistGradientBoostingClassifier(**GBM_PARAMS, categorical_features="from_dtype",
                                           class_weight=class_weight)
    model.fit(X, fit_frame["isFraud"])
    return {"model": model, "ref": frame, "num": cols_num, "cat": cols_cat}


def predict(m, frame):
    """Return the fraud score of a fitted GBM (from fit_gbm) for each row of `frame`."""
    return m["model"].predict_proba(encode(frame, m["ref"], m["num"], m["cat"]))[:, 1]


# --- Branch 1 of the evaluation framework: order of risk ---
def top_k(score, share):
    """Return True for the `share` of transactions with the highest score."""
    k = int(round(share * len(score)))
    flag = np.zeros(len(score), bool)
    flag[np.argsort(-score, kind="stable")[:k]] = True
    return flag


def score_metrics(frame, score, share=0.02):
    """Return the PR-AUC, the AUC, and the recall and the dollar recall in the top `share` of scores."""
    y = frame["isFraud"].to_numpy()
    amt = frame["TransactionAmt"].to_numpy()
    flag = top_k(np.asarray(score), share)
    return {
        "PR-AUC": average_precision_score(y, score),
        "AUC": roc_auc_score(y, score),
        f"Recall, top {share:.0%}": y[flag].sum() / y.sum(),
        f"Dollar recall, top {share:.0%}": amt[flag & (y == 1)].sum() / amt[y == 1].sum(),
    }


def dollar_recall_curve(frame, score, shares):
    """Return the dollar recall for each share of transactions flagged (highest scores first)."""
    y = frame["isFraud"].to_numpy()
    amt = frame["TransactionAmt"].to_numpy()
    order = np.argsort(-np.asarray(score), kind="stable")
    cum = np.cumsum((amt * y)[order]) / (amt * y).sum()
    k = np.clip(np.round(np.asarray(shares) * len(order)).astype(int), 1, len(order))
    return cum[k - 1]


METRIC_FMT = {"PR-AUC": "{:.3f}", "AUC": "{:.3f}", "Recall, top 2%": "{:.1%}",
              "Dollar recall, top 2%": "{:.1%}"}


# --- Branch 3: stability ---
def psi(expected, actual, bins=10):
    """Return the PSI of a variable between two samples. The bins are the deciles of `expected`."""
    e = pd.Series(expected).dropna().to_numpy()
    a = pd.Series(actual).dropna().to_numpy()
    edges = np.unique(np.quantile(e, np.linspace(0, 1, bins + 1)))
    if len(edges) < 3:
        return 0.0  # One value only: no distribution to compare
    edges[0], edges[-1] = -np.inf, np.inf
    pe = np.histogram(e, edges)[0] / len(e) + 1e-6
    pa = np.histogram(a, edges)[0] / len(a) + 1e-6
    return float(np.sum((pa - pe) * np.log(pa / pe)))


# --- Branch 2: level (recalibration, Step 5a) ---
def shift_logodds(p, delta):
    """Return the score after one shift `delta` of its log-odds. The order of the scores does not change."""
    p = np.clip(p, 1e-6, 1 - 1e-6)
    return 1 / (1 + np.exp(-(np.log(p / (1 - p)) + delta)))


# --- Cost of each action (Step 5b) ---
def expected_costs(p, amt, c=None):
    """Return the expected cost of approve, decline and review for a fraud score p and an amount."""
    c = _cost(c)
    approve = p * amt * c["loss_share"]
    decline = (1 - p) * (c["fd_margin"] * amt + c["fd_friction"])
    review = p * amt * c["loss_share"] * (1 - c["review_catch"]) + (1 - p) * c["review_friction"]
    return approve, decline, review


def actual_cost(frame, action, c=None):
    """Return the actual cost of each transaction with its label. action: 0 approve, 1 review, 2 decline."""
    c = _cost(c)
    y = frame["isFraud"].to_numpy()
    a = frame["TransactionAmt"].to_numpy()
    return np.select(
        [action == 0, action == 1, action == 2],
        [y * a * c["loss_share"],
         y * a * c["loss_share"] * (1 - c["review_catch"]) + (1 - y) * c["review_friction"],
         (1 - y) * (c["fd_margin"] * a + c["fd_friction"])])


# --- Policies (Steps 5c and 5d) ---
def capacity(c=None):
    """Return the review capacity: the number of alerts that the analysts can examine each day (as-03)."""
    c = _cost(c)
    return c["analysts"] * c["alerts_per_analyst"]


def threshold_for_capacity(value, days, cap):
    """Return the lowest threshold on `value` that gives at most `cap` alerts each day on average."""
    n_days = len(np.unique(days))
    k = int(cap * n_days)
    if k >= len(value):
        return -np.inf
    return np.sort(value)[::-1][k]


def fit_policy(kind, p, frame, c=None):
    """Return the rule of a policy, with the thresholds set on `frame` (the validation month).

    kind "prob2":  two tiers. One decline threshold on the score, from a grid search.
    kind "prob3":  three tiers. The decline threshold from a grid search. For each one, the review
                   threshold fills the review capacity with the next highest scores.
    kind "dollar": three tiers from the expected cost. Decline if a decline costs less than an approval.
                   Review the transactions where a review decreases the expected cost most, up to the capacity.
    """
    c = _cost(c)
    amt = frame["TransactionAmt"].to_numpy()
    days = frame["day"].to_numpy()
    if kind == "dollar":
        ap, de, re = expected_costs(p, amt, c)
        gain = np.minimum(ap, de) - re  # Decrease of the expected cost from a review
        t = max(threshold_for_capacity(gain, days, capacity(c)), 0.0)
        return {"kind": kind, "t_gain": t, "c": c}
    # Probability thresholds: grid search of the decline threshold, between the 90% and the 99.95% quantile.
    y_cost = {}
    for t_d in np.quantile(p, np.linspace(0.90, 0.9995, 60)):
        rule = {"kind": kind, "t_d": t_d, "c": c}
        if kind == "prob3":
            rest = p < t_d
            rule["t_r"] = threshold_for_capacity(np.where(rest, p, -1), days, capacity(c))
        y_cost[actual_cost(frame, apply_policy(rule, p, frame), c).sum()] = rule
    return y_cost[min(y_cost)]


def apply_policy(rule, p, frame):
    """Return the action of each transaction (0 approve, 1 review, 2 decline) for a policy rule."""
    amt = frame["TransactionAmt"].to_numpy()
    action = np.zeros(len(p), int)
    if rule["kind"] == "dollar":
        ap, de, re = expected_costs(p, amt, rule["c"])
        action[de < ap] = 2
        gain = np.minimum(ap, de) - re
        action[(gain > rule["t_gain"]) & (gain > 0)] = 1
        return action
    action[p >= rule["t_d"]] = 2
    if rule["kind"] == "prob3":
        action[(p < rule["t_d"]) & (p >= rule["t_r"])] = 1
    return action


def summary(frame, action, c=None):
    """Return the business result of a policy: the cost for each month by component, the share of the
    fraud dollars stopped, the decline rate of good transactions and the alerts each day."""
    c = _cost(c)
    y = frame["isFraud"].to_numpy()
    a = frame["TransactionAmt"].to_numpy()
    cost = actual_cost(frame, action, c)
    n_months = frame["day"].nunique() / 30
    loss = np.where(action == 0, y * a * c["loss_share"], 0) + np.where(
        action == 1, y * a * c["loss_share"] * (1 - c["review_catch"]), 0)
    fd = np.where(action == 2, (1 - y) * (c["fd_margin"] * a + c["fd_friction"]), 0)
    alerts_day = pd.Series(action == 1).groupby(frame["day"].to_numpy()).sum()
    return {
        "Fraud loss ($/month)": loss.sum() / n_months,
        "False decline cost ($/month)": fd.sum() / n_months,
        "Review friction ($/month)": (cost.sum() - loss.sum() - fd.sum()) / n_months,
        "Total cost ($/month)": cost.sum() / n_months,
        "Fraud $ stopped": 1 - loss.sum() / (y * a * c["loss_share"]).sum(),
        "Decline rate of good transactions": ((action == 2) & (y == 0)).sum() / (y == 0).sum(),
        "Alerts each day (mean)": alerts_day.mean(),
        "Days over capacity": (alerts_day > capacity(c)).mean(),
    }


SUM_FMT = {"Fraud loss ($/month)": "{:,.0f}", "False decline cost ($/month)": "{:,.0f}",
           "Review friction ($/month)": "{:,.0f}", "Total cost ($/month)": "{:,.0f}",
           "Fraud $ stopped": "{:.1%}", "Decline rate of good transactions": "{:.2%}",
           "Alerts each day (mean)": "{:.0f}", "Days over capacity": "{:.0%}"}