Case 01: Credit lifecycle

Can a new approval cutoff increase approvals at the same bad rate, and does each grade make money?

Disclaimer. This notebook uses public data. The results are for illustration only. The data is not the data of an employer or a client. The costs and the rates are assumptions. The assumption register in the appendix gives the source of each assumption.

# Purpose: Load the libraries and the chart style. Set the paths, the random seed and two helpers.
# Input:   None.
# Output:  Paths of the data files, `N` (numbers for the text), `fmt()` (table format),
#          chart style `cs`.

from pathlib import Path

import duckdb
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
from scipy.optimize import brentq
from sklearn.ensemble import HistGradientBoostingClassifier, HistGradientBoostingRegressor
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import r2_score, roc_auc_score
from sklearn.model_selection import train_test_split

import chart_style as cs  # One chart style for all notebooks (deck style guide)

cs.setup()

# Fixed seed: each run gives the same samples and the same models.
SEED = 42
rng = np.random.default_rng(SEED)

# Paths are relative to this notebook (cases/01-credit-lifecycle/).
DATA_DIR = Path("../../data")  # The raw data is not in the repository. See the README.
RAW_FILE = DATA_DIR / "raw" / "accepted_2007_to_2018Q4.csv.gz"
INTERIM_FILE = DATA_DIR / "interim" / "loans_36m.parquet"
CAND_FILE = DATA_DIR / "interim" / "candidates_36m_2013_2015.parquet"
DICT_FILE = DATA_DIR / "raw" / "LCDataDictionary.xlsx"  # Lending Club data dictionary (Kaggle dataset)
# Stop early with a clear message if a data file is missing.
if not RAW_FILE.exists() and not (INTERIM_FILE.exists() and CAND_FILE.exists()):
    raise FileNotFoundError(f"Download the data to {RAW_FILE}. See the README.")
if not DICT_FILE.exists():
    raise FileNotFoundError(f"Download LCDataDictionary.xlsx to {DICT_FILE}. See the README.")

N = {}  # Numbers for the text after each cell.



def fmt(table, formats):
    """Return a copy of the table with the number formats applied (for display only).
    A format is a format string, or a function for a special format (for example a dollar value)."""
    out = table.copy()
    for col, f in formats.items():
        out[col] = out[col].map(f if callable(f) else f.format)
    return out

How to read this notebook.

• The notebook hides the code. Click Show code to see the code of a step.
• A blue block is a decision. A green block is a business note. A grey block (Technical note) gives detail for data scientists. A business reader can skip it.
• The appendix holds the decision log, the assumptions, the glossary and the data dictionary.

Step 0. What and why

This step gives the business problem, the question for leadership and the method that answers it with historical data. Each later step answers one part of the question.

Purpose. This step answers: what must a new score do for the business, and how can we test it before a real launch? It supports this decision: approve a challenger cutoff for a champion/challenger test.

0a. The business problem

Situation. A lender gives unsecured personal loans with a term of 36 months. A risk grade puts each applicant in a sub-grade, from A1 (the lowest risk) to G5 (the highest risk). A lender approves the applicants with the best grades and declines the others. The public data does not give this rule. Thus we set the champion ourselves (Step 0b).

Complication. The business wants more approvals, but Risk does not accept a higher bad rate. Also, the 2015 loans lost money: the default rate went up and the price went down (Step 2e). A new score must also give reason codes for each decline and pass a model risk review.

Question. Can a new score approve more applicants at the bad rate of the champion, and does it add money?

Resolution. Build a challenger PD (probability of default) model with only the data from the application and the credit bureau. Train it on the 2013–2014 loans and test it on the 2015 loans. Then compare the challenger with the champion on the same 2015 applicants.

If the challenger wins on these loans, test it on 10% of real applications before a full rollout. The cutoff alone does not fix the loss of the 2015 loans. A pricing test must go with it (Part 2: Pricing).

Decision that this notebook supports. Approve the challenger cutoff for a champion/challenger test on 10% of applications.

Success metric. The approval rate at the same bad rate as the champion, and the realized NPV of the approved loans.

0b. Method and data strategy

A technical reader asks one question first. The champion declines some applicants, so where do the outcomes of these applicants come from?

The data has only the loans that Lending Club booked. It does not contain the applicants that Lending Club declined. Thus we cannot see the outcome of a real declined applicant. We solve this problem in three moves:

1. Define a champion on booked loans. The champion approves the best sub-grades, up to about 85% of the 2015 applicants (assumption as-04). It declines the lowest sub-grades, about 15% of the applicants.
2. Use the declined group as a test group. Lending Club booked the loans in the lowest sub-grades. Thus we know if each of these loans defaulted, and how much money it gave back.
3. Compare the two policies on actual outcomes. The challenger approves the same number of applicants, but it puts them in a different order. Some champion declines become approvals (swap-in). Some champion approvals become declines (swap-out). We compare the actual default rate and the actual NPV of the two groups.

This method is a pseudo reject inference. Booked loans take the place of the declined applicants, so the method needs no model of the declined applicants. Each number in the comparison is an actual outcome, and a reviewer can check it.

The champion is our assumption, not the real policy of a lender. A reader must know this before any result. Each gain in this notebook is a gain against this champion.

Question	Answer
What does a real lender have?	A production approval policy. In a real project, that policy is the champion, and this section is not necessary.
What does the data not give?	Lending Club published only the loans that it booked. It did not publish its approval rules or the applicants that it declined.
What do we build?	Rank the 2015 applicants by sub-grade (A1 first). Approve the best sub-grades up to about 85% of the applicants (assumption as-04). Step 3c shows what this champion approves in 2015 and its bad rate.
Why this choice?	The sub-grade is the real risk grade of a real lender, made on the day of the application. Thus the comparison is our score against an industry score. Only the cutoff of 85% is our choice. A fixed rule needs no training data, so Step 3 applies it only to the 2015 test loans.
Which assumptions go with it?	as-04: the cutoff. as-05: the risk of the real declined applicants is not known. as-01 to as-03: the costs for the dollar values.
How can it be wrong?	A real policy can approve more or fewer applicants. The real declined applicants are riskier than the lowest sub-grades in our data, so the swap-in result can be too good. A real policy can also use more data than the sub-grade, so it can be stronger than our champion.
How do we test it?	Step 6d repeats the result for champions that approve 80%, 85% and 90% of applicants. The challenger must win against each of them. Step 7c proposes a small test below the cutoff to measure the real declined applicants.
What does not depend on it?	The order of risk (branch 1): the scorecard has the same KS as the sub-grade, and we report this result as it is. The swap-set uses actual outcomes of booked loans, not estimates.
What does it mean for the claim?	The notebook shows a method and the size of a probable gain. It does not promise a gain against the policy of a specific lender. The champion/challenger test of Step 7 measures the gain against the real policy.

Technical note. The comparison has these parts. Each later step gives the code.

• Champion rule. The code gives each sub-grade a rank: A1 = 0 to G5 = 34. The cutoff is the rank of the 2015 applicant at the 85th percentile. The champion approves each applicant with a rank below the cutoff. A full sub-grade is in or out, so the real approval rate is a little below 85%.
• Champion bad rate. The default rate of the 2015 loans that the champion approves. This rate is the risk limit for the challenger.
• Challenger score. A scorecard (a logistic regression with WoE) gives the PD of each applicant from the application and the credit bureau data. A GBM is the benchmark (Step 4c). The grade, the sub-grade and the interest rate are not features, because they are outputs of the Lending Club policy (Step 2b). Step 5 moves the level of the PD to the recent default rate.
• Approval at the same bad rate. The code sorts the 2015 applicants from the lowest PD. For each count of approvals, it calculates the bad rate of the approved loans. The largest count with a bad rate at or below the champion bad rate gives the approval rate. One scan of the sorted list gives the answer, so no search algorithm is necessary.
• Swap-set at the same approval count. The challenger approves the same number of applicants as the champion. Thus the swap-in group and the swap-out group have the same size. The code compares their default rate and their realized NPV (Step 2e).
• Selection bias. All loans passed the credit policy of Lending Club. Thus the result is valid only above that policy. A random test below the cutoff removes this limit.

0c. How the notebook answers the question

Each step answers one part of the question:

• Step 1. Which data gives the actual outcome of each applicant, and what does it contain?
• Step 2. Which loans, which target and which columns are fair to use? What did the current policy give?
• Step 3. How do we test the score on a period that the model did not see? What does the champion approve on that period?
• Step 4. Which columns and which model type put the applicants in the best order of risk?
• Step 5. Is the PD at the correct level, and can the lender give a reason for each decline?
• Step 6. Is the challenger better than the champion, in model metrics and in dollars?
• Step 7. Where do we put the cutoff, and how do we test it with real applicants?
• Part 2. What is the lifetime loss of each vintage? Does the price pay for the risk? Which charged-off loans give the most money back?

Business note. The question is a trade: more approvals at the same risk, with more money. The method compares the two policies only on actual outcomes, so each result is a fact of the past, not an estimate. Step 1 shows the data that gives these outcomes.

Step 1. Get data

Step 0 set the question and the method: compare two policies on the actual outcomes of booked loans. This step answers: which data gives these outcomes, and what does it contain?

Purpose. This step gets the data and its meaning. It supports this decision: if the data can answer the business question.

1a. Why this data

The data is the file of the loans that Lending Club accepted from 2007 to 2018. Source: Kaggle (wordsforthewise/lending-club), file accepted_2007_to_2018Q4.csv.gz. The data is a fair proxy for unsecured consumer credit, for four reasons:

• The same product. Lending Club gave unsecured personal loans with a fixed term and a fixed rate. Many borrowers used the loan to pay their credit card debt.
• The same data at the decision. Each loan has the application data and the credit bureau data from the day of the application. A bank uses the same data for its decision.
• Actual outcomes and cash. Each loan has its final status and the total of its payments. Thus we can calculate the default and the NPV of each loan.
• A public source. Each reader can get the same file and repeat each result.

The model uses the 36-month loans from 2013 to 2015. The data ends in March 2019, so each of these loans has a full outcome window of 36 months. Part 2 also uses the younger loans of 2016–2018.

The data cannot show three things:

What the data does not show	Why it matters	What the notebook does
Declined applicants	We do not know the risk of the applicants that Lending Club declined.	The pseudo reject inference of Step 0, and a test below the cutoff (Step 7).
Credit card behavior	A card has a credit line and a balance that changes each month. A personal loan has a fixed payment.	The method applies to cards, but with a different target and more data.
Macroeconomic variables	The data has no unemployment rate and no interest rate of the market. The loans come from 2013–2018, a period without a US recession.	The results do not show a downturn. The rollout plan uses guardrails and kill criteria.

1b. Key variables

The table gives the columns that the story uses most. The target is default. Step 2 makes it from loan_status. Appendix A gives the meaning and the use of all columns of the file.

Column	Meaning in plain words	Type	Role
loan_status → default	The final status of the loan. default = 1 for “Charged Off”, “Default” or “Late (31-120 days)”. default = 0 for “Fully Paid”.	Text → 1/0	Target
grade, sub_grade	The risk grade that Lending Club gave: A to G, and A1 to G5.	Category with an order	Policy: the champion rule. Not a feature.
int_rate	The interest rate that Lending Club gave to the loan.	Number (%)	Policy and price. NPV and Part 2: Pricing. Not a feature.
fico	The credit score of the applicant. Step 2 takes the middle of fico_range_low and fico_range_high.	Number	Candidate feature
dti	Monthly debt payments, without the mortgage and the new loan, divided by the monthly income.	Number (%)	Candidate feature
annual_inc	The annual income that the applicant gave.	Number (\() | Candidate feature | | `revol_util` | The revolving balance divided by the revolving credit limit. | Number (%) | Candidate feature | | `loan_amnt` | The amount that the applicant asked for. | Number (\))	Candidate feature
emp_length	The number of years in the current job, from “< 1 year” to “10+ years”. Step 2 makes it a number.	Text → number	Candidate feature
home_ownership	Rent, own, mortgage or other.	Category	Candidate feature
purpose	The reason for the loan, for example debt consolidation or credit card.	Category	Candidate feature
total_pymnt, recoveries, last_pymnt_d	The total that the borrower paid, the money that came back after a charge-off, and the month of the last payment.	Number ($), date	Outcome: the NPV of each loan. Never a feature.

A candidate feature goes to the column selection of Step 4. A column with the role “Policy” or “Outcome” is never a feature. The lender does not know an outcome on the day of the application.

1c. Load the data and look at it

The code reads only the 36-month loans. It uses SQL (DuckDB) on the CSV file. The first run reads the full file one time and writes a small Parquet copy. Later runs read the copy in a few seconds.

The code reads two sets of columns:

• The main columns for all 36-month loans of 2007–2018: the application data, the price, the status and the payments.
• More credit bureau columns, only for the loans of 2013–2015. Only the column selection of Step 4 uses them.

# Purpose: Read the Lending Club file with SQL (DuckDB). Keep a Parquet copy for fast reloads.
# Input:   data/raw/accepted_2007_to_2018Q4.csv.gz, data/raw/LCDataDictionary.xlsx.
# Output:  `loans` (all 36-month loans), `extra` (more bureau columns, 2013–2015), `lc_dict` (definitions).

# 1. Columns for all 36-month loans. DuckDB reads them as text, then makes numbers.
COLS_NUMERIC = [
    "loan_amnt", "funded_amnt", "int_rate", "installment", "annual_inc", "dti",
    "delinq_2yrs", "fico_range_low", "fico_range_high", "inq_last_6mths",
    "mths_since_last_delinq", "open_acc", "pub_rec", "revol_bal", "revol_util",
    "total_acc", "mort_acc", "pub_rec_bankruptcies", "acc_open_past_24mths",
    "bc_util", "num_tl_op_past_12m", "mo_sin_old_rev_tl_op", "tot_cur_bal",
    "total_rev_hi_lim", "total_pymnt", "total_rec_prncp", "total_rec_int",
    "total_rec_late_fee", "recoveries", "collection_recovery_fee",
]
COLS_TEXT = [
    "id", "term", "grade", "sub_grade", "emp_length", "home_ownership",
    "verification_status", "issue_d", "loan_status", "purpose", "addr_state",
    "earliest_cr_line", "last_pymnt_d", "application_type",
]

# First run only: read the 393 MB CSV once and write a small Parquet file.
if not INTERIM_FILE.exists():
    INTERIM_FILE.parent.mkdir(parents=True, exist_ok=True)
    select_cols = ",\n  ".join(
        [f"TRY_CAST({c} AS DOUBLE) AS {c}" for c in COLS_NUMERIC]
        + COLS_TEXT
        + ["strptime(issue_d, '%b-%Y')::DATE AS issue_date"]
    )
    duckdb.sql(f"""
        COPY (
          SELECT {select_cols}
          FROM read_csv('{RAW_FILE.as_posix()}', header = true, all_varchar = true)
          WHERE trim(term) = '36 months'
        ) TO '{INTERIM_FILE.as_posix()}' (FORMAT parquet)
    """)

# Later runs: read the Parquet file (a few seconds).
loans = duckdb.sql(f"SELECT * FROM read_parquet('{INTERIM_FILE.as_posix()}')").df()
loans["issue_date"] = pd.to_datetime(loans["issue_date"])
# Credit bureau columns that only the selection funnel (dec-10) needs. Loans of 2013–2015 only.
EXTRA_NUMERIC = [
    "mths_since_last_record", "collections_12_mths_ex_med", "mths_since_last_major_derog", "acc_now_delinq",
    "tot_coll_amt", "open_acc_6m", "open_act_il", "open_il_12m", "open_il_24m", "mths_since_rcnt_il",
    "total_bal_il", "il_util", "open_rv_12m", "open_rv_24m", "max_bal_bc", "all_util", "inq_fi", "total_cu_tl",
    "inq_last_12m", "avg_cur_bal", "bc_open_to_buy", "chargeoff_within_12_mths", "delinq_amnt",
    "mo_sin_old_il_acct", "mo_sin_rcnt_rev_tl_op", "mo_sin_rcnt_tl", "mths_since_recent_bc",
    "mths_since_recent_bc_dlq", "mths_since_recent_inq", "mths_since_recent_revol_delinq",
    "num_accts_ever_120_pd", "num_actv_bc_tl", "num_actv_rev_tl", "num_bc_sats", "num_bc_tl", "num_il_tl",
    "num_op_rev_tl", "num_rev_accts", "num_rev_tl_bal_gt_0", "num_sats", "num_tl_120dpd_2m", "num_tl_30dpd",
    "num_tl_90g_dpd_24m", "pct_tl_nvr_dlq", "percent_bc_gt_75", "tax_liens", "tot_hi_cred_lim",
    "total_bal_ex_mort", "total_bc_limit", "total_il_high_credit_limit",
]
# 2. Same method for the extra bureau columns, loans of 2013–2015 only.
if not CAND_FILE.exists():
    select_extra = ",\n  ".join(["id"] + [f"TRY_CAST({c} AS DOUBLE) AS {c}" for c in EXTRA_NUMERIC])
    duckdb.sql(f"""
        COPY (
          SELECT {select_extra}
          FROM read_csv('{RAW_FILE.as_posix()}', header = true, all_varchar = true)
          WHERE trim(term) = '36 months'
            AND strptime(issue_d, '%b-%Y') BETWEEN DATE '2013-01-01' AND DATE '2015-12-01'
        ) TO '{CAND_FILE.as_posix()}' (FORMAT parquet)
    """)
extra = duckdb.sql(f"SELECT * FROM read_parquet('{CAND_FILE.as_posix()}')").df()

# 3. Lending Club data dictionary: one definition for each column.
lc_dict = pd.read_excel(DICT_FILE, sheet_name="LoanStats").dropna(subset=["LoanStatNew"])
lc_dict = (lc_dict.assign(column=lc_dict["LoanStatNew"].str.strip()
                          .replace({"verified_status_joint": "verification_status_joint"}))  # Spelling in the file
           .set_index("column")["Description"].str.strip())
print(f"36-month loans: {len(loans):,}. Extra bureau columns: {len(EXTRA_NUMERIC)} for {len(extra):,} loans "
      f"of 2013–2015. Columns in the data dictionary: {len(lc_dict)}.")

36-month loans: 1,609,754. Extra bureau columns: 50 for 546,165 loans of 2013–2015. Columns in the data dictionary: 151.

The next cell shows the initial state of the data. It gives the size, the dates, the final status of the loans and a sample of ten key columns.

# Purpose: Show the initial state of the raw data before any change: size, dates, status and a sample.
# Input:   `loans`, `extra`, `lc_dict`.
# Output:  Numbers for the text (`N`), table of the final status, sample of ten key columns.

# Size and date range. `last_pymnt_d` shows the end of the data.
last_pay = pd.to_datetime(loans["last_pymnt_d"], format="%b-%Y", errors="coerce")
model_years = loans["issue_date"].dt.year.between(2013, 2015)
# Share of the 2013–2015 loans that pay other debt (a check of the proxy for consumer credit).
debt_share = loans.loc[model_years, "purpose"].isin(["debt_consolidation", "credit_card"]).mean()
N.update({
    "s1_rows": f"{len(loans):,}", "s1_cols": f"{loans.shape[1]}",
    "s1_issue_first": f"{loans['issue_date'].min():%B %Y}", "s1_issue_last": f"{loans['issue_date'].max():%B %Y}",
    "s1_last_pay": f"{last_pay.max():%B %Y}", "s1_rows_13_15": f"{model_years.sum():,}",
    "s1_extra_cols": f"{len(EXTRA_NUMERIC)}", "s1_extra_rows": f"{len(extra):,}",
    "s1_dict_cols": f"{len(lc_dict)}", "s1_debt_share": f"{debt_share:.0%}",
})
print(f"Rows: {len(loans):,}. Columns: {loans.shape[1]}. "
      f"Issue dates: {N['s1_issue_first']} to {N['s1_issue_last']}. Last payment: {N['s1_last_pay']}.")

# Final status of all 36-month loans. Step 2 makes the target from this column.
status = loans["loan_status"].value_counts().rename("Loans").to_frame()
status["Share"] = status["Loans"] / len(loans)
print(fmt(status, {"Loans": "{:,.0f}", "Share": "{:.1%}"}).to_string())

# A random sample of ten key columns (the target, the policy, the application and the outcome).
KEY_COLS = ["issue_d", "loan_status", "grade", "sub_grade", "int_rate", "loan_amnt",
            "fico_range_low", "dti", "annual_inc", "purpose"]
loans.sample(8, random_state=SEED).sort_values("issue_date")[KEY_COLS]

Rows: 1,609,754. Columns: 45. Issue dates: June 2007 to December 2018. Last payment: March 2019.
                                                       Loans  Share
loan_status                                                        
Fully Paid                                           857,491  53.3%
Current                                              566,695  35.2%
Charged Off                                          163,252  10.1%
Late (31-120 days)                                    12,474   0.8%
In Grace Period                                        4,834   0.3%
Late (16-30 days)                                      2,545   0.2%
Does not meet the credit policy. Status:Fully Paid     1,789   0.1%
Does not meet the credit policy. Status:Charged Off      649   0.0%
Default                                                   25   0.0%

	issue_d	loan_status	grade	sub_grade	int_rate	loan_amnt	fico_range_low	dti	annual_inc	purpose
946421	Jan-2014	Fully Paid	C	C1	13.98	15275.0	660.0	11.05	58000.00	credit_card
920455	Mar-2014	Fully Paid	A	A2	6.62	17700.0	720.0	12.12	52000.00	debt_consolidation
864928	Jul-2014	Fully Paid	C	C3	13.98	14375.0	660.0	24.89	38000.00	credit_card
261603	Jan-2015	Fully Paid	B	B2	9.49	4675.0	665.0	19.88	40507.92	debt_consolidation
93641	Sep-2015	Fully Paid	C	C1	12.29	30000.0	660.0	22.14	175000.00	debt_consolidation
1184634	Mar-2017	Current	A	A3	7.24	35000.0	715.0	16.35	125000.00	credit_card
1484690	Nov-2017	Fully Paid	A	A3	6.72	10000.0	680.0	8.52	145000.00	credit_card
1458979	Dec-2017	Current	B	B1	9.44	10000.0	690.0	28.11	83000.00	other

The data has 1,609,754 loans with a term of 36 months and 45 columns. Lending Club issued them from June 2007 to December 2018. The last payment in the data is in March 2019. The model years 2013–2015 have 546,165 loans. For these loans, the code also reads 50 more credit bureau columns.

The data dictionary of Lending Club defines 151 columns. Of the 2013–2015 loans, 82% pay other debt (debt consolidation or credit card). This share supports the use of the data as a proxy for consumer credit.

The status table shows the source of the target. Many recent loans are still “Current”, so they have no final outcome yet. Step 2 removes them from the scope of the model. The status “Does not meet the credit policy” marks loans outside the normal policy of Lending Club. Step 2 also removes them.

Business note. The data gives the actual outcome and the cash of each booked loan, with the data from the day of the application. It cannot show declined applicants, card behavior or a recession, and the notebook says where each limit matters. Step 2 selects the loans, the target and the columns that the model can use fairly.

Step 2. Clean and prepare data

Step 1 showed that the Lending Club file can answer the business question. It has the loans, the application data and the outcome of each loan. But the raw file also has rows that the model must not learn from. It also has columns that the lender cannot use on the day of the decision. This step answers: which rows, which columns and which values can the model use?

Purpose. This step makes sure that the data can answer the business question without bias or leakage. It supports this decision: the scope of the model, the target and the columns that the model can use.

The data preparation follows one tree with three levels. Each level has one split axis. Each axis divides all cases into groups that do not overlap and that together cover all cases (MECE).

Level	Split axis	Groups	Section
Rows	Is the record valid? If valid, is the loan in the business scope?	Invalid · valid but out of scope · in scope	2a
Columns	May the lender use the value on the day of the decision? If yes, does the column hold usable data?	Not permitted · no usable data · candidate	2b
Values	Why is a cell empty or extreme?	Structural · operational · random missing values. Impossible · extreme but real values.	2c

Then Step 2d makes new features from the clean data, and Step 2e measures the baseline that the challenger must beat.

2a. Rows: which loans can the model learn from?

The first split asks if a record is technically valid. A record is not valid if it repeats a loan, if a key field is empty, or if an amount or a date is impossible. The code checks these four conditions. The code removes these rows before any other step.

# Purpose: Remove the rows that are not valid and the policy exceptions. Add `issue_year` and `default` (dec-01).
# Input:   `loans`.
# Output:  `loans` with `issue_year` and `default` (1 = default, 0 = no default). `row_checks` (table).

BAD_STATUS = ["Charged Off", "Default", "Late (31-120 days)"]
CLOSED_STATUS = ["Charged Off", "Default", "Fully Paid"]
DATA_END = pd.Timestamp("2019-03-01")  # Last month with payments in the file
n_loaded = len(loans)
N["n_loaded"] = f"{n_loaded:,}"

# Level 1: technical checks. Each check gives True for a row that is not valid.
row_checks = {
    "Duplicate loan id": loans["id"].duplicated(keep="first"),
    "Empty key field (id, issue date, amount, status)": loans[
        ["id", "issue_date", "loan_amnt", "loan_status"]].isna().any(axis=1),
    "Impossible amount (loan, funded amount or installment of $0 or less)": (
        loans[["loan_amnt", "funded_amnt", "installment"]] <= 0).any(axis=1),
    "Issue date after the end of the data": loans["issue_date"] > DATA_END,
}
invalid = pd.concat(row_checks, axis=1).any(axis=1)
# Level 2, first group: loans that Lending Club approved outside its credit policy.
exceptions = loans["loan_status"].str.startswith("Does not meet")
N.update({"n_invalid": f"{invalid.sum():,}", "n_exceptions": f"{exceptions.sum():,}"})
loans = loans[~invalid & ~exceptions].copy()
loans["issue_year"] = loans["issue_date"].dt.year
# Target: 1 if the final status is a default (see the glossary).
loans["default"] = loans["loan_status"].isin(BAD_STATUS).astype(int)
pd.Series({k: v.sum() for k, v in row_checks.items()}, name="Rows that fail the check").to_frame()

	Rows that fail the check
Duplicate loan id	0
Empty key field (id, issue date, amount, status)	0
Impossible amount (loan, funded amount or installment of $0 or less)	0
Issue date after the end of the data	0

All 1,609,754 rows pass the four technical checks: 0 rows fail. The file from Lending Club is clean at the row level. The real work is the second split: which valid loans are in the business scope?

A valid loan can be out of scope for five reasons. Each reason is a different condition of the business question:

• Policy. Lending Club approved 2,438 loans outside its credit policy (“Does not meet the credit policy”). The new policy will not make these exceptions, so the model must not learn from them.
• Product. The question is about 36-month loans. Step 1 reads only the 36-month loans. A 60-month loan has a different risk, and a 60-month loan from 2015 has not ended at the end of the data.
• Time. The model uses the loans issued from 2013 to 2015. For the 2012 loans, some credit bureau columns are mostly empty, because Lending Club added these columns during 2012. The decision below gives the numbers. The loans after 2015 have no full outcome window (see the decision below).
• Applicant. Joint applications use a second applicant and other income and DTI columns. Lending Club started them in late 2015.
• Outcome. A loan without a final status (“Current”, “In Grace Period”, “Late (16-30 days)”) has no outcome yet. The model cannot learn from it.

The outcome window. A 36-month loan needs 36 months to show its outcome. The last loan in scope has its issue date in December 2015. Its term ends in December 2018, and the data ends in March 2019. Thus each loan in scope had its full term to pay or to default.

Which loans give a default rate that we can trust?

Why this question matters. The PD sets the cutoff. If the default rate in the training data is too high, the cutoff declines good applicants. If it is too low, the lender approves too much risk.

Ways to do it. The problem is the loans that are not closed yet. There are only three things to do with a loan without an outcome: ignore it, wait for it, or model it.

• Ignore it: use all closed loans of all years. This gives the most rows. The risk: a default closes a loan earlier than a full payment. Thus the closed loans of a young vintage contain too many defaults.
• Wait for it: use only the vintages with a full outcome window. Each loan had 36 months to show its outcome, so the target has no bias. The cost: fewer rows, and the newest vintages are not in the model.
• Model it: keep the open loans in a survival model. This uses all rows. But the cutoff needs one PD for a fixed period, and a survival model is harder to explain in a model risk review.

Evidence. The table gives, for each issue year, the share of loans with a final status and the default rate of the closed loans. The first method uses this default rate. If the method has no bias, the rate must stay near the same level across the years.

# Purpose: Show why a full outcome window is necessary (dec-01).
# Input:   `loans`.
# Output:  Table `dec01`: loans, share closed and default rate of the closed loans by issue year.

indiv = loans[loans["application_type"] == "Individual"]
by_year = indiv.groupby("issue_year")
# The first method uses only the closed loans. Young vintages have few closed loans.
closed = indiv[indiv["loan_status"].isin(CLOSED_STATUS)]
dec01 = pd.DataFrame({
    "Loans": by_year.size(),
    "Share closed": by_year["loan_status"].apply(lambda s: s.isin(CLOSED_STATUS).mean()),
    "Default rate of closed loans": closed.groupby("issue_year")["default"].mean(),
}).loc[2012:2018]
# 2012: share of empty values in the credit bureau columns that Lending Club added during 2012.
miss_2012 = indiv.loc[indiv["issue_year"] == 2012, COLS_NUMERIC].isna().mean()
N.update({
    "miss_2012": f"{miss_2012.max():.0%}", "n_miss_2012": f"{(miss_2012 > 0.3).sum()}",
    "closed_16": f"{dec01.loc[2016, 'Share closed']:.0%}", "closed_17": f"{dec01.loc[2017, 'Share closed']:.0%}",
    "dr_closed_lo": f"{dec01.loc[2013:2015, 'Default rate of closed loans'].min():.0%}",
    "dr_closed_hi": f"{dec01.loc[2013:2015, 'Default rate of closed loans'].max():.0%}",
    "dr_closed_17": f"{dec01.loc[2017, 'Default rate of closed loans']:.0%}",
})
del indiv, closed
fmt(dec01, {"Loans": "{:,.0f}", "Share closed": "{:.0%}", "Default rate of closed loans": "{:.1%}"})

	Loans	Share closed	Default rate of closed loans
issue_year
2012	43,470	100%	13.6%
2013	100,422	100%	12.3%
2014	162,570	100%	13.7%
2015	282,934	100%	14.9%
2016	318,277	72%	19.8%
2017	296,484	41%	19.8%
2018	304,292	12%	12.6%

Decision. Wait for the outcome: the PD model uses the 36-month loans from 2013 to 2015, each with a full outcome window. The loss forecast in Part 2 models the open loans with a survival method (Kaplan-Meier).

Why. For 2013–2015, almost all loans have a final status, and the default rate of the closed loans stays between 12% and 15%. For 2016 and 2017, only 72% and 41% of the loans have a final status. The default rate of the closed 2017 loans jumps to 20%, because the defaults close first. A PD from these loans is too high, and the cutoff then declines too many good applicants.

The survival model is the right tool for the open loans. But the cutoff needs one PD for each applicant, so we use the survival model only in Part 2.

What would change the decision. If the business wants a cutoff for 60-month loans, it needs a separate model or the survival method.

The code applies the scope conditions in a fixed order. Each loan gets the first condition that removes it. Thus the groups do not overlap, and the table adds up to all loans.

# Purpose: Keep the loans in the scope of the model (dec-01) and add the extra bureau columns.
# Input:   `loans`, `extra`.
# Output:  `df`: one row for each loan in scope. Table `row_tree`: the rows that each condition removes.

# Scope conditions after the technical checks and the policy exceptions (cell `clean`).
year_ok = loans["issue_year"].between(2013, 2015)
indiv_ok = loans["application_type"] == "Individual"
final_ok = loans["loan_status"].isin(BAD_STATUS + CLOSED_STATUS)
in_scope = year_ok & indiv_ok & final_ok

# Row tree: each row gets the first condition that removes it.
row_tree = pd.DataFrame([
    ("Not valid", "Fails a technical check", invalid.sum()),
    ("Valid, out of scope", "Policy exception", exceptions.sum()),
    ("Valid, out of scope", "Issue year not in 2013–2015", (~year_ok).sum()),
    ("Valid, out of scope", "Joint application", (year_ok & ~indiv_ok).sum()),
    ("Valid, out of scope", "No final status", (year_ok & indiv_ok & ~final_ok).sum()),
    ("In scope", "Loans for the model", in_scope.sum()),
], columns=["Group", "Reason", "Loans"]).set_index(["Group", "Reason"])
assert row_tree["Loans"].sum() == n_loaded, "The row tree must add up to all loaded loans."
row_tree["Share of loaded loans"] = row_tree["Loans"] / n_loaded

# Add the extra bureau columns. Each loan must match one row (one_to_one).
df = loans[in_scope].merge(extra, on="id", how="left", validate="one_to_one")
df.index = loans.index[in_scope]  # Keep the row labels of loans
# Check: almost all loans must find their extra columns.
assert df["num_sats"].notna().mean() > 0.99, "The extra bureau columns did not match the loans."
del extra  # Free memory
N.update({"bad_scope": f"{df['default'].mean():.1%}", "n_scope": f"{len(df):,}",
          "n_joint": f"{row_tree.loc[('Valid, out of scope', 'Joint application'), 'Loans']:,}",
          "n_open": f"{row_tree.loc[('Valid, out of scope', 'No final status'), 'Loans']:,}"})
fmt(row_tree, {"Loans": "{:,.0f}", "Share of loaded loans": "{:.1%}"})

		Loans	Share of loaded loans
Group	Reason
Not valid	Fails a technical check	0	0.0%
Valid, out of scope	Policy exception	2,438	0.2%
	Issue year not in 2013–2015	1,061,151	65.9%
	Joint application	239	0.0%
	No final status	86	0.0%
In scope	Loans for the model	545,840	33.9%

The model learns from 545,840 loans with a default rate of 14.1%. The time condition removes the most rows. The joint applications (239) and the loans without a final status (86) are small groups.

2b. Columns: which fields may enter a model?

A column can be a feature only if two things are true. The lender has its value on the day of the decision, and the lender may use it. The tree below uses these two questions as the split axes. Each column gets the first rule that removes it, so the buckets do not overlap.

Bucket 1: not permitted. The column has data, but the lender must not use it for the decision.

• Rule 3. Known only after the decision. Payments, recoveries, later credit bureau data, hardship and settlement data. These columns contain the outcome. A model with them shows a perfect test result and fails in production. This is leakage.
• Rule 4. The decision of Lending Club. The grade, the sub-grade, the interest rate, the installment and the funded amount. Lending Club set them from its own score. The challenger must replace this decision, not copy it (see Step 4).
• Rule 6. Fair lending risk. The state and the zip code can act as a proxy for a protected class. The job title is free text with the same risk.
• Rule 5. Out of scope. Columns for the second applicant of a joint application.

Bucket 2: no usable data about the borrower. The lender may use the column, but it does not describe the risk of the borrower.

• Rule 1. Not about the borrower. Identifiers, links, free text and process fields of Lending Club.
• Rule 2. Constant in scope. The columns that select the loans (term, issue date, application type) have one value for all loans in scope.
• Rule 8. Not collected in the training years. Lending Club added some columns in late 2015. More than 95% of their values are empty in 2013 or 2014, so the model cannot learn from them.

Bucket 3: no mathematical value. Rule 7 removes the raw columns that a better calculated column replaces (Step 2d). Step 4 adds three more rules of this bucket: weak, duplicate and unstable columns.

# Purpose: Apply rules 1–7 of the selection funnel (dec-10). These rules use only the column name.
# Input:   `lc_dict` (all columns of the file), `df`.
# Output:  `funnel` (each column with its rule), `RULE_BUCKET`, `candidates`, `CALCULATED`, `ENGINEERED`, `CAT_ALL`.

# Rules 1–7: a list of column names for each rule. Rules 3 and 5 also use prefixes.
NAME_RULES = {
    "1. Not about the borrower": ["id", "member_id", "url", "desc", "title", "policy_code", "initial_list_status",
                                  "funded_amnt_inv", "disbursement_method"],
    "2. Scope column": ["term", "issue_d", "application_type"],
    "3. Known only after the issue date": [
        "loan_status", "pymnt_plan", "out_prncp", "out_prncp_inv", "total_pymnt", "total_pymnt_inv",
        "total_rec_prncp", "total_rec_int", "total_rec_late_fee", "recoveries", "collection_recovery_fee",
        "last_pymnt_d", "last_pymnt_amnt", "next_pymnt_d", "last_credit_pull_d", "last_fico_range_high",
        "last_fico_range_low", "deferral_term", "payment_plan_start_date",
        "orig_projected_additional_accrued_interest", "debt_settlement_flag", "debt_settlement_flag_date"],
    "4. Output of the Lending Club policy": ["grade", "sub_grade", "int_rate", "installment", "funded_amnt"],
    "5. Joint applications only": [],
    "6. Fair lending risk": ["addr_state", "zip_code", "emp_title"],
    "7. Replaced by a calculated column": ["fico_range_low", "fico_range_high", "emp_length", "earliest_cr_line"],
}
# The three buckets of the column tree. Rules 9–11 come in Step 4.
RULE_BUCKET = {
    "3. Known only after the issue date": "1. Not permitted",
    "4. Output of the Lending Club policy": "1. Not permitted",
    "6. Fair lending risk": "1. Not permitted",
    "5. Joint applications only": "1. Not permitted",
    "1. Not about the borrower": "2. No usable data",
    "2. Scope column": "2. No usable data",
    "8. Not collected in the training years": "2. No usable data",
    "7. Replaced by a calculated column": "3. No mathematical value",
    "9. Weak (IV < 0.01)": "3. No mathematical value",
    "10. Duplicate of a stronger column": "3. No mathematical value",
    "11. Unstable (PSI > 0.1)": "3. No mathematical value",
}
# Calculated features (Step 2d) and the raw columns that each one uses.
CALCULATED = {
    "fico": "fico_range_low, fico_range_high", "emp_years": "emp_length",
    "credit_hist_months": "earliest_cr_line, issue_d",
    "loan_to_income": "loan_amnt, annual_inc", "loan_to_rev_limit": "loan_amnt, total_rev_hi_lim",
    "new_acc_share": "acc_open_past_24mths, total_acc", "bal_to_limit": "tot_cur_bal, tot_hi_cred_lim",
}
ENGINEERED = ["loan_to_income", "loan_to_rev_limit", "new_acc_share", "bal_to_limit"]  # New ratios (Step 2d)
CAT_ALL = ["home_ownership", "verification_status", "purpose"]  # Categorical candidates


def name_rule(col):
    """The first rule (1–7) that removes the column, or None."""
    if col.startswith(("hardship_", "settlement_")):
        return "3. Known only after the issue date"
    if col.startswith("sec_app_") or col.endswith("_joint"):
        return "5. Joint applications only"
    return next((rule for rule, cols in NAME_RULES.items() if col in cols), None)


# One row for each column of the data dictionary, with its first rule.
funnel = pd.DataFrame({"column": lc_dict.index})
funnel["rule"] = funnel["column"].map(name_rule)
# The columns that no rule removes, plus the calculated features.
candidates = list(funnel.loc[funnel["rule"].isna(), "column"]) + list(CALCULATED)
missing_cols = [c for c in candidates if c not in df.columns and c not in CALCULATED]
# Check: the code must read each candidate column.
assert not missing_cols, f"Candidate columns not loaded: {missing_cols}"
N["n_dict"] = f"{len(funnel)}"
N["n_after_names"] = f"{len(candidates)}"

Rule 8 needs the data, not only the name. It removes a column if more than 95% of its values are empty in 2013 or in 2014. The rule uses only the training years and does not use the target.

# Purpose: Apply rule 8: remove the columns that are empty in the training years.
# Input:   `candidates`, `df` (2013–2014 only, no target).
# Output:  `CANDIDATES` (the columns that go to Step 4), `miss_by_year`, `not_collected`. Table by bucket.

# Threshold of rule 8.
MAX_MISSING = 0.95
num_candidates = [c for c in candidates if c not in CAT_ALL and c not in CALCULATED]
# Share of missing values for each column and each training year. No target.
miss_by_year = df.loc[df["issue_year"] <= 2014].groupby("issue_year")[num_candidates].agg(
    lambda s: s.isna().mean()).T
not_collected = list(miss_by_year.index[miss_by_year.max(axis=1) > MAX_MISSING])
funnel.loc[funnel["column"].isin(not_collected), "rule"] = "8. Not collected in the training years"
CANDIDATES = [c for c in candidates if c not in not_collected]
N.update({"n_not_collected": f"{len(not_collected)}", "n_candidates": f"{len(CANDIDATES)}",
          "n_engineered": f"{len(ENGINEERED)}", "n_calc": f"{len(CALCULATED)}",
          "n_raw_left": f"{len(CANDIDATES) - len(CALCULATED)}"})

# Table: the columns that each rule removes, grouped by bucket.
by_rule = funnel.dropna(subset=["rule"]).groupby("rule")["column"].agg(list)
col_tree = pd.DataFrame({
    "Bucket": [RULE_BUCKET[r] for r in by_rule.index],
    "Rule": by_rule.index,
    "Columns": by_rule.map(len).to_numpy(),
    "Examples": by_rule.map(lambda c: ", ".join(c[:4])).to_numpy(),
})
col_tree.loc[len(col_tree)] = ["Candidate for Step 4", f"Raw columns + {len(CALCULATED)} calculated",
                               len(CANDIDATES), ", ".join(CANDIDATES[:4])]
col_tree.sort_values(["Bucket", "Rule"]).set_index(["Bucket", "Rule"])

		Columns	Examples
Bucket	Rule
1. Not permitted	3. Known only after the issue date	39	collection_recovery_fee, last_credit_pull_d, l...
	4. Output of the Lending Club policy	5	funded_amnt, grade, installment, int_rate
	5. Joint applications only	16	annual_inc_joint, dti_joint, verification_stat...
	6. Fair lending risk	3	addr_state, emp_title, zip_code
2. No usable data	1. Not about the borrower	9	desc, funded_amnt_inv, id, initial_list_status
	2. Scope column	3	application_type, issue_d, term
	8. Not collected in the training years	14	all_util, il_util, inq_fi, inq_last_12m
3. No mathematical value	7. Replaced by a calculated column	4	earliest_cr_line, emp_length, fico_range_high,...
Candidate for Step 4	Raw columns + 7 calculated	65	acc_now_delinq, acc_open_past_24mths, annual_i...

Rules 1–7 keep 79 columns, with the 7 calculated columns of Step 2d. Rule 8 removes 14 more. Thus 65 columns go to the selection funnel in Step 4: 58 raw columns and 7 calculated columns.

2c. Values: missing values and outliers

The candidate columns can still have empty cells and extreme values. The split axis for an empty cell is the cause. Either the event did not occur (structural), or a process did not record a real value (operational). Or a rare error lost one value (random). For an extreme value, the axis is reality: the value is impossible, or it is extreme but real.

# Purpose: Classify the missing values of the candidates by cause, and count the impossible values.
# Input:   `df`, `not_collected`, `miss_by_year`.
# Output:  Table `missing_types`. Table `value_checks`. Numbers for the text.

# Missing values: examples of each cause, with the share for the loans in scope.
MISSING_TYPES = {
    "Structural: the event did not occur": ["mths_since_last_delinq", "mths_since_last_record",
                                            "mths_since_recent_revol_delinq", "mo_sin_old_il_acct", "bc_util"],
    "Operational: the process did not record the value": ["emp_length", "num_tl_120dpd_2m"],
    "Random: a few cells": ["revol_util", "pct_tl_nvr_dlq"],
}
missing_types = pd.DataFrame([
    {"Type": t, "Column": c, "Missing (2013–2015)": df[c].isna().mean(),
     "Missing 2013": df.loc[df["issue_year"] == 2013, c].isna().mean(),
     "Missing 2015": df.loc[df["issue_year"] == 2015, c].isna().mean()}
    for t, cols in MISSING_TYPES.items() for c in cols])
# Rule 8 columns are operational too. One summary row for them.
missing_types.loc[len(missing_types)] = [
    "Operational: the process did not record the value", f"{len(not_collected)} columns of rule 8",
    df[not_collected].isna().mean().mean(),
    df.loc[df["issue_year"] == 2013, not_collected].isna().mean().mean(),
    df.loc[df["issue_year"] == 2015, not_collected].isna().mean().mean()]
# Keep the order of the types: structural, operational, random.
missing_types["Type"] = pd.Categorical(missing_types["Type"], categories=list(MISSING_TYPES), ordered=True)
missing_types = missing_types.sort_values("Type", kind="stable")

# Impossible values: a check for each type of column. Each check counts the cells that fail.
value_checks = pd.Series({
    "Income of $0 or less": (df["annual_inc"] <= 0).sum(),
    "DTI below 0": (df["dti"] < 0).sum(),
    "FICO outside 300–850": (~df["fico_range_low"].between(300, 850)).sum(),
    "Negative balance or count": (df[["revol_bal", "tot_cur_bal", "open_acc", "total_acc"]] < 0).sum().sum(),
    "First credit line after the issue date": (
        pd.to_datetime(df["earliest_cr_line"], format="%b-%Y", errors="coerce") > df["issue_date"]).sum(),
}, name="Cells that fail the check")
N.update({
    "miss_delinq": f"{df['mths_since_last_delinq'].isna().mean():.0%}",
    "miss_emp_13": f"{df.loc[df['issue_year'] == 2013, 'emp_length'].isna().mean():.1%}",
    "miss_emp_15": f"{df.loc[df['issue_year'] == 2015, 'emp_length'].isna().mean():.1%}",
    "miss_120_13": f"{df.loc[df['issue_year'] == 2013, 'num_tl_120dpd_2m'].isna().mean():.1%}",
    "miss_120_15": f"{df.loc[df['issue_year'] == 2015, 'num_tl_120dpd_2m'].isna().mean():.1%}",
    "miss_revol": f"{df['revol_util'].isna().mean():.2%}",
    "n_impossible": f"{int(value_checks.sum()):,}",
    "inc_max": f"${df['annual_inc'].max() / 1e6:,.1f} million", "inc_p99": f"${df['annual_inc'].quantile(0.99):,.0f}",
    "util_max": f"{df['revol_util'].max():,.0f}%",
})
print(value_checks.to_string(), "\n")
fmt(missing_types.set_index(["Type", "Column"]),
    {"Missing (2013–2015)": "{:.2%}", "Missing 2013": "{:.2%}", "Missing 2015": "{:.2%}"})

Income of $0 or less                      0
DTI below 0                               0
FICO outside 300–850                      0
Negative balance or count                 0
First credit line after the issue date    0 

		Missing (2013–2015)	Missing 2013	Missing 2015
Type	Column
Structural: the event did not occur	mths_since_last_delinq	49.83%	56.47%	47.99%
	mths_since_last_record	82.37%	87.40%	81.34%
	mths_since_recent_revol_delinq	64.64%	69.79%	63.45%
	mo_sin_old_il_acct	3.70%	4.29%	3.52%
	bc_util	1.05%	0.84%	1.07%
Operational: the process did not record the value	emp_length	6.15%	4.97%	6.69%
	num_tl_120dpd_2m	3.83%	0.21%	5.03%
	14 columns of rule 8	97.32%	100.00%	94.82%
Random: a few cells	revol_util	0.05%	0.06%	0.04%
	pct_tl_nvr_dlq	0.02%	0.11%	0.00%

What we do with each group:

Group	Example in this data	What we do
Structural missing value	mths_since_last_delinq is empty for 50% of the loans: these borrowers never had a delinquency.	Keep the empty cell. “Never” is information.
Operational missing value	emp_length is empty for 5.0% (2013) to 6.7% (2015) of the loans. num_tl_120dpd_2m is empty for 0.2% in 2013 and 5.0% in 2015, so the process changed.	Keep the empty cell as its own group. Rule 8 removes the columns that are almost empty in the training years.
Random missing value	revol_util is empty for 0.05% of the loans.	Keep the empty cell. The group is too small to change the model.
Impossible value	The checks above find 0 cells, for example an income of $0 or less.	If the check finds a value, set it to empty. In this data, no change is necessary.
Extreme but real value	annual_inc goes up to $9.0 million. The 99th percentile is $250,000. revol_util goes up to 892% (a balance above the limit).	Keep the value. Do not cap it.

Why the data needs no imputation and no cap. Both models in Step 4 use only the order of the values, and both give the empty cells their own group. Thus an extreme value only goes into the highest bin, and an empty cell keeps its own risk.

Technical note. The WoE encoder (Step 3) cuts each numeric feature into 10 bins at the deciles of the training data. An income of $9 million falls into the top bin, together with all incomes above the 90th percentile, so its size has no effect. Empty cells go into a separate “missing” bin with their own WoE. The GBM splits each feature at thresholds, so it also uses only the order of the values. At each split, it sends the empty cells to the side that gives the better fit.

An imputation with the median or a fixed value (for example 999) puts an empty cell into a bin with real values. The model then loses the information that the cell was empty. For mths_since_last_delinq, this information is the most important part. The loans with an empty cell have a different default rate from the loans with a recent delinquency. A cap at the 1st and 99th percentiles (winsorization) changes nothing for a model that uses ranks. Thus it adds a step without effect.

2d. Feature engineering

Feature engineering makes a better input from the raw columns. The split axis is the number of raw columns that a new feature uses: one column, or more than one. The table shows what this data needs in each group, and what it does not need.

Group	Method	In this data
One column: numbers	Bins and WoE	Step 3 (the WoE encoder) does this for each numeric feature. A log or square root change is not necessary, because the bins use ranks.
One column: dates	A date becomes a duration	earliest_cr_line becomes credit_hist_months: the months from the first credit line to the issue date.
One column: text categories	A text becomes a number	emp_length (“10+ years”, “< 1 year”) becomes emp_years (10, 0).
One column: categories	Merge rare levels	home_ownership “ANY” and “NONE” become “OTHER”. Both models use the other categories directly. Rule 6 does not let us replace zip_code or emp_title with their default rates.
One column: a range	A range becomes one value	fico_range_low and fico_range_high become fico, the middle of the range.
More than one column: ratios	Divide one column by another	Four new ratios, see below.
More than one column: differences	Income minus the installment	Not used. The installment is the decision of Lending Club (rule 4).
More than one column: flags	For example DTI > 35% and a delinquency	Not used. A flag loses information. The GBM benchmark (Step 4c) finds such combinations itself, and it adds less than 2 KS points to the scorecard.

The four new ratios use only data from the application and the credit bureau. None of them uses the interest rate or the installment, because these are the output of the champion.

Feature	Formula	Business meaning
loan_to_income	loan_amnt / annual_inc	Capacity: the size of the new loan against the income.
loan_to_rev_limit	loan_amnt / total_rev_hi_lim	The size of the new loan against the credit that other lenders gave.
new_acc_share	acc_open_past_24mths / total_acc	Credit appetite: the share of all accounts that the borrower opened in the last 24 months.
bal_to_limit	tot_cur_bal / tot_hi_cred_lim	Utilization: the share of all credit limits that the borrower uses.

If the column below the line is $0 or empty, the ratio is empty (a structural missing value). The new features go into the selection funnel of Step 4 with all other candidates. A new feature stays in the model only if it passes the same rules.

The code also gives each loan a time and an event. Step 7 and Part 2 use them:

• Default. The event month is the month after the last payment.
• Prepayment or full payment. The event month is the month of the last payment.
• Open loan. The loan has no event. Its time stops at the date of the analysis.

# Purpose: Make the calculated features and give each loan a time and an event.
# Input:   `df`, `loans`.
# Output:  Calculated columns of `CALCULATED` in `df`. `vint`, `o19`, `observe()`, `hazards()`, `months_between()`.

# 1. One column: text to a number, a range to one value, a date to a duration.
emp_map = {"< 1 year": 0, "10+ years": 10}
emp_map.update({f"{i} year{'s' if i > 1 else ''}": i for i in range(1, 10)})
df["emp_years"] = df["emp_length"].map(emp_map)
# One FICO value: the middle of the range.
df["fico"] = (df["fico_range_low"] + df["fico_range_high"]) / 2
first_line = pd.to_datetime(df["earliest_cr_line"], format="%b-%Y", errors="coerce")
df["credit_hist_months"] = (
    (df["issue_date"].dt.year - first_line.dt.year) * 12
    + (df["issue_date"].dt.month - first_line.dt.month)
)
# Merge the rare categories.
df["home_ownership"] = df["home_ownership"].replace({"ANY": "OTHER", "NONE": "OTHER"})


# 2. More than one column: ratios. No division by $0: the ratio is then empty.
def ratio(top, bottom):
    """top / bottom, or NaN if bottom is 0, negative or empty."""
    return top.astype(float) / bottom.astype(float).where(bottom > 0)


df["loan_to_income"] = ratio(df["loan_amnt"], df["annual_inc"])
df["loan_to_rev_limit"] = ratio(df["loan_amnt"], df["total_rev_hi_lim"])
df["new_acc_share"] = ratio(df["acc_open_past_24mths"], df["total_acc"])
df["bal_to_limit"] = ratio(df["tot_cur_bal"], df["tot_hi_cred_lim"])
# Check: no infinite values in the calculated features.
assert np.isfinite(df[list(CALCULATED)].fillna(0).to_numpy()).all(), "A calculated feature has an infinite value."


def months_between(start, end):
    return (end.dt.year - start.dt.year) * 12 + (end.dt.month - start.dt.month)


# 3. Time and event of each loan, for all 36-month loans of 2013–2018.
VINT_COLS = ["issue_date", "issue_year", "grade", "loan_status", "last_pymnt_d"]
vint = loans.loc[loans["issue_year"].between(2013, 2018) & (loans["application_type"] == "Individual"),
                 VINT_COLS].copy()
mob_last = months_between(vint["issue_date"], pd.to_datetime(vint["last_pymnt_d"], format="%b-%Y")).fillna(0)
vint["is_default"] = vint["loan_status"].isin(BAD_STATUS)
vint["is_paid"] = vint["loan_status"] == "Fully Paid"
# Default: month after the last payment. Full payment: month of the last payment. Open: no event.
vint["event_month"] = np.where(vint["is_default"], mob_last + 1,
                               np.where(vint["is_paid"], mob_last.clip(lower=1), np.nan)).clip(max=36)


def observe(v, as_of):
    """The time and the event of each loan, with only the data up to the date of the analysis."""
    age = months_between(v["issue_date"], pd.Series(pd.Timestamp(as_of), index=v.index))
    seen = v["event_month"] <= age
    return pd.DataFrame({
        "time": np.where(seen, v["event_month"], age).clip(0, 36),
        "event": np.where(seen & v["is_default"], 1, np.where(seen & v["is_paid"], 2, 0)),
        "grade": v["grade"].to_numpy(), "vintage": v["issue_year"].to_numpy(),
    }, index=v.index)


def hazards(o):
    """Monthly hazard of default and of prepayment, for months on book 1 to 36."""
    t, ev = o["time"].to_numpy(), o["event"].to_numpy()
    months = np.arange(1, 37)
    at_risk = np.array([(t >= m).sum() for m in months])
    with np.errstate(invalid="ignore", divide="ignore"):
        h_def = np.array([((t == m) & (ev == 1)).sum() for m in months]) / at_risk
        h_pay = np.array([((t == m) & (ev == 2)).sum() for m in months]) / at_risk
    return pd.DataFrame({"at_risk": at_risk, "h_def": h_def, "h_pay": h_pay}, index=months).fillna(0)


# 4. The view of all loans with all data (March 2019).
o19 = observe(vint, DATA_END)  # All data up to the end (March 2019)
del loans, mob_last, first_line  # Free memory. The steps after this use df and vint.
calc_view = df[list(CALCULATED)].describe().T[["mean", "min", "50%", "max"]]
calc_view["Missing"] = df[list(CALCULATED)].isna().mean()
calc_view.index.name = "Calculated feature"
fmt(calc_view, {"mean": "{:,.2f}", "min": "{:,.2f}", "50%": "{:,.2f}", "max": "{:,.2f}", "Missing": "{:.2%}"})

	mean	min	50%	max	Missing
Calculated feature
fico	695.55	662.00	687.00	847.50	0.00%
emp_years	5.93	0.00	6.00	10.00	6.15%
credit_hist_months	196.34	36.00	178.00	852.00	0.00%
loan_to_income	0.20	0.00	0.18	0.50	0.00%
loan_to_rev_limit	0.66	0.00	0.50	200.00	0.04%
new_acc_share	0.20	0.00	0.17	1.00	0.00%
bal_to_limit	0.72	0.00	0.77	9.02	0.00%

2e. Baseline: what did the booked loans give to the lender?

The data is now clean. Before we build a model, we must know the result that the model must beat. Two later steps need a dollar value for each loan. The swap-set (Step 6) compares the loans that each policy approves. The cutoff (Step 7) adds up the NPV of the approved loans. This section calculates that value and shows the baseline.

The data gives the total payments, the date of the last payment and the recoveries of each loan. It does not give each monthly payment. The code makes the monthly cash flows with these rules (as-06):

• The borrower pays the installment each month until the month of the last payment.
• The borrower pays the rest of the total payments in the month of the last payment. For a loan that the borrower pays early, this is the prepayment.
• The recoveries, minus the collection fee, come 12 months after the last payment.

The NPV uses three costs. The cost of funds is 3% for each year (as-01). The servicing cost is 1% of each payment (as-02). The acquisition cost is $300 for each loan (as-03).

Technical note. For a loan with installment I, last payment in month n, total borrower payments P (without recoveries) and monthly discount factor v = 1 / (1 + 0.03/12):

• k = min(n, floor(P / I)) months with a full installment, and the rest R = P − k × I in month n.
• NPV = (1 − 0.01) × [I × (1 − v^k) / (0.03/12) + R × v^n] + (recoveries − fee) × v^(n + 12) − funded amount − $300.

The NPV for each $1 (npv_rate) uses the same formula without the acquisition cost, divided by the funded amount.

# Purpose: Calculate the realized NPV of each loan from its actual payments (as-01 to as-06).
# Input:   `df` (payments, recoveries, dates).
# Output:  Columns `npv` ($) and `npv_rate` (NPV for each $1, before the acquisition cost).

# Assumptions from the assumption register.
COST_OF_FUNDS = 0.03  # as-01, per year
SERVICING = 0.01      # as-02, share of each payment
ACQUISITION = 300     # as-03, $ for each new loan
RECOVERY_LAG = 12     # as-06, months from the last payment to the recovery


def loan_npv(X, cof=COST_OF_FUNDS, acq=ACQUISITION, serv=SERVICING):
    """NPV of each loan from its actual cash flows (as-06)."""
    # Monthly discount rate and discount factor.
    i = cof / 12
    v = 1 / (1 + i)
    # n = month of the last payment, counted from the issue date.
    last = pd.to_datetime(X["last_pymnt_d"], format="%b-%Y")
    n = months_between(X["issue_date"], last).fillna(0).clip(lower=0).to_numpy()
    # Borrower payments without the recoveries.
    paid = (X["total_pymnt"] - X["recoveries"]).to_numpy()
    inst = X["installment"].to_numpy()
    k = np.minimum(n, np.floor(paid / inst))  # Months with a full installment
    rest = paid - k * inst                    # Rest of the payments, in the month of the last payment
    # PV of k full installments, plus the rest in month n.
    pv_paid = inst * (1 - v ** k) / i + rest * v ** np.maximum(n, 1)
    # Recoveries minus the collection fee, 12 months after the last payment (as-06).
    net_rec = (X["recoveries"] - X["collection_recovery_fee"]).to_numpy()
    pv_rec = net_rec * v ** (n + RECOVERY_LAG)
    # NPV = PV of payments after servicing + PV of recoveries - amount lent - acquisition cost.
    return (1 - serv) * pv_paid + pv_rec - X["funded_amnt"].to_numpy() - acq


def npv_rate(X, cof=COST_OF_FUNDS):
    """NPV before the acquisition cost, for each $1 of the loan."""
    return loan_npv(X, cof=cof, acq=0) / X["funded_amnt"].to_numpy()


df["npv"] = loan_npv(df)
df["npv_rate"] = npv_rate(df)

The table gives the baseline by issue year: what the booked loans gave to the lender. It contains only booked loans, so it shows the result of the champion policy of Lending Club.

# Purpose: Show what the booked loans gave to the lender, by issue year (the baseline).
# Input:   `df`.
# Output:  Table `cur` of the baseline. Numbers for the business note.

cur = df.groupby("issue_year").agg(
    Loans=("npv", "size"), amount=("funded_amnt", "sum"), bad=("default", "mean"),
    rate=("int_rate", "mean"), npv=("npv", "sum"), npv_loan=("npv", "mean"))
cur = cur.rename(columns={"amount": "Amount funded ($M)", "bad": "Default rate", "rate": "Interest rate",
                          "npv": "Realized NPV ($M)", "npv_loan": "Realized NPV for each loan"})
# Change the units for display: $M and fractions.
cur["Amount funded ($M)"] /= 1e6
cur["Interest rate"] /= 100
cur["Realized NPV ($M)"] /= 1e6


def money(x, unit="M", dec=1):
    """Text for a dollar value with the minus sign before the $ sign, for example −$32.9M."""
    return f"{'−' if x < 0 else ''}${abs(x):,.{dec}f}{unit}"


N.update({"cur_npv_13_14": money(cur.loc[[2013, 2014], "Realized NPV ($M)"].sum()),
          "cur_npv_15": money(cur.loc[2015, "Realized NPV ($M)"]),
          "cur_npv_13": money(cur.loc[2013, "Realized NPV ($M)"]),
          "cur_npv_loan_15": money(cur.loc[2015, "Realized NPV for each loan"], unit="", dec=0),
          "cur_bad_13": f"{cur.loc[2013, 'Default rate']:.1%}", "cur_bad_15": f"{cur.loc[2015, 'Default rate']:.1%}",
          "cur_rate_13": f"{cur.loc[2013, 'Interest rate']:.1%}", "cur_rate_15": f"{cur.loc[2015, 'Interest rate']:.1%}"})
cur.index.name = "Issue year"
fmt(cur, {"Loans": "{:,.0f}", "Amount funded ($M)": "{:,.0f}", "Default rate": "{:.1%}",
          "Interest rate": "{:.1%}", "Realized NPV ($M)": "{:,.1f}", "Realized NPV for each loan": lambda v: money(v, "", 0)})

	Loans	Amount funded ($M)	Default rate	Interest rate	Realized NPV ($M)	Realized NPV for each loan
Issue year
2013	100,422	1,272	12.3%	13.4%	42.9	$427
2014	162,570	2,046	13.7%	12.5%	28.8	$177
2015	282,848	3,622	14.9%	11.3%	-32.9	−$116

# Purpose: Show the baseline that each later step must beat: the realized NPV by issue year.
# Input:   `cur`.
# Output:  Bar chart.

import chart_style as cs
cs.setup()  # House style (safe to call again)

npv_m = cur["Realized NPV ($M)"]
fig, ax = plt.subplots(figsize=(8, 4))
# Losses in red (with a minus sign), gains in navy.
colors = [cs.NEGATIVE if v < 0 else cs.NAVY for v in npv_m]
ax.bar(npv_m.index.astype(str), npv_m, color=colors, width=0.6)
ax.axhline(0, color=cs.GRAY, lw=0.8)
# Direct labels: NPV, default rate and interest rate of each vintage.
for x, (yr, row) in enumerate(cur.iterrows()):
    v = row["Realized NPV ($M)"]
    ax.annotate(money(v), (x, v), xytext=(0, 4 if v >= 0 else -12),
                textcoords="offset points", ha="center", fontsize=10, fontweight="bold",
                color=cs.NEGATIVE if v < 0 else cs.NAVY)
ax.set_xticks(range(len(cur)))
ax.set_xticklabels([f"{yr}\ndefault {r['Default rate']:.1%} · rate {r['Interest rate']:.1%}"
                    for yr, r in cur.iterrows()])
ax.set_yticks([])
ax.spines["left"].set_visible(False)
ax.margins(y=0.15)
v13, v15 = npv_m.loc[2013], npv_m.loc[2015]
ax.set_title("Realized NPV of the booked 36-month loans, $ millions, by issue year (as-01 to as-06)",
             fontsize=9.5, color=cs.MUTED, fontweight="normal", loc="left")
cs.action_title(fig, f"Realized NPV fell from {money(v13)} (2013) to {money(v15)} (2015)")
plt.show()

How to read: each bar is the realized NPV of all booked loans of one issue year, after all costs (as-01 to as-03). The labels under the bars give the default rate and the mean interest rate.

So what: the 2015 loans lost money. The default rate went up from 12.3% to 14.9%, and the mean interest rate went down from 13.4% to 11.3%. This is the complication of Step 0 in dollars.

Business note. The model learns from 545,840 clean loans and 65 candidate columns, with four new ratio features. The baseline is clear: the 2013–2014 loans gave a realized NPV of $71.7M, but the 2015 loans gave −$32.9M (−$116 for each loan). Thus the problem has two parts: which applicants to approve (Steps 3–7), and at which price (Part 2: Pricing). Step 3 now splits the data so that the test looks like the future.

Step 3. Split data

Step 2 gave a clean target and 65 candidate columns for the 2013–2015 loans. Before we build a model, we must agree on two things. What does “a good score” mean, and which loans give an honest test of it?

Purpose. This step separates the data that trains the model from the data that tests it. It supports this decision: approve the challenger score for the test, with a result that is near to the result in production.

Steps 3 to 7 use the same small set of tools: the metrics, the WoE encoder, the GBM settings and the policy rules. The code keeps them in the file case01_helpers.py, so that the story stays short. The appendix shows the full file. The next cell loads the tools and gives them the categorical columns and the seed.

# Purpose: Load the helper functions that Steps 3–7 use (metrics, PSI, WoE, logistic regression, GBM).
# Input:   `CAT_ALL` (categorical columns, Step 2), `SEED`.
# Output:  The functions and constants of case01_helpers.py in the namespace of the notebook.

import chart_style as cs  # Chart style (deck style guide). A second import is safe.
import case01_helpers
from case01_helpers import *  # noqa: F401,F403

# The GBM uses the seed. split_types() uses the categorical columns.
configure(cat_all=CAT_ALL, seed=SEED)
cs.setup()

3a. What the test must measure

A model project has many metrics. A reader can ask: “Why these metrics, and why not others?” Thus we start from the question that each metric answers.

The tree below splits the risks of a new score in two groups. The first group is statistical: the score can fail on the order of applicants, on the level of the PD, or over time. The second group is business: the score can be correct but add no money, or add money but break a law. A score that passes all five branches is ready for a test. If a score fails a branch, the team must explain the risk before a test. These five branches cover each way that a score can fail in production, and each failure belongs to one branch only.

Branch	Question that it answers	Metric	Rule in this notebook	Where we use it
1. Order of risk	Does the score put the risky applicants above the safe applicants?	AUC, Gini, KS	KS above the champion. A complex model (the GBM) replaces the scorecard only if it adds 2 KS points or more.	3b, 4b, 4c, 6a
2. Level	Is the mean PD equal to the actual default rate?	Mean PD against the actual default rate, for all loans and for each score decile	A difference of less than 1 point after the recalibration	4d, 5a, 6b
3. Stability	Do the applicants of the test period look like the applicants of the training period?	PSI of the score and of each feature	Less than 0.1: stable. 0.1 to 0.25: examine. More than 0.25: do not use.	3b, 4a, 6c
4. Business value	Does the score approve more applicants or add NPV at the same risk?	Approval at the champion bad rate, swap-set, NPV	More approvals than the champion at the same bad rate	4b, 4c, 6d, 7
5. Explainability	Can the lender give the true reasons for each decline?	Reason codes for each declined applicant	Three reasons with a plain-English meaning, for 95% or more of the declined applicants	4b, 4c, 5b, 6e

Branches 1 and 2 are different. A score can put applicants in the correct order but give a PD that is too low. The order sets which applicants pass the cutoff. The level sets the expected NPV. Thus the notebook checks both.

Technical note. How each metric works:

• AUC. The probability that a random default gets a higher score than a random non-default. 0.5 is random, 1.0 is perfect.
• Gini. 2 × AUC − 1. It has the same information as the AUC. Credit teams report it because 0 means “no power”.
• KS. Sort the applicants by score. At each score, calculate the share of all defaults and the share of all non-defaults below that score. The KS is the largest difference between the two shares. Credit teams and model risk reviewers use the KS as a standard, because it shows the separation at the best single cutoff. A cutoff is the decision that this notebook supports.
• PSI. Cut the score of the training data into ten bins of equal size. For each bin, compare the share of the training loans (e) with the share of the test loans (a). PSI = Σ (a − e) × ln(a / e). The thresholds 0.1 and 0.25 are the industry convention.
• Approval at the champion bad rate. Sort the applicants from the lowest PD. Approve them one at a time. Stop before the bad rate of the approved loans goes above the champion bad rate.

3b. Which period tests the model?

Which split gives a test result that is near to production?

Why this question matters. The lender will use the score on applicants of next year, not on applicants of last year. If the test uses the wrong loans, the test result can look good, and the score then fails in production. The approval cutoff then approves too many defaults.

Ways to do it. The split axis is time: where are the test loans in time, compared with the training loans? There are four possible answers, and each split method belongs to one of them.

Group	Method	Advantage	Main risk	Status
Same period	Random split (out-of-sample), for example 80/20, or k-fold cross-validation	Simple. The two sets have the same mix of years and the same default rate.	Temporal leakage: the training data contains loans of the test period. The test cannot show a change over time.	Possible. Test it.
Earlier period	Train on later loans, test on earlier loans	None for this decision.	The model uses the future to predict the past. Production never has this data.	Screened out
Later period, one cut	Out-of-time split: train on 2013–2014, test on 2015	Same condition as production: a model from past loans scores new applicants. It shows a change of the population and of the default rate.	Fewer loans for the training. The test result is often lower.	Possible. Test it.
Later period, many cuts	Rolling time windows: train on 2013, test on 2014. Then train on 2013–2014, test on 2015.	Shows how fast the score loses power.	More work. The first window is short.	Possible. Use it as a check.

Evidence. The code makes each split and trains the same model on each one. It uses a logistic regression with WoE on all 65 candidate columns, because this model is fast and the same for each split. For each test set, the code measures three branches of the framework:

• Branch 1, order of risk: the AUC, the Gini and the KS.
• Branch 2, level: the default rate of the training loans and of the test loans.
• Branch 3, stability: the PSI of the score from the training loans to the test loans.

# Purpose: Compare a random split, an out-of-time split and a rolling window (dec-02).
# Input:   `df`, `CANDIDATES`.
# Output:  `train`, `test`, `y_test` (the out-of-time split), table `dec02`.

# Out-of-time split: the model never sees the 2015 loans in training.
train = df[df["issue_year"] <= 2014].copy()
test = df[df["issue_year"] == 2015].copy()
y_test = test["default"].to_numpy()
# The GBM codes the categories with the levels of the training data only.
configure(train=train)

# The three splits. Each one is (training loans, test loans).
splits = {
    "Random 80/20 (2013–2015 mixed)": train_test_split(df, test_size=0.2, random_state=SEED),
    "Rolling window: train 2013, test 2014": (df[df["issue_year"] == 2013], df[df["issue_year"] == 2014]),
    "Out-of-time: train 2013–2014, test 2015": (train, test),
}
rows = []
for name, (tr, te) in splits.items():
    m = fit_woe_lr(tr, CANDIDATES)
    p_tr, p_te = pd_woe_lr(m, tr), pd_woe_lr(m, te)
    rows.append({"Split": name, "Training loans": len(tr), "Test loans": len(te),
                 "Default rate, training": tr["default"].mean(), "Default rate, test": te["default"].mean(),
                 **score_metrics(te["default"], p_te), "PSI of the score": psi(p_tr, p_te)})
dec02 = pd.DataFrame(rows).set_index("Split")

rnd, oot = dec02.iloc[0], dec02.iloc[2]
N.update({
    "bad_train": f"{train['default'].mean():.1%}", "bad_2015": f"{test['default'].mean():.1%}",
    "rnd_bad_tr": f"{rnd['Default rate, training']:.1%}", "rnd_bad_te": f"{rnd['Default rate, test']:.1%}",
    "auc_rnd": f"{rnd['AUC']:.3f}", "auc_oot": f"{oot['AUC']:.3f}", "auc_roll": f"{dec02.iloc[1]['AUC']:.3f}",
    "psi_oot": f"{oot['PSI of the score']:.3f}",
    "n_train": f"{len(train):,}", "one_in": f"{round(1 / train['default'].mean())}",
})
fmt(dec02, {"Training loans": "{:,.0f}", "Test loans": "{:,.0f}", "Default rate, training": "{:.1%}",
            "Default rate, test": "{:.1%}", "AUC": "{:.3f}", "Gini": "{:.3f}", "KS": "{:.3f}",
            "PSI of the score": "{:.3f}"})

	Training loans	Test loans	Default rate, training	Default rate, test	AUC	Gini	KS	PSI of the score
Split
Random 80/20 (2013–2015 mixed)	436,672	109,168	14.1%	14.2%	0.678	0.356	0.258	0.000
Rolling window: train 2013, test 2014	100,422	162,570	12.3%	13.7%	0.670	0.340	0.245	0.009
Out-of-time: train 2013–2014, test 2015	262,992	282,848	13.2%	14.9%	0.683	0.366	0.266	0.004

How to read: each row is one split. The first two columns of rates show the default rate of the training loans and of the test loans.

So what: the AUC is almost the same for all splits, but only the time splits show that the default rate goes up.

Decision. Out-of-time split: train on the 2013–2014 loans, test on the 2015 loans. The rolling window is a check of the speed of decay.

Why. Three reasons, from the most important:

1. Production scores the future. The model learns from past loans and scores new applicants. The out-of-time split copies this condition. In this data, the default rate goes up from 13.2% in the training loans to 14.9% in 2015. The random split hides this change: its training loans and its test loans have the same default rate (14.1% and 14.2%). Step 4d shows that this change makes the PD too low.
2. Model risk review asks for it. A random split puts 2015 loans into the training data. This is temporal leakage. The model can learn the conditions of 2015 and then “predict” 2015. Model risk review does not accept a random split as the main test of an approval model.
3. It measures the decay of the model. The rolling window gives an AUC of 0.670 one year after the training period. The out-of-time split gives 0.683. Thus the order of applicants by risk is stable over one year. The PSI of the score is 0.004, much less than 0.1. Risk can use the same test each year to plan the next model update.

The random split gives an AUC of 0.678. In this data, the random split does not give a better AUC. The leakage here does not change the order of risk. It hides the change of level. Each year of a credit cycle can give a different result, so the safe method stays the out-of-time split.

What would change the decision. If the business scores only loans from the training period, for example to review the current portfolio, a random split is sufficient.

3c. The champion on the test year

Step 0 defines the champion policy (as-04): approve the best sub-grades, up to about 85% of the applicants. We apply it here, because the policy needs the test year first. All later comparisons use the same 2015 applicants and the same champion.

# Purpose: Apply the champion policy (as-04) to the 2015 applicants.
# Input:   `test`, `y_test`.
# Output:  `champ_rank`, `champ_ok` (True = approved), `champ_bad` (bad rate of the champion).

APPROVAL = 0.85
champ_rank = subgrade_rank(test)
# Approve the best sub-grades (A1 first) until about 85% of applicants are approved.
champ_cut = np.sort(champ_rank)[int(APPROVAL * len(champ_rank))]
champ_ok = champ_rank < champ_cut
champ_bad = y_test[champ_ok].mean()
N.update({"champ_bad": f"{champ_bad:.1%}", "champ_appr": f"{champ_ok.mean():.1%}",
          "champ_last_sg": SUBGRADE_ORDER[champ_cut - 1], "champ_cut_sg": SUBGRADE_ORDER[champ_cut]})
print(f"Champion: approves {champ_ok.mean():.1%} of the 2015 applicants, with a bad rate of {champ_bad:.1%}.")

Champion: approves 84.5% of the 2015 applicants, with a bad rate of 12.5%.

In 2015, the champion approves the sub-grades A1 to C5 and declines D1 and worse. The champion bad rate is the risk limit of branch 4.

Business note. We test each score on the 2015 applicants, as if 2015 is the next year. The champion approves 84.5% of them, with a bad rate of 12.5%. A challenger must approve more applicants at this bad rate. Step 4 builds the challenger and judges it with the five branches of the evaluation framework.

Step 4. Build models

Step 2 left 65 candidate columns, with four new ratio features. Step 3 split the data by time and fixed the champion: it approves 84.5% of the 2015 applicants with a bad rate of 12.5%. This step builds the score that must beat the champion. It first answers which columns the model uses, then which model gives the score.

Purpose. This step builds the score that the cutoff uses. It supports this decision: the challenger score for the champion/challenger test.

4a. Feature selection

Which columns should the model use?

Why this question matters. Each column in the model needs a reason that a model risk reviewer accepts. A column that leaks the outcome makes the test result too good, and the model then fails in production. Too few columns make the score weak, so the challenger approves fewer applicants at the same bad rate. Too many similar columns make the reason codes harder to explain and the model harder to monitor.

Why a column leaves the model. The answer has two parts: the reasons to remove a column, and the method that finds them. The reasons form a tree with three buckets. The split axes are permission, data and mathematical value. Each column gets the first rule that removes it, so the buckets do not overlap. A column that no rule removes is a feature.

Bucket	Question	Rule	Where
1. Not permitted	May the lender use the value for the decision?	3. Known only after the decision (leakage) · 4. Decision of Lending Club · 5. Joint applications (out of scope) · 6. Fair lending risk	Step 2b
2. No usable data	Does the column hold data about the risk of the borrower?	1. Identifiers, free text and process fields · 2. Constant in scope · 8. Not collected in the training years	Step 2b
3. No mathematical value	Does the column add information to the other columns?	7. Replaced by a calculated column · 9. Weak · 10. Duplicate · 11. Unstable	Steps 2b and 4a

Buckets 1 and 2 do not depend on the model. Step 2 applied them. Bucket 3 needs a method that measures the value of a column. There are three groups of methods. The split axis is the role of the model in the selection:

• Filter methods: no model. Each column gets a score alone, for example its IV, its correlation with a stronger column or its PSI. The advantage: fast, and each removed column has a written reason. The risk: a column that is weak alone can help in combination with other columns.
• Embedded methods: the model selects while it trains. An L1 logistic regression sets the weight of a weak column to zero. A GBM uses only the columns that improve its splits. The advantage: one step, and the model sees the combinations. The risk: the model keeps many similar columns, and no column gets a written reason.
• Wrapper methods: the model trains many times on different sets of columns. The method keeps the set with the best result on a validation sample. The advantage: it measures what each column adds to the model. The risk: many fits, and the result can follow the noise of the validation sample.

We screen out two methods. A full search of all sets of columns needs 2 to the power of 65 fits. A separate L1 logistic regression selects only for a straight line on the raw values, and it needs a value in each empty cell. The GBM uses the combinations of columns and the empty cells directly. It is also the strongest model of section 4c, so it shows the most power that the columns can give. Thus the GBM on all candidates represents the embedded group.

Each group gets one test on the same data: the selection funnel (filter), the GBM on all candidates (embedded) and a backward elimination (wrapper). The backward elimination starts with all candidates and removes the weakest columns in each round, as a reviewer asked.

Technical note. Why not R²? R² measures how much of the variance of a continuous outcome a model explains. The target here is 0 or 1. The cutoff needs the order of the applicants, so the right metric is the AUC.

The AUC is the probability that a random default gets a higher PD than a random non-default. The deviance (log loss) measures the fit of the PD level. Step 5 corrects the PD level, so the selection uses the AUC.

Evidence 1: the filter. Rules 9 to 11 use the training data (2013–2014). Rule 11 also uses the distribution of each column in 2015, but not the 2015 outcomes.

Rule	Test	Threshold
9. Weak	IV on the training data	Less than 0.01
10. Duplicate	Spearman correlation with a column that has a higher IV	0.8 or more (absolute value)
11. Unstable	PSI of the column between the training data and the 2015 applicants	More than 0.1

# Purpose: Apply rules 9–11 of the selection funnel on the training data (dec-10).
# Input:   `train`, `test` (no target), `CANDIDATES`.
# Output:  `FUNNEL_SET`, `iv_all`, `weak`, `duplicate_of`, `psi_col`, `unstable`, `woe_all`.

IV_LIMIT, CORR_LIMIT, PSI_LIMIT = 0.01, 0.8, 0.1
# IV of each candidate on the training data. The WoE bins come from the training data only.
num_c, cat_c = split_types(CANDIDATES)
woe_all = WoEEncoder(num_c, cat_c).fit(train, train["default"])
iv_all = pd.Series(woe_all.iv_).sort_values(ascending=False)

# Rule 9: weak
weak = list(iv_all.index[iv_all < IV_LIMIT])
left = [c for c in iv_all.index if c not in weak]

# Rule 10: duplicate. Keep the column with the higher IV.
# Spearman correlation on a sample of 100,000 loans (fast, and precise enough).
sample = train.sample(100_000, random_state=SEED)
rho = sample[[c for c in left if c not in CAT_ALL]].corr(method="spearman").abs()
# Go from the highest IV to the lowest. Keep a column if no kept column is its twin.
kept, duplicate_of = [], {}
for c in left:
    twin = next((k for k in kept if c in rho.index and k in rho.index and rho.loc[c, k] >= CORR_LIMIT), None)
    if twin is None:
        kept.append(c)
    else:
        duplicate_of[c] = twin


# Rule 11: unstable. PSI of the WoE bins of each column, train against test (no target).
def column_psi(col):
    a = woe_all._bin(col, train[col]).value_counts(normalize=True)
    b = woe_all._bin(col, test[col]).value_counts(normalize=True)
    a, b = a.align(b, fill_value=0)
    a, b = np.clip(a, 1e-6, None), np.clip(b, 1e-6, None)
    return float(np.sum((b - a) * np.log(b / a)))


psi_col = pd.Series({c: column_psi(c) for c in kept})
unstable = list(psi_col.index[psi_col > PSI_LIMIT])
FUNNEL_SET = [c for c in kept if c not in unstable]
del sample, rho
eng_result = {c: ("weak, rule 9" if c in weak else f"duplicate of {duplicate_of[c]}, rule 10" if c in duplicate_of
                  else "unstable, rule 11" if c in unstable else "kept") for c in ENGINEERED}
N.update({"n_weak": f"{len(weak)}", "n_dup": f"{len(duplicate_of)}", "n_unstable": f"{len(unstable)}",
          "n_funnel": f"{len(FUNNEL_SET)}", "n_all": f"{len(CANDIDATES)}",
          "iv_bal_to_limit": f"{iv_all['bal_to_limit']:.3f}", "iv_tot_cur_bal": f"{iv_all['tot_cur_bal']:.3f}",
          "eng_kept": ", ".join(f"`{c}`" for c, r in eng_result.items() if r == "kept") or "none",
          "eng_out": ", ".join(f"`{c}` ({r})" for c, r in eng_result.items() if r != "kept") or "none"})
print(f"Filter: {len(CANDIDATES)} candidates. Rule 9 removes {len(weak)}, rule 10 removes {len(duplicate_of)}, "
      f"rule 11 removes {len(unstable)}. {len(FUNNEL_SET)} columns are left.")
fmt(pd.DataFrame({"IV": iv_all.reindex(ENGINEERED), "Result": pd.Series(eng_result).str.capitalize()})
    .rename_axis("New ratio feature"), {"IV": "{:.3f}"})

Filter: 65 candidates. Rule 9 removes 30, rule 10 removes 7, rule 11 removes 0. 28 columns are left.

	IV	Result
New ratio feature
loan_to_income	0.043	Kept
loan_to_rev_limit	0.034	Kept
new_acc_share	0.096	Kept
bal_to_limit	0.005	Weak, rule 9

The filter keeps 28 of the 65 candidates. Of the four new ratios, the filter keeps `loan_to_income`, `loan_to_rev_limit`, `new_acc_share`. It removes `bal_to_limit` (weak, rule 9). Alone, bal_to_limit has an IV of 0.005, less than the balance tot_cur_bal (0.064). Thus a ratio is a good idea only if the data confirms it. But a column that is weak alone can still help in combination, and the wrapper below tests this.

Evidence 2: the wrapper. The backward elimination must not see the 2015 loans, because the 2015 loans are the final test. Thus it uses only the training data, split by time: it trains on the 2013 loans and validates on the 2014 loans. This is the same direction as the main split: the past predicts the next year.

The method has these parameters:

• Start. All 65 candidates. Buckets 1 and 2 already removed the columns that are not permitted or that hold no usable data.
• Importance of a column. The drop of the validation AUC when the code shuffles the column (permutation importance), on a sample of 50,000 loans of 2014.
• Each round. Train the GBM, measure the importance of each column, then remove the 25% of columns with the lowest importance. A group of columns in each round keeps the run time short.
• Stop. Stop when the validation AUC falls more than 0.005 below the AUC with all candidates. The result is the last set of columns above this limit.

Technical note. Permutation importance: for column j, replace its values with a random permutation of the same values, then score the validation sample again. The importance is AUC(original) − AUC(shuffled). A shuffled column keeps its distribution but loses its link to the target and to the other columns.

Thus the drop measures what the model loses without the column. A column with an importance near zero or below zero adds only noise. The GBM uses the settings of Step 3 (GBM_PARAMS), with early stop on 10% of the 2013 loans. The tolerance of 0.005 AUC is 1 point of Gini. A smaller tolerance keeps more columns, and a larger tolerance removes more.

# Purpose: Remove the weakest columns in rounds, with the GBM, on the training years only (dec-10).
# Input:   `train`, `CANDIDATES`, `fit_gbm()`, `gbm_matrix()`.
# Output:  `WRAPPER_SET`, `be_path` (one row for each round), `be_round` (round in which each column leaves).

BE_DROP, BE_TOL, BE_SAMPLE = 0.25, 0.005, 50_000
fit_part = train[train["issue_year"] == 2013]
val_part = train[train["issue_year"] == 2014]
X_val_all = gbm_matrix(val_part, CANDIDATES)
y_val = val_part["default"].to_numpy()
# Fixed sample of 2014 loans for the permutation importance.
imp_idx = np.random.default_rng(SEED).choice(len(val_part), BE_SAMPLE, replace=False)
perm_rng = np.random.default_rng(SEED)


def permutation_importance(model, cols):
    """Drop of the AUC on the sample when the code shuffles each column."""
    Xs, ys = X_val_all[cols].iloc[imp_idx], y_val[imp_idx]
    base = roc_auc_score(ys, model.predict_proba(Xs)[:, 1])
    out = {}
    for c in cols:
        Xp = Xs.copy()
        Xp[c] = perm_rng.permutation(Xp[c].to_numpy())
        out[c] = base - roc_auc_score(ys, model.predict_proba(Xp)[:, 1])
    return pd.Series(out).sort_values()


cols, path, be_round = list(CANDIDATES), [], {}
while True:
    model = fit_gbm(fit_part, cols)
    auc = roc_auc_score(y_val, model.predict_proba(X_val_all[cols])[:, 1])
    path.append({"Round": len(path), "Columns": len(cols), "Validation AUC (2014)": auc, "cols": list(cols)})
    # Stop: the AUC falls more than the tolerance below the AUC with all candidates.
    if auc < path[0]["Validation AUC (2014)"] - BE_TOL or len(cols) <= 5:
        break
    imp = permutation_importance(model, cols)
    drop = list(imp.index[:int(np.ceil(BE_DROP * len(cols)))])
    be_round.update({c: len(path) for c in drop})
    cols = [c for c in cols if c not in drop]

be_path = pd.DataFrame(path).set_index("Round")
# Result: the last set with an AUC above the limit.
ok_rounds = be_path[be_path["Validation AUC (2014)"] >= be_path["Validation AUC (2014)"].iloc[0] - BE_TOL]
best_round = ok_rounds.index.max()
WRAPPER_SET = be_path.loc[best_round, "cols"]
N.update({"n_wrapper": f"{len(WRAPPER_SET)}", "be_rounds": f"{len(be_path) - 1}",
          "be_auc0": f"{be_path['Validation AUC (2014)'].iloc[0]:.3f}",
          "be_auc_best": f"{be_path.loc[best_round, 'Validation AUC (2014)']:.3f}",
          "be_auc_last": f"{be_path['Validation AUC (2014)'].iloc[-1]:.3f}",
          "be_n_last": f"{be_path['Columns'].iloc[-1]}"})
fmt(be_path.drop(columns="cols"), {"Validation AUC (2014)": "{:.4f}"})

	Columns	Validation AUC (2014)
Round
0	65	0.6792
1	48	0.6794
2	36	0.6786
3	27	0.6773
4	20	0.6753
5	15	0.6731

# Purpose: Show where the backward elimination stops.
# Input:   `be_path`, `best_round`, `BE_TOL`.
# Output:  Line chart.

import chart_style as cs
cs.setup()  # House style (safe to call again)

fig, ax = plt.subplots(figsize=(8, 4))
x, y = be_path["Columns"], be_path["Validation AUC (2014)"]
ax.plot(x, y, "o-", color=cs.NAVY)
# The limit: AUC with all candidates minus the tolerance.
limit = y.iloc[0] - BE_TOL
ax.axhline(limit, color=cs.GRAY, lw=1, ls="--")
cs.label_end(ax, x.max(), limit, "Stop limit", color=cs.GRAY, dy=-8)
# The selected set in blue.
ax.plot(x.loc[best_round], y.loc[best_round], "o", color=cs.BLUE, ms=10)
cs.label_end(ax, x.loc[best_round], y.loc[best_round], f"{x.loc[best_round]} columns", color=cs.BLUE,
             dy=10, bold=True)
ax.invert_xaxis()
ax.set_xlabel("Number of columns in the GBM (rounds go from left to right)")
ax.set_ylabel("AUC on the 2014 loans")
ax.set_title("Backward elimination: train on 2013, validate on 2014, remove 25% of the columns in each round",
             fontsize=9.5, color=cs.MUTED, fontweight="normal", loc="left")
cs.action_title(fig, f"{x.loc[best_round]} of {x.iloc[0]} columns keep the AUC within {BE_TOL} of all columns")
plt.show()

How to read: each point is one round of the backward elimination. The line goes from all candidates on the left to the smallest set on the right. The blue point is the last set above the dashed stop limit.

So what: the GBM loses almost nothing from 65 to 20 columns. The validation AUC goes from 0.679 to 0.675. With fewer columns, the AUC falls fast (0.673 with 15 columns). Thus most of the candidates carry the same information.

Evidence 3: the three sets on the 2015 test. The code trains the same GBM on all 2013–2014 loans with each set of columns. It then scores the 2015 loans.

Three metrics compare the sets. The AUC measures the order of the applicants. The KS measures the best separation of defaults and non-defaults. The approval rate at the bad rate of the champion is the business test.

# Purpose: Compare the three selection methods with the same GBM on the 2015 loans (dec-10).
# Input:   `train`, `test`, `FUNNEL_SET`, `WRAPPER_SET`, `CANDIDATES`, `champ_bad`.
# Output:  `gbm_by_option` (model and PD on the test for each set), table `dec10`.

options = {"Filter: selection funnel": FUNNEL_SET, "Wrapper: backward elimination": WRAPPER_SET,
           "Embedded: GBM on all candidates": CANDIDATES}
fits, rows = {}, []
for name, cols in options.items():
    model = fit_gbm(train, cols)
    p = model.predict_proba(gbm_matrix(test, cols))[:, 1]
    fits[name] = (model, p)
    rows.append({"Method": name, "Features": len(cols), **score_metrics(test["default"], p),
                 "Approval at champion bad rate": approval_at_bad_rate(test["default"], p, champ_bad)})
dec10 = pd.DataFrame(rows).set_index("Method")
in_both = sorted(set(FUNNEL_SET) & set(WRAPPER_SET))
N.update({
    "auc_funnel": f"{dec10.iloc[0]['AUC']:.3f}", "auc_wrapper": f"{dec10.iloc[1]['AUC']:.3f}",
    "auc_all": f"{dec10.iloc[2]['AUC']:.3f}",
    "appr_funnel": f"{dec10.iloc[0]['Approval at champion bad rate']:.1%}",
    "appr_wrapper": f"{dec10.iloc[1]['Approval at champion bad rate']:.1%}",
    "appr_all": f"{dec10.iloc[2]['Approval at champion bad rate']:.1%}",
    "n_both": f"{len(in_both)}",
    "only_wrapper": ", ".join(f"`{c}`" for c in sorted(set(WRAPPER_SET) - set(FUNNEL_SET))) or "none",
})
fmt(dec10, {"AUC": "{:.3f}", "Gini": "{:.3f}", "KS": "{:.3f}", "Approval at champion bad rate": "{:.1%}"})

	Features	AUC	Gini	KS	Approval at champion bad rate
Method
Filter: selection funnel	28	0.691	0.381	0.277	86.5%
Wrapper: backward elimination	20	0.687	0.375	0.270	86.2%
Embedded: GBM on all candidates	65	0.694	0.387	0.281	86.8%

Decision. Use the columns of the selection funnel (the filter). Use the backward elimination as a check of the funnel.

Why. The three methods give almost the same result on the 2015 loans. The AUC is 0.691 for the funnel, 0.687 for the wrapper and 0.694 for all candidates. At the champion bad rate, they approve 86.5%, 86.2% and 86.8% of the applicants. Thus more columns, or a search with the model, add almost no power.

When the methods give the same power, the reason for each column decides. The funnel gives each removed column a written rule, and Appendix A shows it for each column. The wrapper gives only “the model did not need it”, and its set can change with the validation sample.

The two methods agree on 16 columns. The wrapper also keeps these columns that the funnel removes: `avg_cur_bal`, `bal_to_limit`, `inq_last_6mths`, `total_il_high_credit_limit`. They can help a model in combination with other columns. But the wrapper set approves 86.2% of the applicants, against 86.5% for the funnel.

What would change the decision. If the wrapper approves 1 point more applicants on a later vintage at the champion bad rate, use its set. Then write a reason for each of its columns.

# Purpose: Fix the final features. Give each column its bucket, rule and detail (dec-10, Appendix A).
# Input:   `funnel`, `FUNNEL_SET`, `weak`, `duplicate_of`, `unstable`, `fits`, `be_round`, `lc_dict`.
# Output:  `FEATURES`, `NUM_FEATURES`, `CAT_FEATURES`, `gbm_by_option`, `column_status`, table of the funnel.

FEATURES = list(FUNNEL_SET)
NUM_FEATURES, CAT_FEATURES = split_types(FEATURES)
# Step 4b and later use the key "B. Selection funnel" for the GBM on the final features.
gbm_by_option = {"B. Selection funnel": fits["Filter: selection funnel"],
                 "Backward elimination": fits["Wrapper: backward elimination"],
                 "All candidates": fits["Embedded: GBM on all candidates"]}

# Rules 9–11 and the features, for the columns of the file.
for cols, rule in [(weak, "9. Weak (IV < 0.01)"), (list(duplicate_of), "10. Duplicate of a stronger column"),
                   (unstable, "11. Unstable (PSI > 0.1)")]:
    funnel.loc[funnel["column"].isin(cols), "rule"] = rule
funnel.loc[funnel["column"].isin(FEATURES), "rule"] = "Feature"

# One row for each column of the file, then one row for each calculated feature.
REPLACED_BY = {raw: calc for calc, raws in CALCULATED.items() if calc in ("fico", "emp_years", "credit_hist_months")
               for raw in raws.split(", ") if raw != "issue_d"}
rule_of = funnel.set_index("column")["rule"].to_dict()
for c in CALCULATED:
    rule_of[c] = ("9. Weak (IV < 0.01)" if c in weak else "10. Duplicate of a stronger column" if c in duplicate_of
                  else "11. Unstable (PSI > 0.1)" if c in unstable else "Feature")


def status_detail(c):
    """Short text with the evidence for the status of one column."""
    rule = rule_of.get(c)
    if rule == "Feature" or (rule and rule.startswith("9.")):
        text = f"IV {iv_all[c]:.3f}"
    elif rule and rule.startswith("10."):
        text = f"Duplicate of {duplicate_of[c]} (IV {iv_all[c]:.3f})"
    elif rule and rule.startswith("11."):
        text = f"PSI {psi_col[c]:.2f}"
    elif rule and rule.startswith("8."):
        text = f"Missing up to {miss_by_year.loc[c].max():.0%} in 2013–2014"
    elif rule and rule.startswith("7."):
        text = f"Replaced by {REPLACED_BY[c]}"
    else:
        return ""
    if c in CANDIDATES:  # Result of the wrapper as a check
        text += "; wrapper keeps it" if c in WRAPPER_SET else f"; wrapper removes it in round {be_round[c]}"
    return text


column_status = pd.DataFrame({"column": list(lc_dict.index) + list(CALCULATED)})
column_status["rule"] = column_status["column"].map(rule_of).fillna("Candidate")
column_status["bucket"] = column_status["rule"].map(RULE_BUCKET).fillna(column_status["rule"])
column_status["detail"] = column_status["column"].map(status_detail)
column_status = column_status[["column", "bucket", "rule", "detail"]]
assert column_status["rule"].ne("Candidate").all(), "Each column must have a status."

N.update({"n_features": f"{len(FEATURES)}",
          # Same keys as the first version, for the text of later parts.
          "auc_b": N["auc_funnel"], "appr_b": N["appr_funnel"], "auc_c": N["auc_all"], "appr_c": N["appr_all"]})

# Table of the funnel: the columns that each rule removes, by bucket.
table = (column_status.groupby(["bucket", "rule"])["column"]
         .agg(Columns="size", Examples=lambda c: ", ".join(list(c)[:4])).reset_index())
# Order: buckets 1–3 with the rules by number, then the features.
table["_r"] = table["rule"].str.extract(r"^(\d+)\.")[0].astype(float).fillna(99)
table = (table.sort_values(["bucket", "_r"]).drop(columns="_r")
         .rename(columns={"bucket": "Bucket", "rule": "Rule"}).set_index(["Bucket", "Rule"]))
table

		Columns	Examples
Bucket	Rule
1. Not permitted	3. Known only after the issue date	39	collection_recovery_fee, last_credit_pull_d, l...
	4. Output of the Lending Club policy	5	funded_amnt, grade, installment, int_rate
	5. Joint applications only	16	annual_inc_joint, dti_joint, verification_stat...
	6. Fair lending risk	3	addr_state, emp_title, zip_code
2. No usable data	1. Not about the borrower	9	desc, funded_amnt_inv, id, initial_list_status
	2. Scope column	3	application_type, issue_d, term
	8. Not collected in the training years	14	all_util, il_util, inq_fi, inq_last_12m
3. No mathematical value	7. Replaced by a calculated column	4	earliest_cr_line, emp_length, fico_range_high,...
	9. Weak (IV < 0.01)	30	acc_now_delinq, chargeoff_within_12_mths, coll...
	10. Duplicate of a stronger column	7	avg_cur_bal, bc_util, inq_last_6mths, num_rev_...
Feature	Feature	28	acc_open_past_24mths, annual_inc, bc_open_to_b...

The funnel keeps 28 columns as features. The table gives the IV of each feature on the training data.

# Purpose: Show the IV of each final feature.
# Input:   `iv_all`, `FEATURES`, `ENGINEERED`.
# Output:  Table of IV values.

iv_table = iv_all.reindex(FEATURES).rename("IV").to_frame()
iv_table["Type"] = ["New ratio (Step 2d)" if c in ENGINEERED else "Calculated (Step 2d)" if c in CALCULATED
                    else "Raw column" for c in FEATURES]
fmt(iv_table, {"IV": "{:.3f}"})

	IV	Type
fico	0.131	Calculated (Step 2d)
new_acc_share	0.096	New ratio (Step 2d)
annual_inc	0.088	Raw column
bc_open_to_buy	0.078	Raw column
tot_hi_cred_lim	0.078	Raw column
total_bc_limit	0.072	Raw column
acc_open_past_24mths	0.057	Raw column
dti	0.052	Raw column
num_tl_op_past_12m	0.049	Raw column
mort_acc	0.047	Raw column
loan_to_income	0.043	New ratio (Step 2d)
home_ownership	0.042	Raw column
mo_sin_rcnt_tl	0.038	Raw column
percent_bc_gt_75	0.035	Raw column
mo_sin_old_rev_tl_op	0.034	Raw column
loan_to_rev_limit	0.034	New ratio (Step 2d)
mo_sin_rcnt_rev_tl_op	0.034	Raw column
mths_since_recent_inq	0.033	Raw column
mths_since_recent_bc	0.029	Raw column
mo_sin_old_il_acct	0.021	Raw column
num_actv_rev_tl	0.021	Raw column
purpose	0.020	Raw column
emp_years	0.020	Calculated (Step 2d)
revol_bal	0.019	Raw column
revol_util	0.018	Raw column
loan_amnt	0.010	Raw column
verification_status	0.010	Raw column
total_acc	0.010	Raw column

4b. Lending Club sub-grade

Step 4a selected the columns from the application and the credit bureau. One strong column is not in the list: the Lending Club sub-grade. Rule 4 of Step 2 removed it, because it is the output of the Lending Club policy. This section examines if that rule costs too much power.

Can the challenger use the Lending Club sub-grade?

Why this question matters. The sub-grade is the score of the champion. If the challenger uses it, the challenger can be stronger, but it is then not a separate policy. A lender must also explain each decline, and nobody outside Lending Club can explain the sub-grade.

Ways to do it. The axis is the role of the sub-grade in the challenger. There are three roles, and they cover all cases: an input, no role, or a first stage.

Role	What it is	Advantage	Main risk	Status
Input	The sub-grade is one feature of the challenger.	Higher AUC: the sub-grade contains data that we do not have.	Part of each decline comes from a model that we cannot see. No true reason code for that part.	Test it
No role	The challenger uses only the application and the credit bureau. The sub-grade is only the benchmark.	A true challenger. Each reason code comes from a feature that we can explain.	Lower AUC than the input role.	Test it
First stage	The champion decides first. The challenger then changes the decision only inside each grade (an overlay).	The reason codes of the champion stay.	It needs the champion in production. It is a different project.	Screened out here. See the last line of this block.

Evidence. The code trains a logistic regression with WoE twice, with and without the sub-grade. It compares both with the champion on the 2015 loans. Each score approves the same share of applicants as the champion. The table shows branch 1 (order of risk) and branch 4 (bad rate of the approved loans). The swap-set column gives the share of applicants where the score and the champion do not agree.

# Purpose: Compare the champion with two challengers: with and without the sub-grade (dec-03).
# Input:   `train`, `test`, `FEATURES`, `champ_ok`.
# Output:  `m_lr`, `pd_test_lr` (PD of the logistic regression), table `dec03`.

# No role: logistic regression with WoE, without the sub-grade.
m_lr = fit_woe_lr(train, FEATURES)
pd_test_lr = pd_woe_lr(m_lr, test)

# Input: the same model, plus the sub-grade as a number (A1 = 0, G5 = 34).
train_g = train.assign(sub_grade_num=subgrade_rank(train))
test_g = test.assign(sub_grade_num=subgrade_rank(test))
m_lr_g = fit_woe_lr(train_g, FEATURES + ["sub_grade_num"])
pd_test_lr_g = pd_woe_lr(m_lr_g, test_g)

# Each score approves the same share as the champion. Compare the bad rate and the swap-set.
rows = []
for name, risk in [("Champion: sub-grade only", champ_rank + rng.random(len(champ_rank)) * 0.01),
                   ("Challenger, sub-grade as an input", pd_test_lr_g),
                   ("Challenger, no sub-grade", pd_test_lr)]:
    ok = approve_top(risk, champ_ok.mean())
    rows.append({"Score (logistic regression with WoE)": name,
                 **score_metrics(test["default"], risk),
                 "Swap-set": np.mean(ok != champ_ok),
                 "Bad rate of approved": test["default"].to_numpy()[ok].mean()})
dec03 = pd.DataFrame(rows).set_index("Score (logistic regression with WoE)")
N.update({"auc_with_sg": f"{dec03.iloc[1]['AUC']:.3f}", "auc_no_sg": f"{dec03.iloc[2]['AUC']:.3f}",
          "auc_champ": f"{dec03.iloc[0]['AUC']:.3f}"})
fmt(dec03, {"AUC": "{:.3f}", "Gini": "{:.3f}", "KS": "{:.3f}",
            "Swap-set": "{:.1%}", "Bad rate of approved": "{:.2%}"})

	AUC	Gini	KS	Swap-set	Bad rate of approved
Score (logistic regression with WoE)
Champion: sub-grade only	0.679	0.357	0.264	0.0%	12.46%
Challenger, sub-grade as an input	0.699	0.397	0.290	12.5%	12.05%
Challenger, no sub-grade	0.681	0.362	0.262	17.3%	12.30%

Decision. The challenger does not use the sub-grade. The sub-grade is only the benchmark.

Why. With the sub-grade as an input, the AUC is 0.699, the highest of the three. But an adverse action notice must give the real reasons for a decline. With the sub-grade as an input, part of each decline comes from a source that we cannot explain (branch 5 fails).

Without the sub-grade, the logistic regression has an AUC of 0.681. The champion has 0.679. Thus a logistic regression on our features is at the level of the champion, without the score of Lending Club. Section 4c examines if a different model type finds more risk in the same features.

What would change the decision. If the champion stays in production and the challenger becomes an overlay on the champion score, use the first-stage role. The lender then has the reason codes of the champion.

4c. Model type

Section 4b showed that a logistic regression with WoE on our features reaches the level of the champion. A different model type can find more risk in the same features, but it can also be harder to explain. Thus the next question is: which type of model gives the score, and is the extra power worth the cost?

Which type of model gives the score?

Why this question matters. The model type sets how much risk the score finds. It also sets how easily the lender explains a decline, and how much work the model risk review needs. A model that is strong but that the lender cannot explain cannot go into production.

Ways to do it. We group all models for a yes/no target by how they make a decision. There are four groups, and each common model belongs to one of them. Each group then goes through three filters:

• Predictive power on tabular data. Our data is a table of numbers and categories, with missing values.
• Explainability for the law. The Equal Credit Opportunity Act (ECOA) and Regulation B require the true reasons for each decline. The Federal Reserve guidance SR 11-7 requires that a reviewer can understand and test the model.
• Deployment and control. The model must give the same score in the decision system each day. The team must also check it each month.

Group: how the model decides	Models	Predictive power	Explainability	Deployment	Result
Distance or geometry: an applicant is risky if it is near risky applicants or on the wrong side of a boundary	KNN, SVM	Medium. They need scaled numbers and no missing values. KNN is slow on 262,992 loans.	Poor. “You are near five past defaults” is not a reason that a borrower can act on.	KNN keeps all training loans in the system.	Screened out
Connectionist: many layers of weights learn new features	Neural networks	Strong on images and text. On tables, usually not better than tree ensembles.	Poor. A black box.	More effort to fit and to monitor.	Screened out
Trees and rules: the model splits the applicants by rules	Decision tree, random forest, GBM	GBM: usually the best on tabular data. One tree: weak. Random forest: strong, but usually below the GBM.	Medium. Reason codes need a separate method, for example SHAP values.	Standard in risk systems.	Keep the GBM for the test
Parametric: one equation with one weight for each feature	Naive Bayes, logistic regression, logistic regression with WoE	Naive Bayes assumes that the features are independent. That is not true here (income and loan amount move together). Logistic regression: good. With WoE, it also captures curves and missing values.	Strong. Each WoE bin gives points. The reasons come directly from the points.	Simple: a table of points (a scorecard).	Keep logistic regression with WoE

The shortlist has two models. The logistic regression with WoE is the industry scorecard: easy to explain, but it adds the effect of each feature without interactions. The GBM finds interactions, for example “short credit history and high card use”, but it needs a separate method for reason codes.

The plan before the evidence: use the logistic regression with WoE, unless the GBM adds 2 KS points or more.

Technical note. Why WoE helps the logistic regression.

A plain logistic regression assumes that the log-odds change in a straight line with each raw value. It also cannot use a missing value. WoE first cuts each feature into ten bins on the training data, with a separate bin for missing values. It then replaces each value with the log-odds of its bin. Thus a curve in the raw value becomes a straight line in WoE, and a missing value gets its own risk.

The GBM uses the same features with the settings of GBM_PARAMS:

• learning rate 0.05 and at most 500 trees,
• at most 31 leaves for each tree and at least 200 loans in each leaf,
• an early stop on 10% of the training loans.

Evidence. The code compares the champion and the two shortlisted models on the 2015 loans. Both models use the same features and the same training loans. The table gives branch 1 (AUC, Gini, KS) and branch 4. For branch 4, each model approves the most applicants at the champion bad rate. The last column gives the realized NPV of these approvals minus the realized NPV of the champion approvals.

# Purpose: Compare the scorecard (logistic regression with WoE) and the GBM with the champion (dec-04).
# Input:   `pd_test_lr`, `gbm_by_option`, `champ_rank`, `champ_ok`, `champ_bad`, `test['npv']`.
# Output:  `gbm` and `pd_test_gbm` (the benchmark), table `dec04`.

gbm, pd_test_gbm = gbm_by_option["B. Selection funnel"]  # The GBM of section 4a, on FEATURES
npv_all = test["npv"].to_numpy()


def gain_at_bad_rate(risk):
    """Return the approval share at the champion bad rate and its realized NPV against the champion ($M)."""
    share = approval_at_bad_rate(test["default"], risk, champ_bad)
    approved = np.argsort(risk, kind="stable")[: int(round(share * len(risk)))]
    return share, (npv_all[approved].sum() - npv_all[champ_ok].sum()) / 1e6


# Approval at the champion bad rate and its realized NPV: the business test of each score.
rows = [{"Model": "Champion: sub-grade only", **score_metrics(test["default"], champ_rank),
         "Approval at champion bad rate": champ_ok.mean(), "Realized NPV against champion ($M)": 0.0}]
for name, risk in [("Scorecard: logistic regression with WoE", pd_test_lr), ("GBM (benchmark)", pd_test_gbm)]:
    share, gain = gain_at_bad_rate(risk)
    rows.append({"Model": name, **score_metrics(test["default"], risk),
                 "Approval at champion bad rate": share, "Realized NPV against champion ($M)": gain})
dec04 = pd.DataFrame(rows).set_index("Model")
dec04.insert(4, "Gain over champion (points)", (dec04["Approval at champion bad rate"] - champ_ok.mean()) * 100)

# The plan before the evidence: use the GBM only if it adds 2 KS points or more over the scorecard.
lr_row, gbm_row = dec04.iloc[1], dec04.iloc[2]
ks_gain = (gbm_row["KS"] - lr_row["KS"]) * 100
N.update({
    "lr_appr": f"{lr_row['Approval at champion bad rate']:.1%}",
    "gbm_appr": f"{gbm_row['Approval at champion bad rate']:.1%}",
    "lr_gain": f"{lr_row['Gain over champion (points)']:+.1f}",
    "gbm_gain": f"{gbm_row['Gain over champion (points)']:+.1f}",
    "ks_gain": f"{ks_gain:.1f}", "ks_vs_plan": "more than" if ks_gain >= 2 else "less than",
    "ks_lr": f"{lr_row['KS']:.3f}", "ks_gbm": f"{gbm_row['KS']:.3f}",
    "lr_npv_gain": money(lr_row["Realized NPV against champion ($M)"]),
    "gbm_npv_gain": money(gbm_row["Realized NPV against champion ($M)"]),
    "gbm_extra_appr": f"{gbm_row['Gain over champion (points)'] - lr_row['Gain over champion (points)']:.1f}",
    "gbm_extra_npv": money(gbm_row["Realized NPV against champion ($M)"] - lr_row["Realized NPV against champion ($M)"]),
})
fmt(dec04, {"AUC": "{:.3f}", "Gini": "{:.3f}", "KS": "{:.3f}", "Approval at champion bad rate": "{:.1%}",
            "Gain over champion (points)": "{:+.1f}", "Realized NPV against champion ($M)": "{:+,.1f}"})

	AUC	Gini	KS	Approval at champion bad rate	Gain over champion (points)	Realized NPV against champion ($M)
Model
Champion: sub-grade only	0.679	0.357	0.262	84.5%	+0.0	+0.0
Scorecard: logistic regression with WoE	0.681	0.362	0.262	85.7%	+1.2	+7.1
GBM (benchmark)	0.691	0.381	0.277	86.5%	+2.0	+9.4

Decision. The challenger is the scorecard: the logistic regression with WoE. The GBM stays as a benchmark. It shows the highest power that a black-box model gets from the same features. We keep the rule that we set before the evidence.

Why. The GBM has a KS of 0.277. The scorecard has 0.262. Thus the GBM adds only 1.5 KS points, less than the limit of 2 points in the plan. The evidence confirms the plan.

The business result agrees. At the champion bad rate of 12.5%, the champion approves 84.5% of applicants. The scorecard approves 85.7% (+1.2 points) and adds $7.1M of realized NPV on the 2015 loans. The GBM approves 86.5% (+2.0 points) and adds $9.4M.

Thus the GBM gives 0.8 more points of approval and $2.3M more NPV than the scorecard. The lender accepts this smaller gain for three benefits:

• Each decline gets its reasons directly from the points of the scorecard (Step 5b).
• The model risk review is shorter, because a reviewer can read and test each weight.
• The scorecard is a table of points, so the decision system can apply it easily.

We set the rule of 2 KS points before we saw the evidence. A rule that changes after the evidence can follow the result that we want. Thus we keep the rule.

What would change the decision. Examine the GBM again in one of two conditions. A new vintage shows that the GBM adds 2 KS points or more. Or the business puts a higher value on the extra $2.3M than on the cost of a longer model risk review.

From here, the word “challenger” means the scorecard. The GBM appears again only as a benchmark. The next cell gives the challenger one name, so that Steps 5 to 7 do not depend on the model type.

# Purpose: Give the challenger one set of names for Steps 5 to 7 (dec-04).
# Input:   `m_lr`, `pd_test_lr`, `train`.
# Output:  `ch_name`, `ch_model`, `pd_test_ch` (PD on the 2015 loans), `pd_train_ch` (PD on the training loans).

ch_name = "Scorecard (logistic regression with WoE)"
ch_model = m_lr  # (WoE encoder, features, logistic regression)
pd_test_ch = pd_test_lr
pd_train_ch = pd_woe_lr(m_lr, train)
N["ch_name"] = ch_name

4d. Default rate: class imbalance and a change of level

The scorecard approves more applicants than the champion at the same bad rate. But Step 7 multiplies its PD by dollars to get the expected NPV. Thus the level of the PD must also be correct (branch 2). Two different problems can make the level wrong, and teams often mix them.

1. Class imbalance. Only about one loan in 8 is a default (13.2% in the training loans). Some teams then give more weight to the defaults, with a class weight or with synthetic defaults (SMOTE). These methods make the model behave as if defaults are more common than they are. Thus they push the PD up, far above the real default rate.
2. A change of level over time. The default rate went up from 13.2% in the training loans to 14.9% in 2015 (Step 3b). A model that learns from 2013–2014 gives a PD at the level of 2013–2014.

The second problem is the more dangerous one, because it hides in the money. The chain is:

Low default rate in training → PD too low for 2015 → expected NPV too high → a cutoff that approves loans with a negative NPV.

How do we handle the default rate in the training data?

Why this question matters. The PD enters the expected NPV in dollars. A PD that is too high declines good applicants. A PD that is too low approves loans that lose money.

Ways to do it. The axis is where the correction happens: in the training data, in the model, or after the model. These three places cover all methods.

Where	Method	Advantage	Main risk	Status
Training data	SMOTE (synthetic defaults) or fewer non-defaults	Balanced classes.	Synthetic rows. The PD becomes too high. Banks rarely use SMOTE for a PD model.	Screened out: the same effect as a class weight, with more risk
Model	Class weight: each default counts more in the fit	More weight to defaults.	The PD becomes too high. It needs a recalibration after the fit.	Test it
After the model	No change in the fit. Then move the level of the PD to the most recent default rate (recalibration).	The PD stays near the real rate. One constant to explain.	It needs recent loans with a known outcome.	Test it

A default rate of 13.2% is not a rare event. Rare events, for example 1% or less, are a different case.

Evidence. The code trains the same scorecard again with a class weight, on the same WoE features. For each score decile of the 2015 loans, it compares the mean PD with the actual default rate (branch 2).

# Purpose: Compare the calibration of the scorecard with and without a class weight (dec-05).
# Input:   `train`, `test`, `pd_test_ch`, `FEATURES`.
# Output:  Calibration chart and table `dec05`.

# The same scorecard with a class weight ("balanced": the defaults get the same total weight as the non-defaults).
m_lr_w = fit_woe_lr(train, FEATURES, class_weight="balanced")
pd_test_w = pd_woe_lr(m_lr_w, test)

# Mean PD and actual default rate for each score decile.
calib = pd.DataFrame({
    "decile": pd.qcut(pd_test_ch, 10, labels=False) + 1,
    "Actual": test["default"].to_numpy(),
    "No change": pd_test_ch,
    "Class weight": pd_test_w,
}).groupby("decile").mean()

N.update({"pd_mean": f"{pd_test_ch.mean():.1%}", "pd_mean_w": f"{pd_test_w.mean():.1%}",
          "bad_2015": f"{test['default'].mean():.1%}", "bad_train": f"{train['default'].mean():.1%}"})

fig, ax = plt.subplots(figsize=(8, 4))
styles = {"Actual": (cs.GRAY, "o", "Actual"), "No change": (cs.NAVY, "s", "Scorecard"),
          "Class weight": (cs.BLUE, "^", "Scorecard, class weight")}
for col, (color, marker, label) in styles.items():
    ax.plot(calib.index, calib[col], marker=marker, color=color, lw=1.5 if col == "Actual" else 2)
    # "Actual" and "Scorecard" end near the same point. A vertical offset keeps the labels apart.
    dy = {"Actual": 6, "No change": -6, "Class weight": 0}[col]
    cs.label_end(ax, calib.index[-1], calib[col].iloc[-1], label, color=color, dy=dy, bold=(col == "Class weight"))
ax.set_xlabel("Score decile of the scorecard (1 = lowest risk)")
ax.set_ylabel("Default rate or mean PD")
ax.set_xticks(calib.index)
ax.set_xlim(0.5, 12.8)
cs.percent_axis(ax)
cs.action_title(fig, f"A class weight puts the mean PD at {pd_test_w.mean():.0%} against "
                     f"{test['default'].mean():.1%} actual",
                "Mean PD of the scorecard and actual default rate by score decile, 2015 loans")
plt.show()

dec05 = pd.DataFrame({
    "Option": ["No change", "Class weight"],
    "Mean predicted PD": [pd_test_ch.mean(), pd_test_w.mean()],
    "Actual default rate": [test["default"].mean()] * 2,
    "AUC": [roc_auc_score(test["default"], pd_test_ch), roc_auc_score(test["default"], pd_test_w)],
}).set_index("Option")
fmt(dec05, {"Mean predicted PD": "{:.1%}", "Actual default rate": "{:.1%}", "AUC": "{:.3f}"})

	Mean predicted PD	Actual default rate	AUC
Option
No change	13.6%	14.9%	0.681
Class weight	47.3%	14.9%	0.680

How to read: each point is one tenth of the 2015 loans, in order of the scorecard. The gray line is the actual default rate. The navy line is the scorecard without a change. The blue line is the scorecard with a class weight.

So what: the class weight keeps the order but puts the PD far too high. Without a change, the PD is close but a little too low in each decile.

Decision. No class weight and no SMOTE. In Step 5a, move the level of the PD to the default rate of the most recent loans (recalibration).

Why. Both options give almost the same AUC, so they put applicants in almost the same order (branch 1). But the class weight gives a mean PD of 47.3% against an actual default rate of 14.9% (branch 2 fails). The expected NPV in Step 7 needs a correct PD.

Without a change, the mean PD is 13.6%. This is the level of the training loans, not of 2015. The order is correct, but the level is too low. One constant on the log-odds scale corrects the level and keeps the order. Step 5a applies it.

What would change the decision. If the default rate is 1% or less, the model can miss the defaults. Then use a class weight and recalibrate after the fit.

Business note. The challenger is a scorecard on the application and credit bureau data, without the Lending Club sub-grade. At the champion bad rate, it approves 85.7% of applicants against 84.5% for the champion. The GBM approves 86.5%, but the lender keeps the scorecard for its clear reasons and its shorter review. The PD of the scorecard is too low for 2015, so Step 5 corrects the level and gives the reasons for each decline.

Step 5. Make predictions

Step 4 selected the scorecard as the challenger, with the GBM as a benchmark. It left two open points. The PD of the scorecard is at the level of 2013–2014, too low for 2015. Each decline also needs its reasons in plain English. This step closes both points: what PD and what reasons does each 2015 applicant get?

Purpose. This step gives each 2015 applicant a PD at the correct level and the reasons for a decline. It supports this decision: the score and the reason codes for the champion/challenger test.

5a. Recalibration of the PD

Section 4d found that the mean PD of the scorecard is 13.6%, but the 2015 loans default at 14.9%. If we do not correct this, the expected NPV in Step 7 is too high, and the cutoff approves loans that lose money.

The correction uses one constant on the log-odds scale. The code adds the constant to the log-odds of each PD. The constant makes the mean PD equal to the default rate of the loans from January to June 2015. The loans from July to December 2015 then check the result. The recalibration does not change the order of the applicants. Thus the AUC, the KS and the swap-set do not change.

Technical note. PD after the recalibration = 1 / (1 + exp(−(ln(PD / (1 − PD)) + s))). The code finds the constant s with a root search (Brent’s method) between −3 and 3. On the log-odds scale, one constant keeps the order of the applicants and keeps each PD between 0 and 1.

In production, the lender uses the most recent loans with a known outcome. An early sign of default, for example a payment 60 days late, can also help. Here the first half of 2015 has this role. The second half of 2015 is an honest check, because the constant never sees it.

# Purpose: Move the PD to the recent default rate with one constant on the log-odds scale (dec-05).
# Input:   `pd_test_ch` (PD of the scorecard), `test`, `y_test`, `logit()`, `expit()` (case01_helpers.py).
# Output:  `pd_cal` (the PD of the scorecard after the recalibration), `shift` (the constant), `h1`.

# Calibrate on January–June 2015. Check on July–December 2015.
h1 = (test["issue_date"].dt.month <= 6).to_numpy()
# Find the constant that makes the mean PD equal to the actual default rate.
shift = brentq(lambda s: expit(logit(pd_test_ch[h1]) + s).mean() - y_test[h1].mean(), -3, 3)
pd_cal = expit(logit(pd_test_ch) + shift)

N.update({"pd_h2_before": f"{pd_test_ch[~h1].mean():.1%}", "pd_h2_after": f"{pd_cal[~h1].mean():.1%}",
          "bad_h2": f"{y_test[~h1].mean():.1%}", "shift": f"{shift:+.3f}"})
fmt(pd.DataFrame({
    "Sample": ["January to June 2015 (calibration)", "July to December 2015 (check)"],
    "Actual default rate": [y_test[h1].mean(), y_test[~h1].mean()],
    "Mean PD before": [pd_test_ch[h1].mean(), pd_test_ch[~h1].mean()],
    "Mean PD after": [pd_cal[h1].mean(), pd_cal[~h1].mean()],
}).set_index("Sample"), {"Actual default rate": "{:.1%}", "Mean PD before": "{:.1%}",
                         "Mean PD after": "{:.1%}"})

	Actual default rate	Mean PD before	Mean PD after
Sample
January to June 2015 (calibration)	15.1%	13.9%	15.1%
July to December 2015 (check)	14.7%	13.4%	14.6%

The constant is +0.102 on the log-odds scale. On the check sample, the mean PD goes from 13.4% before the recalibration to 14.6% after it. The actual default rate is 14.7%.

5b. Reason codes

A declined applicant must get the main reasons for the decline (ECOA and Regulation B). A scorecard gives the reasons directly from its points. This is the standard method for a scorecard, and model risk reviewers know it. It has three steps:

1. Typical applicant. Take the applicants that the scorecard approves at the champion approval rate. The typical value of each feature is the median for a number and the most common value for a category. The scorecard changes each typical value into its WoE.
2. Points for each feature. The points of a feature are its weight times a difference: the WoE of the applicant minus the WoE of the typical applicant. Positive points increase the log-odds of default, so they increase the PD.
3. Reasons. The three features with the most points are the reason codes. The table also gives the PD decrease if the feature takes the typical WoE.

Technical note. The scorecard gives log-odds of default = b₀ + Σ bⱼ × WoEⱼ. The points of feature j for an applicant are bⱼ × (WoEⱼ of the applicant − WoEⱼ of the typical approved applicant). The sum of the points is the difference in log-odds between the applicant and the typical applicant.

The PD goes up when the log-odds go up. Thus the order of the features by points is the same as the order by PD decrease. The reasons are exact for this model: no approximation, no sample and no random seed.

A reason code is a column name, for example bc_open_to_buy. A borrower cannot read it. Thus each feature has a plain-English meaning in the dictionary REASON_TEXT (in case01_helpers.py). The direction tells if the value of the applicant is above or below the typical value. The direction then selects the text for the borrower, for example “Little unused credit on bank cards”. If a feature is not in the dictionary, the column name becomes the text.

The table below gives the meaning of each feature of the challenger.

# Purpose: Show the plain-English meaning of each feature that can become a reason code.
# Input:   `FEATURES`, `REASON_TEXT` (case01_helpers.py).
# Output:  Table: feature, meaning, text if the value is high, text if the value is low.

reason_dict = pd.DataFrame(
    [(c, *REASON_TEXT.get(c, (c, f"High value of {c}", f"Low value of {c}"))) for c in FEATURES],
    columns=["Feature", "Meaning", "Text if the value is high", "Text if the value is low"],
).set_index("Feature")
N["n_no_text"] = f"{sum(c not in REASON_TEXT for c in FEATURES)}"
reason_dict

	Meaning	Text if the value is high	Text if the value is low
Feature
fico	FICO score	High FICO score	Low FICO score
new_acc_share	Accounts opened in the last 24 months as a sha...	Many of the accounts are new	Few of the accounts are new
annual_inc	Annual income	High income	Low income
bc_open_to_buy	Unused credit on bank cards	Much unused credit on bank cards	Little unused credit on bank cards
tot_hi_cred_lim	Total credit limit on all accounts	High total credit limit	Low total credit limit
total_bc_limit	Total credit limit on bank cards	High credit limit on bank cards	Low credit limit on bank cards
acc_open_past_24mths	Accounts opened in the last 24 months	Many accounts opened recently	Few accounts opened recently
dti	Monthly debt payments as a share of income	High debt payments compared with income	Low debt payments compared with income
num_tl_op_past_12m	Accounts opened in the last 12 months	Many accounts opened in the last year	Few accounts opened in the last year
mort_acc	Number of mortgage accounts	Many mortgage accounts	No mortgage account
loan_to_income	Loan amount as a share of income	Loan amount is high compared with income	Loan amount is low compared with income
home_ownership	Home ownership status	Home ownership status	Home ownership status
mo_sin_rcnt_tl	Months since the most recent new account	No recent new account	Recent new account
percent_bc_gt_75	Share of bank cards above 75% of the limit	Many bank cards near the limit	Few bank cards near the limit
mo_sin_old_rev_tl_op	Age of the oldest revolving account	Old revolving accounts	No old revolving accounts
loan_to_rev_limit	Loan amount as a share of the total revolving ...	Loan amount is high compared with the credit l...	Loan amount is low compared with the credit li...
mo_sin_rcnt_rev_tl_op	Months since the most recent new revolving acc...	No recent new revolving account	Recent new revolving account
mths_since_recent_inq	Months since the most recent credit inquiry	No recent credit inquiry	Recent credit inquiry
mths_since_recent_bc	Months since the most recent new bank card	No recent new bank card	Recent new bank card
mo_sin_old_il_acct	Age of the oldest installment account	Old installment accounts	No old installment accounts
num_actv_rev_tl	Number of active revolving accounts	Many active revolving accounts	Few active revolving accounts
purpose	Purpose of the loan	Purpose of the loan	Purpose of the loan
emp_years	Years with the current employer	Long time with the current employer	Short time with the current employer
revol_bal	Balance on revolving accounts	High balance on revolving accounts	Low balance on revolving accounts
revol_util	Share of the revolving credit limit in use	High use of revolving credit	Low use of revolving credit
loan_amnt	Amount of the loan	Large loan amount	Small loan amount
verification_status	Income verification status	Income verification status	Income verification status
total_acc	Number of accounts in the credit file	Many accounts in the credit file	Few accounts in the credit file

# Purpose: Find the three features with the most points for one declined applicant of the scorecard.
# Input:   `ch_model`, `test`, `pd_test_ch`, `champ_ok`, `FEATURES`, `CAT_FEATURES`.
# Output:  `typical`, `woe_typ`, table of reason codes and the text for the borrower.

# The typical approved applicant: median of each number, most common category. Then its WoE.
ch_ok = approve_top(pd_test_ch, champ_ok.mean())
typical = test.loc[ch_ok, NUM_FEATURES].median()
for c in CAT_FEATURES:
    typical[c] = test.loc[ch_ok, c].mode().iloc[0]
woe_typ = typical_woe(ch_model, typical)

# One declined applicant: points and PD decrease of each feature.
example = test[~ch_ok].iloc[[0]]
pd_example = pd_woe_lr(ch_model, example)[0]
points = scorecard_points(ch_model, example, woe_typ).iloc[0]
effect = scorecard_reason_effects(ch_model, example, woe_typ).iloc[0]
top3 = points.sort_values(ascending=False).head(3)

rows = []
for c in top3.index:
    value, typ = example[c].iloc[0], typical[c]
    meaning, direction, text = describe_reason(c, value, typ, categorical=c in CAT_FEATURES)
    show = lambda v: v if isinstance(v, str) or pd.isna(v) else f"{v:,.4g}"  # Short text for a number
    rows.append({"Reason code": c, "Meaning": meaning, "Applicant value": show(value),
                 "Typical approved value": show(typ),
                 "Direction": direction, "Points (log-odds)": f"{top3[c]:+.3f}",
                 "PD decrease if typical": f"{effect[c]:.1%}", "Text for the borrower": text})
reasons = pd.DataFrame(rows, index=[1, 2, 3])

# The adverse action text: the three reasons in plain English.
notice = "; ".join(f"({i}) {t}" for i, t in zip(reasons.index, reasons["Text for the borrower"]))
N.update({"pd_example": f"{pd_example:.1%}", "notice": notice})
print(f"PD of the applicant after the recalibration: {expit(logit(pd_example) + shift):.1%}.")
print(f"Text for the borrower: We cannot approve your application now. The main reasons are: {notice}.")
reasons

PD of the applicant after the recalibration: 35.6%.
Text for the borrower: We cannot approve your application now. The main reasons are: (1) Many active revolving accounts; (2) Many of the accounts are new; (3) High debt payments compared with income.

	Reason code	Meaning	Applicant value	Typical approved value	Direction	Points (log-odds)	PD decrease if typical	Text for the borrower
1	num_actv_rev_tl	Number of active revolving accounts	17	5	Higher than typical	+0.217	4.6%	Many active revolving accounts
2	new_acc_share	Accounts opened in the last 24 months as a sha...	0.8947	0.1591	Higher than typical	+0.205	4.4%	Many of the accounts are new
3	dti	Monthly debt payments as a share of income	25.49	16.82	Higher than typical	+0.163	3.5%	High debt payments compared with income

How to read: each row is one reason, from the strongest. The “Meaning” column says what the feature measures. The “Points” column gives the increase of the log-odds of default. The “Text for the borrower” column is the sentence for the adverse action notice.

So what: the lender can send this text: “We cannot approve your application now. The main reasons are: (1) Many active revolving accounts; (2) Many of the accounts are new; (3) High debt payments compared with income.”

The points come from the weights of the model, not from an approximation. Thus a model risk reviewer can repeat each reason with a calculator.

Business note. Each 2015 applicant now has a PD at the 2015 level and, if declined, three reasons in plain English. Step 6 checks if the challenger passes all five branches of the evaluation framework.

Step 6. Evaluate

Step 5 gave a PD at the correct level and reasons for each decline. Now we must judge the challenger as a whole. This step uses the five branches of the evaluation framework of Step 3a, one at a time. We keep the rules of Step 3a, also when a result is near a limit. If the challenger fails a branch, this step says why and what it means for the test.

Purpose. This step shows if the score is good enough for a test, in model metrics and in dollars. It supports this decision: approve the challenger score for the champion/challenger test.

6a. Branch 1: order of risk

The question: does the challenger put risky applicants above safe applicants better than the champion? The rule of section 4c also applies: if the GBM benchmark is less than 2 KS points above the scorecard, the scorecard is enough. Section 3a gives the full rule. The table compares the scorecard with the champion and with the GBM on the 2015 loans. Section 3a explains the AUC, the Gini and the KS.

# Purpose: Measure branch 1 (order of risk) on the 2015 loans.
# Input:   `champ_rank`, `pd_cal`, `pd_test_gbm`, `y_test`.
# Output:  Table `eval_order`.

eval_order = pd.DataFrame([
    {"Score": "Champion: sub-grade", **score_metrics(y_test, champ_rank)},
    {"Score": "Challenger: scorecard after the recalibration", **score_metrics(y_test, pd_cal)},
    {"Score": "Benchmark: GBM", **score_metrics(y_test, pd_test_gbm)},
]).set_index("Score")
m_ch, m_champ = eval_order.iloc[1], eval_order.iloc[0]
N.update({"ks_ch": f"{m_ch['KS']:.3f}", "ks_champ": f"{m_champ['KS']:.3f}",
          "auc_ch": f"{m_ch['AUC']:.3f}", "auc_champ6": f"{m_champ['AUC']:.3f}",
          "gini_ch": f"{m_ch['Gini']:.3f}", "gini_champ": f"{m_champ['Gini']:.3f}"})
fmt(eval_order, {"AUC": "{:.3f}", "Gini": "{:.3f}", "KS": "{:.3f}"})

	AUC	Gini	KS
Score
Champion: sub-grade	0.679	0.357	0.262
Challenger: scorecard after the recalibration	0.681	0.362	0.262
Benchmark: GBM	0.691	0.381	0.277

The challenger has an AUC of 0.681 against 0.679 for the champion, and a Gini of 0.362 against 0.357. The gain is small. On the KS, the two scores are almost equal: 0.262 against 0.262.

The rule of branch 1 is a KS above the champion. The scorecard does not pass it: on the whole population, it ranks risk about as well as the sub-grades. We do not change the rule after the evidence. The value of the scorecard is in a smaller region: the applicants near the cutoff. Branch 4 measures this region directly.

The GBM benchmark is 1.5 KS points above the scorecard, less than the limit of 2 points. The recalibration does not change the AUC or the KS of the scorecard.

6b. Branch 2: level

The question: does the mean PD match the actual default rate, on loans that the recalibration did not use? The check uses the loans of July to December 2015. The chart compares the PD before and after the recalibration with the actual default rate, for each score decile.

# Purpose: Measure branch 2 (level) on the check sample (July–December 2015).
# Input:   `pd_test_ch`, `pd_cal`, `y_test`, `h1`.
# Output:  Calibration chart. `level_gap` (mean PD after the recalibration minus the actual rate).

chk = ~h1
level = pd.DataFrame({
    "decile": pd.qcut(pd_cal[chk], 10, labels=False) + 1,
    "Actual": y_test[chk], "Before": pd_test_ch[chk], "After": pd_cal[chk],
}).groupby("decile").mean()
level_gap = pd_cal[chk].mean() - y_test[chk].mean()

fig, ax = plt.subplots(figsize=(8, 4))
for col, color, marker in [("Actual", cs.GRAY, "o"), ("Before", cs.LIGHT_GRAY, "s"), ("After", cs.NAVY, "^")]:
    ax.plot(level.index, level[col], marker=marker, color=color, lw=2 if col == "After" else 1.5)
    label = {"Actual": "Actual", "Before": "Scorecard, before", "After": "Scorecard, after"}[col]
    # The three lines end near the same point. A vertical offset keeps the labels apart.
    dy = {"Actual": 10, "Before": -10, "After": 0}[col]
    cs.label_end(ax, level.index[-1], level[col].iloc[-1], label, color=color, dy=dy, bold=(col == "After"))
ax.set_xlabel("Score decile of the scorecard (1 = lowest risk)")
ax.set_ylabel("Default rate or mean PD")
ax.set_xticks(level.index)
ax.set_xlim(0.5, 12.3)
cs.percent_axis(ax)
cs.action_title(fig, f"After the recalibration, the mean PD is {pd_cal[chk].mean():.1%} "
                     f"against {y_test[chk].mean():.1%} actual",
                "Mean PD and actual default rate by score decile, loans of July to December 2015")
plt.show()

How to read: each point is one tenth of the check loans, in order of the scorecard. The gray line is the actual default rate. The light gray line is the PD of the scorecard before the recalibration, and the navy line is the PD after it.

So what: after the recalibration, the PD follows the actual default rate. The expected NPV in Step 7 can use it.

6c. Branch 3: stability

The question: do the 2015 applicants look like the training applicants? If not, the score can lose power soon after the start of the test. The code calculates the PSI of the challenger score from the training loans to the 2015 loans. Rule 11 of section 4a already removed each feature with a PSI of more than 0.1. The table repeats the largest PSI of the final features.

# Purpose: Measure branch 3 (stability): PSI of the score and of each feature.
# Input:   `pd_train_ch`, `pd_cal`, `shift`, `psi_col` (section 4a), `FEATURES`.
# Output:  `psi_score`, table of the PSI.

# PD of the training loans, with the same recalibration, for the PSI.
psi_score = psi(expit(logit(pd_train_ch) + shift), pd_cal)
psi_feat = psi_col.reindex(FEATURES).dropna()
N.update({"psi_score": f"{psi_score:.3f}", "psi_feat_max": f"{psi_feat.max():.3f}",
          "psi_feat_top": psi_feat.idxmax()})
fmt(pd.DataFrame({
    "PSI": [psi_score, psi_feat.max(), psi_feat.median()],
    "Rule": ["Less than 0.1"] * 3,
}, index=["Challenger score", f"Largest feature PSI ({psi_feat.idxmax()})", "Median feature PSI"]),
    {"PSI": "{:.3f}"})

	PSI	Rule
Challenger score	0.005	Less than 0.1
Largest feature PSI (verification_status)	0.057	Less than 0.1
Median feature PSI	0.006	Less than 0.1

The PSI of the score is 0.005. The largest feature PSI is 0.057 (verification_status). Both are less than 0.1, so the population is stable. Section 3b showed that the level of the default rate changed. The recalibration in Step 5a corrects that change.

6d. Branch 4: business value

The question: at the same risk, does the challenger approve more applicants and add NPV? Two tests answer it.

• Approval at the champion bad rate. How many applicants can the challenger approve, at the bad rate of the champion?
• Swap-set. At the same number of approvals, which loans does the challenger change, and are the new loans better?

# Purpose: Measure the approval of the challenger at the champion bad rate.
# Input:   `pd_cal`, `y_test`, `champ_bad`, `champ_ok`.
# Output:  `appr_ch` (approval share of the challenger).

appr_ch = approval_at_bad_rate(y_test, pd_cal, champ_bad)
N.update({"appr_ch": f"{appr_ch:.1%}", "appr_gain": f"{(appr_ch - champ_ok.mean()) * 100:+.1f}"})
print(f"At the champion bad rate of {champ_bad:.1%}: the champion approves {champ_ok.mean():.1%}, "
      f"the challenger approves {appr_ch:.1%}.")

At the champion bad rate of 12.5%: the champion approves 84.5%, the challenger approves 85.7%.

At the champion bad rate, the challenger approves 85.7% of applicants, against 84.5% for the champion (+1.2 points).

How the swap-set works. The test gives both policies the same number of approvals: the champion approval share of 84.5%. Each 2015 applicant then falls into one of four groups:

Group	Champion	Challenger	Meaning
Both approve	Approve	Approve	No change.
Swap-in	Decline	Approve	New approvals of the challenger.
Swap-out	Approve	Decline	New declines of the challenger.
Both decline	Decline	Decline	No change.

Both policies approve the same number of applicants. Thus the swap-in group and the swap-out group have the same size. Only these two groups change the result.

The champion declines the sub-grades D1 and worse. Lending Club booked all these loans, so we know the real outcome of each swap-in loan. This is a form of reject inference with real outcomes. The limit: the data has no applicant that Lending Club declined. Step 7 examines this limit.

# Purpose: Compare the champion and the challenger at the same approval rate (swap-set).
# Input:   `pd_cal`, `champ_ok`, `test`, `y_test`.
# Output:  Table `swap` with four groups. `npv_test` (realized NPV of each 2015 loan), `chall_ok`.

npv_test = test["npv"].to_numpy()
# The challenger approves the same number of applicants as the champion.
chall_ok = approve_top(pd_cal, champ_ok.mean())
# Four groups. The swap-in and swap-out groups have the same size.
groups = {
    "Both approve": champ_ok & chall_ok,
    "Swap-in (challenger approves, champion declines)": chall_ok & ~champ_ok,
    "Swap-out (champion approves, challenger declines)": champ_ok & ~chall_ok,
    "Both decline": ~champ_ok & ~chall_ok,
}
swap = pd.DataFrame([{
    "Group": name, "Loans": m.sum(), "Default rate": y_test[m].mean(),
    "Interest rate": test["int_rate"].to_numpy()[m].mean() / 100,
    "Realized NPV for each loan": npv_test[m].mean(),
    "Total realized NPV ($M)": npv_test[m].sum() / 1e6,
} for name, m in groups.items()]).set_index("Group")

s_in, s_out = swap.iloc[1], swap.iloc[2]
swap_gain = s_in["Total realized NPV ($M)"] - s_out["Total realized NPV ($M)"]
N.update({
    "swap_n": f"{s_in['Loans']:,.0f}", "swap_share": f"{s_in['Loans'] / len(test):.1%}",
    "swap_in_bad": f"{s_in['Default rate']:.1%}", "swap_out_bad": f"{s_out['Default rate']:.1%}",
    "swap_in_rate": f"{s_in['Interest rate']:.1%}", "swap_out_rate": f"{s_out['Interest rate']:.1%}",
    "swap_gain": money(swap_gain),
})
fmt(swap, {"Loans": "{:,.0f}", "Default rate": "{:.1%}", "Interest rate": "{:.1%}",
           "Realized NPV for each loan": lambda v: money(v, "", 0), "Total realized NPV ($M)": "{:,.1f}"})

	Loans	Default rate	Interest rate	Realized NPV for each loan	Total realized NPV ($M)
Group
Both approve	214,716	11.0%	9.9%	−$13	-2.9
Swap-in (challenger approves, champion declines)	24,426	23.6%	17.1%	−$153	-3.7
Swap-out (champion approves, challenger declines)	24,426	25.1%	12.7%	−$493	-12.0
Both decline	19,280	34.2%	17.8%	−$741	-14.3

The two policies do not agree on 24,426 loans in each group (8.6% of the 2015 loans). The swap-in loans have a default rate of 23.6%. The swap-out loans have a default rate of 25.1%. With the same number of approvals, the challenger adds $8.3M of realized NPV.

The swap-in loans also have a higher interest rate (17.1% against 12.7%). The challenger finds applicants in lower grades with less risk than their grade shows. Lending Club gave these applicants a high interest rate.

The champion is an assumption (as-04). The table below repeats the approval test for three champion approval rates: 80%, 85% and 90%. For each one, the challenger must reach the bad rate of that champion.

# Purpose: Repeat the approval gain for three champion approval rates (as-04).
# Input:   `champ_rank`, `pd_cal`, `y_test`.
# Output:  Table `sens` of the gain for each champion.

rows = []
# For each champion: its bad rate, then the challenger approval at that bad rate.
for share in (0.80, 0.85, 0.90):
    cut = np.sort(champ_rank)[int(share * len(champ_rank))]
    ok = champ_rank < cut
    bad = y_test[ok].mean()
    rows.append({"Champion approval target": share, "Champion approval": ok.mean(),
                 "Champion bad rate": bad,
                 "Challenger approval at the same bad rate": approval_at_bad_rate(y_test, pd_cal, bad)})
sens = pd.DataFrame(rows).set_index("Champion approval target")
sens["Gain (points)"] = (sens["Challenger approval at the same bad rate"] - sens["Champion approval"]) * 100
sens.index = [f"{s:.0%}" for s in sens.index]
N["sens_min_gain"] = f"{sens['Gain (points)'].min():+.1f}"
fmt(sens, {"Champion approval": "{:.1%}", "Champion bad rate": "{:.1%}",
           "Challenger approval at the same bad rate": "{:.1%}", "Gain (points)": "{:+.1f}"})

	Champion approval	Champion bad rate	Challenger approval at the same bad rate	Gain (points)
80%	75.5%	11.3%	76.8%	+1.2
85%	84.5%	12.5%	85.7%	+1.2
90%	88.0%	12.9%	88.7%	+0.7

The smallest gain of the three champions is +0.7 points. Thus the result does not depend on the exact champion assumption.

6e. Branch 5: explainability

The question: can the lender give true reasons for each decline, not only for the example of Step 5b? The code repeats the points method on a random sample of 1,000 declined applicants. For each applicant, it counts the features with positive points. It also checks that each of the three reasons has a plain-English text.

# Purpose: Measure branch 5: do the declined applicants get three reasons with a plain-English text?
# Input:   `ch_model`, `test`, `chall_ok`, `woe_typ`, `REASON_TEXT`.
# Output:  `reason_ok` (share with three valid reasons), table of the most common first reasons.

# A random sample of 1,000 declined applicants (fixed seed).
declined = test[~chall_ok].sample(1000, random_state=SEED)
eff = scorecard_points(ch_model, declined, woe_typ)
# For each applicant: the three strongest reasons. A valid reason increases the PD and has a text.
top = eff.apply(lambda r: list(r.sort_values(ascending=False).index[:3]), axis=1)
n_up = (eff > 0).sum(axis=1)
has_text = top.apply(lambda cols: all(c in REASON_TEXT for c in cols))
reason_ok = ((n_up >= 3) & has_text).mean()
N.update({"reason_ok": f"{reason_ok:.0%}"})

# The most common first reason in the sample.
first = top.str[0].value_counts(normalize=True).head(5)
N["first_reasons"] = ", ".join(REASON_TEXT.get(c, (c,))[0].lower() for c in first.index[:3])
fmt(pd.DataFrame({"Meaning": [REASON_TEXT.get(c, (c,))[0] for c in first.index],
                  "Share of declines": first.to_numpy()}, index=first.index.rename("First reason")),
    {"Share of declines": "{:.0%}"})

	Meaning	Share of declines
First reason
loan_to_income	Loan amount as a share of income	26%
dti	Monthly debt payments as a share of income	20%
loan_to_rev_limit	Loan amount as a share of the total revolving ...	17%
new_acc_share	Accounts opened in the last 24 months as a sha...	13%
emp_years	Years with the current employer	10%

100% of the sampled declines have three reasons with positive points, each with a plain-English text. The table shows the most common first reasons. The three most common are: loan amount as a share of income, monthly debt payments as a share of income, loan amount as a share of the total revolving credit limit. A credit officer can check that each one makes sense for a decline.

6f. Summary of the evaluation

The table collects one result for each branch.

# Purpose: Collect the result of the challenger for each branch of the evaluation framework.
# Input:   Results of sections 6a–6e, `ks_gain` (section 4c).
# Output:  Table `evaluation`.

evaluation = pd.DataFrame([
    ("1. Order of risk", "KS: challenger / champion; KS of the GBM benchmark minus the challenger",
     f"{m_ch['KS']:.3f} / {m_champ['KS']:.3f}; {ks_gain:+.1f} points",
     "KS above the champion; GBM less than 2 KS points above (the scorecard is enough)",
     round(m_ch["KS"], 3) > round(m_champ["KS"], 3) and ks_gain < 2),
    ("2. Level", "Mean PD minus actual rate, July–December 2015", f"{level_gap * 100:+.1f} points",
     "Less than 1 point", abs(level_gap) < 0.01),
    ("3. Stability", "PSI of the score; largest feature PSI", f"{psi_score:.3f}; {psi_feat.max():.3f}",
     "Less than 0.1", psi_score < 0.1 and psi_feat.max() < 0.1),
    ("4. Business value", "Approval at the champion bad rate; swap-set NPV",
     f"{appr_ch:.1%} against {champ_ok.mean():.1%}; {swap_gain:+.1f} $M",
     "More approvals and a positive NPV", appr_ch > champ_ok.mean() and swap_gain > 0),
    ("5. Explainability", "Declines with three reasons in plain English", f"{reason_ok:.0%}",
     "95% or more", reason_ok >= 0.95),
], columns=["Branch", "Metric", "Result", "Rule", "Pass"]).set_index("Branch")
evaluation["Pass"] = evaluation["Pass"].map({True: "Yes", False: "No"})
N["n_pass"] = f"{(evaluation['Pass'] == 'Yes').sum()}"
evaluation

	Metric	Result	Rule	Pass
Branch
1. Order of risk	KS: challenger / champion; KS of the GBM bench...	0.262 / 0.262; +1.5 points	KS above the champion; GBM less than 2 KS poin...	No
2. Level	Mean PD minus actual rate, July–December 2015	-0.2 points	Less than 1 point	Yes
3. Stability	PSI of the score; largest feature PSI	0.005; 0.057	Less than 0.1	Yes
4. Business value	Approval at the champion bad rate; swap-set NPV	85.7% against 84.5%; +8.3 $M	More approvals and a positive NPV	Yes
5. Explainability	Declines with three reasons in plain English	100%	95% or more	Yes

Business note. The challenger passes 4 of the 5 branches of the evaluation framework. It does not rank risk better than the sub-grades on the whole population (branch 1). But it orders the applicants near the cutoff better, and that is where the approval decision happens.

With the same number of approvals as the champion, it replaces 24,426 loans. The new loans default less (23.6% against 25.1%) and add $8.3M of realized NPV on the 2015 loans. Step 7 changes the score into a cutoff and a test plan.

Step 7. Implementation

Step 6 showed that the challenger, the scorecard, passes 4 of the 5 branches of the evaluation framework. It ranks the whole population about as well as the sub-grades, but it orders the applicants near the cutoff better. At the same number of approvals, it swaps 24,426 loans and adds $8.3M of realized NPV. A better order is not yet a policy. This step answers the last question: where do we put the cutoff, how much is it worth, and how do we test it safely?

Purpose. This step changes the score into an approval rule, a dollar value and a test plan. It supports this decision: the cutoff for the champion/challenger test, and go or no-go.

7a. What is an applicant worth on the day of the application?

A cutoff decides on the day of the application. On that day, nobody knows if the loan will pay back. Thus we need two kinds of NPV:

• Realized NPV is the money that a loan actually made, after the cost of funds. We know it only after the 36 months of the loan. We use it to grade a policy on the 2015 loans.
• Expected NPV is our estimate of the realized NPV on the day of the application. It uses only the PD and the interest rate. A cutoff can use it.

The expected NPV adds two possible futures, each with its probability. If the loan does not default, the lender earns the interest. If the loan defaults, the lender loses a part of the principal. The economics of each future come from the 2013–2014 loans (as-07).

Technical note. For an applicant with loan amount \(L\), interest rate \(r\), grade \(g\) and PD \(p\) after the recalibration:

\[\text{Expected NPV} = L \times \big[(1 - p)\, v_0(r) + p\, v_1(g)\big] - \text{acquisition cost}\]

• \(v_0(r)\) is the NPV for each $1 of a loan without default. A straight line on the interest rate, fit on the 2013–2014 loans without default.
• \(v_1(g)\) is the NPV for each $1 of a loan with default. The mean for each grade, on the 2013–2014 loans with default.
• The acquisition cost is $300 for each new loan (as-03). The cost of funds is 3% per year (as-01).

# Purpose: Calculate the expected NPV of each applicant on the day of the application (as-07).
# Input:   `train` (economics), `test`, `pd_cal`.
# Output:  `ECON`, `test['enpv']`, functions `fit_economics()` and `expected_npv()`.

# NPV for each $1 from the 2013–2014 loans: a line on the interest rate (no default), a mean by grade (default).
def fit_economics(tr, cof=COST_OF_FUNDS):
    """Return the line (NPV for each $1 without default) and the mean NPV for each $1 with default by grade."""
    rate = npv_rate(tr, cof)
    is_def = tr["default"].to_numpy() == 1
    line = np.polyfit(tr["int_rate"].to_numpy()[~is_def], rate[~is_def], 1)
    by_grade = pd.Series(rate[is_def]).groupby(tr["grade"].to_numpy()[is_def]).mean()
    return line, by_grade


# Expected NPV = amount x [(1 - PD) x NPV without default + PD x NPV with default] - acquisition cost.
def expected_npv(X, pd_, econ, acq=ACQUISITION, rate_change=0.0):
    """Return the expected NPV of each applicant in X."""
    line, by_grade = econ
    rate_no_default = np.polyval(line, X["int_rate"].to_numpy() + rate_change)
    rate_default = X["grade"].map(by_grade).to_numpy()
    return X["funded_amnt"].to_numpy() * ((1 - pd_) * rate_no_default + pd_ * rate_default) - acq


ECON = fit_economics(train)
test["enpv"] = expected_npv(test, pd_cal, ECON)
N.update({"line_slope": f"{ECON[0][0] * 100:.2f}",
          "default_loss_cents": f"{-ECON[1].mean() * 100:.0f}",
          "enpv_mean": money(test["enpv"].mean(), "", 0), "npv_mean": money(test["npv"].mean(), "", 0)})
fmt(pd.DataFrame({
    "Grade": ECON[1].index,
    "Interest rate (2015)": test.groupby("grade")["int_rate"].mean().reindex(ECON[1].index).to_numpy() / 100,
    "NPV for each $1 without default (line)": np.polyval(
        ECON[0], test.groupby("grade")["int_rate"].mean().reindex(ECON[1].index).to_numpy()),
    "NPV for each $1 with default": ECON[1].to_numpy(),
}).set_index("Grade"), {"Interest rate (2015)": "{:.1%}",
                        "NPV for each $1 without default (line)": "{:.3f}",
                        "NPV for each $1 with default": "{:.3f}"})

	Interest rate (2015)	NPV for each $1 without default (line)	NPV for each $1 with default
Grade
A	6.9%	0.040	-0.378
B	10.0%	0.079	-0.374
C	13.2%	0.120	-0.379
D	16.6%	0.163	-0.390
E	19.1%	0.195	-0.395
F	23.5%	0.251	-0.402
G	26.9%	0.293	-0.394

Each increase of 1 point in the interest rate adds 1.27 cents of NPV to each $1 of a loan without default. A default loses about 39 cents for each $1, in each grade.

On the 2015 loans, the mean expected NPV is −$1. The mean realized NPV is −$116. Thus the expected NPV puts the applicants in the correct order, but its level is too high. The 2015 loans performed worse than the 2013–2014 loans. This gap is important for the cutoff below, and Part 2: Pricing explains it by grade.

7b. Cutoff

Where do we put the cutoff of the challenger?

Why this question matters. The cutoff changes the score into money. If the cutoff is too strict, the lender declines good applicants and loses volume. If the cutoff is too loose, the bad rate goes above the limit of Risk, and the test stops before it can show a gain.

Ways to do it. A cutoff rule can use two kinds of information: the risk limit of the champion, and the dollar value of each applicant. Each rule uses one, the other, both or neither. Thus four rules cover all cases:

Rule	Risk limit	Dollar value	What it does	Advantage	Main risk
Same approval rate as the champion	No	No	Approves the same number of applicants.	Step 6 used it for the swap-set.	It gives no new approvals. We screen it out.
A. Risk-constrained	Yes	No	Approves the most applicants at the champion bad rate.	Risk accepts it easily. It does not use cost assumptions.	It does not use the dollar value of a loan.
B. Maximum value	No	Yes	Approves the applicants up to the maximum total expected NPV.	The best dollar result, if the costs are correct.	It depends on the cost assumptions. The bad rate can go up.
C. Both	Yes	Yes	The maximum expected NPV, with a bad rate not above the champion.	Dollar value and the Risk limit together.	It depends on the cost assumptions. Two numbers to explain.

The plan before the evidence was rule C, because it uses the most information.

Evidence. The code puts the 2015 applicants in order from the lowest PD to the highest PD. Then it adds one applicant at a time and records three values: the approval rate, the bad rate and the total NPV. Each rule then selects one point on this curve. To grade each rule, the code uses the realized NPV of the 2015 loans, not the expected NPV.

Technical note. The code does not search for the cutoff. It sorts the applicants by PD and calculates the cumulative bad rate and the cumulative NPV after each applicant.

• Rule A: the last applicant with a cumulative bad rate not above the champion bad rate (12.5%).
• Rule B: the applicant with the maximum cumulative expected NPV.
• Rule C: the applicant with the maximum cumulative expected NPV, from the applicants that rule A approves.

The champion curve uses the sub-grade order (as-04), with a random order inside each sub-grade.

# Purpose: Find the cutoff of each rule and give its realized NPV (dec-06).
# Input:   `pd_cal`, `test['enpv']`, `npv_test`, `champ_rank`, `champ_bad`.
# Output:  `opts` (the cutoff index of each rule), table `dec06`.

import chart_style as cs

cs.setup()  # Safe to call again: the house chart style


# Sort applicants from the lowest risk. Each row is one more approval.
def policy_curve(risk, enpv):
    """Approval rate, bad rate and cumulative NPV if the policy approves the lowest risk first."""
    o = np.argsort(risk, kind="stable")
    n = np.arange(1, len(o) + 1)
    return pd.DataFrame({"approval": n / len(o), "bad_rate": np.cumsum(y_test[o]) / n,
                         "enpv": np.cumsum(enpv[o]) / 1e6, "npv": np.cumsum(npv_test[o]) / 1e6})


# Index of the last approved applicant for each rule.
def cutoff_options(curve, max_bad):
    """Return the index of the last approved applicant for rules A, B and C."""
    i_a = curve.index[curve["bad_rate"] <= max_bad].max()
    i_b = curve["enpv"].idxmax()
    i_c = curve.loc[:i_a, "enpv"].idxmax()
    return {"A": i_a, "B": i_b, "C": i_c}


# Random tiebreak inside each sub-grade.
champ_order = champ_rank + rng.random(len(champ_rank)) * 0.01
cur_champ = policy_curve(champ_order, test["enpv"].to_numpy())
cur_chall = policy_curve(pd_cal, test["enpv"].to_numpy())
champ_i = int(champ_ok.sum()) - 1
opts = cutoff_options(cur_chall, champ_bad)

rows = [{"Policy": "Champion (sub-grade)", **cur_champ.loc[champ_i].to_dict()}]
names = {"A": "A. Risk-constrained: same bad rate, maximum approvals", "B": "B. Maximum value: maximum expected NPV",
         "C": "C. Both: maximum expected NPV, bad rate limit"}
rows += [{"Policy": names[k], **cur_chall.loc[i].to_dict()} for k, i in opts.items()]
dec06 = pd.DataFrame(rows).set_index("Policy")
dec06["Realized NPV against champion ($M)"] = dec06["npv"] - dec06.loc["Champion (sub-grade)", "npv"]
dec06 = dec06.rename(columns={"approval": "Approval rate", "bad_rate": "Bad rate",
                              "enpv": "Expected NPV ($M)", "npv": "Realized NPV ($M)"})

# Numbers for the text and the chart title.
a, b = dec06.iloc[1], dec06.iloc[2]
i_best = cur_chall["npv"].idxmax()
N.update({
    "a_appr": f"{a['Approval rate']:.1%}", "a_gain": money(a["Realized NPV against champion ($M)"]),
    "a_bad": f"{a['Bad rate']:.1%}",
    "b_appr": f"{b['Approval rate']:.1%}", "b_bad": f"{b['Bad rate']:.1%}",
    "b_gain": money(b["Realized NPV against champion ($M)"]),
    "real_best": f"{cur_chall.loc[i_best, 'approval']:.0%}",
    "c_same_b": "the same as" if opts["B"] == opts["C"] else "different from",
    "appr_gain_pts": f"{(a['Approval rate'] - dec06.iloc[0]['Approval rate']) * 100:.1f}",
})
fmt(dec06, {"Approval rate": "{:.1%}", "Bad rate": "{:.1%}", "Expected NPV ($M)": "{:,.1f}",
            "Realized NPV ($M)": "{:,.1f}", "Realized NPV against champion ($M)": "{:+,.1f}"})

	Approval rate	Bad rate	Expected NPV ($M)	Realized NPV ($M)	Realized NPV against champion ($M)
Policy
Champion (sub-grade)	84.5%	12.5%	-13.0	-14.9	+0.0
A. Risk-constrained: same bad rate, maximum approvals	85.7%	12.5%	18.0	-7.8	+7.1
B. Maximum value: maximum expected NPV	70.8%	10.7%	22.1	3.2	+18.1
C. Both: maximum expected NPV, bad rate limit	70.8%	10.7%	22.1	3.2	+18.1

The chart shows the same curves. The left panel shows the risk of each policy. The right panel shows its value.

# Purpose: Show the cutoff of rule A of the scorecard on the risk curve and on the value curve (dec-06).
# Input:   `cur_champ`, `cur_chall`, `opts`, `champ_i`, `champ_bad`, `N`.
# Output:  Chart with two panels.

fig, (ax1, ax2) = plt.subplots(1, 2, figsize=(11, 4.3))
# The bad rate of the first few approvals is noise. Show the curves from 2% approval.
shown = cur_chall["approval"] >= 0.02
ax1.plot(cur_champ.loc[shown, "approval"], cur_champ.loc[shown, "bad_rate"], color=cs.GRAY, lw=1.6)
ax1.plot(cur_chall.loc[shown, "approval"], cur_chall.loc[shown, "bad_rate"], color=cs.NAVY, lw=2)
ax1.axhline(champ_bad, color=cs.GRAY, lw=0.8, ls="--")
ax1.annotate(f"Champion bad rate {champ_bad:.1%}", (0.03, champ_bad), xytext=(0, 4),
             textcoords="offset points", fontsize=8.5, color=cs.MUTED)

# Right panel: realized NPV of both policies, and the expected NPV of the challenger.
ax2.plot(cur_champ["approval"], cur_champ["npv"], color=cs.GRAY, lw=1.6)
ax2.plot(cur_chall["approval"], cur_chall["npv"], color=cs.NAVY, lw=2)
ax2.plot(cur_chall["approval"], cur_chall["enpv"], color=cs.NAVY, lw=1, ls=":")
ax2.axhline(0, color=cs.LIGHT_GRAY, lw=0.8)

# Direct labels on the lines instead of a legend. All curves meet at 100%, so the labels sit at 50%.
def at(cur, col, x=0.5):
    """Return the value of the curve at approval rate x."""
    return cur.loc[(cur["approval"] - x).abs().idxmin(), col]


ax1.annotate("Champion", (0.5, at(cur_champ, "bad_rate")), xytext=(-10, 10), textcoords="offset points",
             color=cs.GRAY, fontsize=9, ha="right")
ax1.annotate("Scorecard", (0.5, at(cur_chall, "bad_rate")), xytext=(10, -12), textcoords="offset points",
             color=cs.NAVY, fontsize=9)
ax2.annotate("Scorecard, realized", (0.35, at(cur_chall, "npv", 0.35)), xytext=(0, 14), textcoords="offset points",
             color=cs.NAVY, fontsize=9, ha="center")
ax2.annotate("Scorecard, expected", (0.55, at(cur_chall, "enpv", 0.55)), xytext=(0, 7), textcoords="offset points",
             color=cs.NAVY, fontsize=9, ha="center")
ax2.annotate("Champion, realized", (0.45, at(cur_champ, "npv", 0.45)), xytext=(0, -14), textcoords="offset points",
             color=cs.GRAY, fontsize=9, ha="center")

# Blue: the cutoff of rule A, the one point of the chart. Gray: the champion cutoff.
i_a = opts["A"]
for ax, col in ((ax1, "bad_rate"), (ax2, "npv")):
    ax.plot(cur_champ.loc[champ_i, "approval"], cur_champ.loc[champ_i, col], "o", color=cs.GRAY, ms=6)
    ax.plot(cur_chall.loc[i_a, "approval"], cur_chall.loc[i_a, col], "o", color=cs.BLUE, ms=8, zorder=5)
    ax.plot(cur_chall.loc[opts["C"], "approval"], cur_chall.loc[opts["C"], col], "o", mfc="white",
            mec=cs.NAVY, ms=6, zorder=5)
ax1.annotate(f"A: {N['a_appr']} approved", (cur_chall.loc[i_a, "approval"], cur_chall.loc[i_a, "bad_rate"]),
             xytext=(-95, 14), textcoords="offset points", color=cs.BLUE, fontsize=9, fontweight="bold")
ax2.annotate(f"A: {N['a_gain']} more\nthan the champion",
             (cur_chall.loc[i_a, "approval"], cur_chall.loc[i_a, "npv"]),
             xytext=(0.3, cur_chall["npv"].min() * 0.7), textcoords="data", color=cs.BLUE, fontsize=9,
             fontweight="bold", arrowprops={"arrowstyle": "-", "color": cs.BLUE, "lw": 0.8})
b_label = "B = C" if opts["B"] == opts["C"] else "C"
ax2.annotate(f"{b_label}: maximum expected NPV", (cur_chall.loc[opts["C"], "approval"], cur_chall.loc[opts["C"], "npv"]),
             xytext=(8, 8), textcoords="offset points", color=cs.NAVY, fontsize=8.5)

for ax in (ax1, ax2):
    cs.percent_axis(ax, "x")
    ax.set_xlim(0, 1.02)
    ax.set_xlabel("Approval rate (2015 applicants, lowest PD first)")
cs.percent_axis(ax1, "y")
ax1.set_ylabel("Bad rate of approved loans")
ax2.set_ylabel("Cumulative NPV ($M)")
ax1.set_title("Risk", loc="left", fontsize=10, color=cs.INK)
ax2.set_title("Value of the 2015 loans", loc="left", fontsize=10, color=cs.INK)
cs.action_title(fig, f"At the champion bad rate, the scorecard approves {N['a_appr']} "
                     f"vs {N['champ_appr']} and adds {N['a_gain']}",
                "2015 applicants. Navy: scorecard (challenger). Gray: champion. Dotted: expected NPV of the scorecard.")
plt.show()

How to read: Each curve adds applicants from the lowest PD to the highest PD. A curve that is lower in the left panel has less risk at the same approval rate. A curve that is higher in the right panel makes more money. The blue point is rule A.

So what: Near the champion cutoff, the navy scorecard curve has a lower bad rate than the gray champion curve. Thus the challenger can approve 1.2 points more applicants at the same risk, and the realized NPV goes up by $7.1M. The realized NPV curve reaches its maximum near 45% approval. Thus the riskier approved loans lose money at the 2015 prices.

The second table repeats the selection for other cost assumptions. The cost of funds takes the values 2%, 3% and 5%. The acquisition cost takes the values $150, $300 and $450.

# Purpose: Repeat dec-06 for other costs of funds and acquisition costs.
# Input:   `train`, `test`, `pd_cal`, `opts`.
# Output:  Table `sens06`.

rows = []
# For each cost assumption: refit the economics, find the cutoff of rule C, and the gain of rule A.
for cof in (0.02, 0.03, 0.05):
    econ = fit_economics(train, cof)
    npv_cof = loan_npv(test, cof=cof, acq=0)
    for acq in (150, 300, 450):
        enpv = expected_npv(test, pd_cal, econ, acq=acq)
        cur = policy_curve(pd_cal, enpv)
        i_c = cutoff_options(cur, champ_bad)["C"]
        npv = npv_cof - acq
        o = np.argsort(pd_cal, kind="stable")
        gain_a = npv[o][: opts["A"] + 1].sum() - npv[champ_ok].sum()
        rows.append({"Cost of funds": cof, "Acquisition cost": acq,
                     # Approval 0% means that no cutoff gives a positive expected NPV.
                     "Rule C: approval rate": cur.loc[i_c, "approval"] if cur.loc[i_c, "enpv"] > 0 else 0.0,
                     "Rule A: realized NPV against champion ($M)": gain_a / 1e6})
sens06 = pd.DataFrame(rows).set_index(["Cost of funds", "Acquisition cost"])
N.update({"c_min": f"{sens06['Rule C: approval rate'].min():.0%}",
          "c_max": f"{sens06['Rule C: approval rate'].max():.0%}",
          "a_gain_min": money(sens06["Rule A: realized NPV against champion ($M)"].min())})
sens06.index = [f"{c:.0%}, ${a}" for c, a in sens06.index]
sens06.index.name = "Cost of funds, acquisition cost"
fmt(sens06, {"Rule C: approval rate": "{:.1%}",
             "Rule A: realized NPV against champion ($M)": "{:+,.1f}"})

	Rule C: approval rate	Rule A: realized NPV against champion ($M)
Cost of funds, acquisition cost
2%, $150	85.7%	+8.4
2%, $300	82.4%	+7.9
2%, $450	70.8%	+7.4
3%, $150	82.6%	+7.6
3%, $300	70.8%	+7.1
3%, $450	0.0%	+6.6
5%, $150	0.0%	+6.1
5%, $300	0.0%	+5.6
5%, $450	0.0%	+5.1

Decision. Use rule A (risk-constrained) for the champion/challenger test. Use rule C only as a signal: it shows the applicants that need a higher price. The plan was rule C. The evidence changed the decision.

Why. Rule A approves 85.7% of applicants at the champion bad rate and adds $7.1M of realized NPV on the 2015 loans. For each cost assumption in the sensitivity table, the gain of rule A is at least $5.1M. Risk can accept it, and Finance does not need to agree on costs first.

Rule C is the same as rule B, because the bad rate limit does not stop it. It approves only 70.8% of applicants, with a bad rate of 10.7%, and adds $18.1M. But its approval rate moves from 0% to 86% when the cost assumptions change. An approval rate of 0% means that no cutoff gives a positive expected NPV. A cutoff that moves this much is not ready for production.

The expected NPV is also too high for the 2015 loans. Thus the applicants between the cutoffs of rule C and rule A do not pay for their risk at the 2015 prices. The problem is the price, not the order of the applicants. Part 2: Pricing examines it.

What would change the decision. Use rule C in two conditions. Finance confirms the cost of funds and the acquisition cost. The expected NPV matches the realized NPV on a new vintage. If the lender increases the price for the riskier applicants, the cutoff of rule C moves up to rule A.

7c. Test below the cutoff

How do we learn the outcome of applicants that no policy approves?

Why this question matters. Step 6 used a pseudo reject inference. The champion declines the lowest sub-grades, about 15% of the 2015 applicants (as-04). Lending Club booked these loans, so we know their outcomes.

But Lending Club also declined applicants that are not in the data. In production, the challenger will see these applicants too. If their risk is higher than we think, the approval gain is too high.

Ways to do it. The outcome of a declined applicant can come from four sources. These sources cover all cases:

Source of the outcome	Method	Advantage	Main risk
None	Accept the gap. Write the limitation.	Honest. Simple. No cost.	The swap-in result can be too good.
A model	Parceling, augmentation or fuzzy augmentation: estimate the outcome from the approved loans.	No new cost.	Strong assumptions. Model risk reviewers often do not trust it. It cannot correct a bias that it cannot see.
Another lender	Credit bureau outcomes of declined applicants on other loans.	Real outcomes, fast.	The data has no declined applicants, so we cannot link them. It costs money.
The lender	Approve a small random sample below the cutoff.	Real outcomes on our own product.	It costs money and takes 12 months or more.

Evidence. The data has no declined applicants, so it cannot compare the methods. The code gives the cost of the last method. It uses the 2015 loans that rule A declines. Lending Club booked these loans, so they give a lower bound on the risk of the true declined applicants.

# Purpose: Calculate the cost of a test that approves 1,000 applicants below the cutoff (dec-07).
# Input:   `pd_cal`, `opts`, `npv_test`, `y_test`.
# Output:  Table of the test cost.

# Booked 2015 loans that rule A declines: a lower bound on the risk of true declines.
below = np.ones(len(test), bool)
below[np.argsort(pd_cal, kind="stable")[: opts["A"] + 1]] = False
TEST_SIZE = 1000
test_cost = -npv_test[below].mean() * TEST_SIZE
N.update({"test_cost": money(test_cost / 1e6), "below_bad": f"{y_test[below].mean():.1%}",
          "champ_decl": f"{1 - champ_ok.mean():.0%}"})
pd.DataFrame({"Value": [f"{below.sum():,}", f"{y_test[below].mean():.1%}", money(npv_test[below].mean(), "", 0),
                        money(test_cost, "", 0)]},
             index=["Loans below the cutoff of rule A", "Default rate below the cutoff",
                    "Realized NPV for each loan below the cutoff", f"Cost of a test with {TEST_SIZE:,} loans"])

	Value
Loans below the cutoff of rule A	40,404
Default rate below the cutoff	29.6%
Realized NPV for each loan below the cutoff	−$623
Cost of a test with 1,000 loans	$622,808

Decision. Accept the gap now and write it as a limitation. Add a test below the cutoff to the rollout plan.

Why. A test gives real data. A model of the declined applicants does not.

The booked loans below the cutoff have a default rate of 29.6%. A test with 1,000 such loans costs about $0.6M of NPV. The true declined applicants have a higher risk, so the true cost is higher. This cost is small against the gain of rule A.

What would change the decision. If the lender can buy credit bureau outcomes for its declined applicants, use them first. They are faster and cheaper than a test.

7d. Excel P&L model

Finance needs a model that it can change without Python. The code collects the inputs of three policies from the 2015 loans. Then it writes the Excel file pnl_model.xlsx. The Excel file has these sheets:

• Inputs. The inputs of each policy and the cost assumptions. Blue values are inputs. Finance can change them.
• Curves. The monthly hazard of default and of prepayment of each policy, from the 2015 loans.
• Cash flows. The monthly cash flows of one loan for each policy, with the NPV.
• P&L. The result for 100,000 applicants, and the difference against the champion.
• Tornado. The change of the NPV of the challenger when each input goes up or down, with a chart.
• Breakeven. The lifetime default rate at which the NPV of a loan is zero.
• Engine. The same model for each scenario of the Tornado and Breakeven sheets.

# Purpose: Collect the inputs of three policies and write the Excel P&L model.
# Input:   `test`, `pd_cal`, `opts`, `champ_ok`, `o19`.
# Output:  File pnl_model.xlsx. Table of the inputs.

from build_pnl_excel import build_workbook

# Prepayment: full payment before month 35.
o_test = o19.loc[test.index]  # Time and event of each 2015 loan (vintage data)
paid_early = ((o_test["event"] == 2) & (o_test["time"] < 35)).to_numpy()
order = np.argsort(pd_cal, kind="stable")


# Approve the k applicants with the lowest PD.
def top_k(k):
    """Return a mask that approves the k applicants with the lowest PD."""
    ok = np.zeros(len(test), bool)
    ok[order[:k]] = True
    return ok


# The inputs of one policy for the Excel model.
def policy_inputs(ok):
    """Return the inputs of the Excel model for the approved loans in `ok`."""
    t = test[ok]
    bad = t["default"] == 1
    return {
        "Approval rate": ok.mean(),
        "Mean loan amount": t["funded_amnt"].mean(),
        "Mean interest rate": t["int_rate"].mean() / 100,
        "Net recovery rate": (t.loc[bad, "recoveries"] - t.loc[bad, "collection_recovery_fee"]).sum()
                             / (t.loc[bad, "funded_amnt"] - t.loc[bad, "total_rec_prncp"]).sum(),
        "Lifetime default rate": bad.mean(),
        "Prepayment rate": paid_early[ok].mean(),
        "Realized NPV for each loan (notebook)": t["npv"].mean(),
    }


policies = {"Champion": champ_ok, "Challenger, rule A": top_k(opts["A"] + 1),
            "Challenger, rule C": top_k(opts["C"] + 1)}
excel_inputs = pd.DataFrame({name: policy_inputs(ok) for name, ok in policies.items()})
# Monthly hazard of default and of prepayment of each policy (sheet Curves).
curves = pd.concat({name: hazards(o_test[ok])[["h_def", "h_pay"]].set_axis(["default", "prepayment"], axis=1)
                    for name, ok in policies.items()}, axis=1)
costs = {"Cost of funds": COST_OF_FUNDS, "Servicing cost": SERVICING,
         "Acquisition cost": ACQUISITION, "Recovery lag (months)": RECOVERY_LAG}
# Write pnl_model.xlsx. All results in the file are Excel formulas.
build_workbook(excel_inputs, curves, costs, "pnl_model.xlsx")
fmt(excel_inputs.T, {"Approval rate": "{:.1%}", "Mean loan amount": lambda v: money(v, "", 0), "Mean interest rate": "{:.2%}",
                     "Lifetime default rate": "{:.1%}", "Prepayment rate": "{:.1%}",
                     "Net recovery rate": "{:.1%}", "Realized NPV for each loan (notebook)": lambda v: money(v, "", 0)}).T

	Champion	Challenger, rule A	Challenger, rule C
Approval rate	84.5%	85.7%	70.8%
Mean loan amount	$12,888	$13,051	$13,334
Mean interest rate	10.15%	10.65%	10.09%
Net recovery rate	10.7%	10.9%	11.0%
Lifetime default rate	12.5%	12.5%	10.7%
Prepayment rate	54.7%	55.3%	56.6%
Realized NPV for each loan (notebook)	−$62	−$32	$16

The Excel model uses the mean loan of each policy. The notebook uses each loan. Thus the NPV of the Excel model is near to the realized NPV of the notebook, but not equal. The Cash flows sheet shows both values. In the Excel file, “rule A” and “rule C” are the rules of the cutoff decision above.

7e. Rollout plan

The back-test on the 2015 loans is not proof for the future. The economy, the applicants and the prices change. A controlled test gives the proof before a full rollout. The plan has five parts.

1. Test design. Send a random 10% of applications to the challenger cutoff (rule A). The other 90% stay with the champion. Random allocation makes the two arms equal in all other ways, so the difference comes from the policy. Also approve a small random sample below both cutoffs, as the decision above says.

2. Main KPI. The approval rate of each arm, and its default rate at month 12. The 12-month default rate gives an early estimate of the lifetime default rate (Part 2: Loss forecast). The final KPI is the realized NPV of each arm at month 36.

3. Guardrails. Check these values each month:

• The PSI of the score, against the 2015 applicants. A value below 0.1 is stable.
• The mean PD of each monthly vintage against its actual default rate.
• The approval rate of each arm, against the plan.

4. Kill criteria. Stop the test in either of two cases:

• The 12-month default rate of the challenger arm is more than 1 point above the champion arm.
• The PSI of the score is more than 0.25.

5. Top risks.

Risk	Effect	Control
The true declined applicants are riskier than the booked loans.	The approval gain is too high.	The test below the cutoff measures their default rate.
The 2015 prices do not pay for the risk.	The challenger arm can lose money, also at the same bad rate.	The pricing test in Part 2: Pricing.
The applicants change after 2015.	The score loses power.	The PSI guardrail and the kill criteria.
A complex model can find more risk than the scorecard.	The lender approves fewer applicants than it can.	Compare the GBM benchmark with the scorecard on each new vintage. Examine the GBM again if it adds 2 KS points or more (Step 4c).

Decision at the end. If the challenger arm keeps the bad rate and adds NPV, roll out the challenger. Use the result of the pricing test to set the final cutoff.

Business note. At the champion bad rate of 12.5%, the challenger approves 85.7% of applicants against 84.5%. It adds $7.1M of realized NPV on the 2015 loans. The gain stays positive for each cost assumption that we tested.

The recommendation is a champion/challenger test on 10% of applications. But the realized NPV of the whole 2015 book is negative, so a better cutoff alone is not enough. Part 2 examines the price, the loss and the recoveries.

Part 2: Across the life of a loan

Step 7 gave a better cutoff, but it also showed a problem. The mean realized NPV of a 2015 loan is −$116, so the whole 2015 book lost money. A cutoff decides only who gets a loan. Part 2 uses the same data for three more questions across the life of a loan.

Purpose. This part uses the same data for three more business questions: the lifetime loss, the price and the recoveries. Each section follows the same order: question, method and evidence, decision and dollars.

Each question belongs to a different business team:

Stage of the loan	Question	Why it matters to the business team
Before the loss	Loss forecast. What will each vintage lose over its life?	Finance sets the loss reserves and the budget with this number. A low forecast hides losses until it is too late to act.
At the application	Pricing. Does the interest rate pay for the loss and the cost of funds?	A better cutoff cannot fix a grade that loses money at every approval rate. Only the price can fix it.
After the loss	Recoveries. Which charged-off loans give the most money back?	Each dollar of recovery reduces the net loss. The collection team has a fixed capacity, so the order of the work matters.

Loss forecast: vintage curves

Finance needs the lifetime default rate of each vintage to plan the losses. The vintages from 2016 to 2018 are not complete at the end of the data (March 2019). Their current default rate shows only the first months of their life.

Step 2d gives each loan a time and an event: default, prepayment or still open. The methods below use these values.

Technical note. The hazard of a month is the share of the open loans at the start of the month that default in that month. The prepayment hazard is the same share for a full payment before the end of the term. A vintage curve multiplies the monthly hazards from month 1 to month 36.

# Purpose: Define the two methods for the cumulative default rate (dec-08).
# Input:   Hazards from `hazards()`.
# Output:  Functions `cumulative_default()` and `vintage_forecast()`.

def cumulative_default(h, competing=True):
    """Kaplan-Meier: prepayment stops the observation. Competing event: prepayment ends the loan."""
    # Method C: Kaplan-Meier. A prepaid loan leaves the data but can still default in the method.
    if not competing:
        return 1 - np.cumprod(1 - h["h_def"].to_numpy())
    # Method D: a prepaid loan cannot default. alive = share of loans still open.
    alive = np.cumprod(1 - h["h_def"].to_numpy() - h["h_pay"].to_numpy())
    return np.cumsum(np.r_[1, alive[:-1]] * h["h_def"].to_numpy())


def vintage_forecast(o, vintage, competing=True, min_at_risk=1000):
    """Curve for each grade: the hazard of the vintage while enough loans are at risk,
    then the hazard of all loans of the grade (older vintages). Weights: the grade mix."""
    target = o[o["vintage"] == vintage]
    curve = np.zeros(36)
    for g, og in target.groupby("grade"):
        own, pool = hazards(og), hazards(o[o["grade"] == g])
        # Use the hazard of the vintage if 1,000 loans or more are at risk. Else use older vintages.
        h = own.where(own["at_risk"] >= min_at_risk, pool)
        curve += len(og) / len(target) * cumulative_default(h, competing)
    return curve

How do we know the final default rate of a vintage that is still open?

Why this question matters. A young vintage has only a few months of data. Most of its defaults come later. If Finance plans with the current default rate, the loss reserves are too low, and the losses arrive as a surprise.

Ways to do it. A method must fill the months that the vintage has not lived yet. There are three ways to fill them, and one of them has two variants:

How the method fills the future months	Method	Advantage	Main risk
It does not fill them.	A. Current default rate	Simple. No assumption.	Too low for young vintages.
It copies the growth of an older vintage.	B. Chain-ladder	Finance teams know it.	It assumes that new vintages follow the old ones.
It uses the monthly hazards of all loans at risk, by grade. A prepaid loan leaves the data, but the method assumes that it can still default.	C. Kaplan-Meier by grade	It uses the open loans correctly. It gives a curve by month on book.	Too high, if many loans prepay.
The same, but a prepaid loan ends and cannot default.	D. Kaplan-Meier by grade with cumulative incidence	It uses the open loans correctly. A prepaid loan cannot default.	One more hazard to explain.

A Cox model with application features is also a hazard method. It needs more work, so we keep it for a later step.

The plan before the evidence was method C.

Evidence. The code goes back to December 2016. At that date, the 2015 vintage was 12 to 23 months old. Each method estimates the 36-month default rate of the 2015 vintage with only the data up to December 2016. Then the code compares each estimate with the actual value from March 2019. This back-test is fair, because each method sees only the data of its date.

Technical note. Method C: cumulative default = 1 − ∏(1 − h_def). Method D: the share of loans that are still open at month m is ∏(1 − h_def − h_pay). The cumulative default rate adds, for each month, the open share at the start of the month × h_def.

Each grade uses its own hazard while 1,000 loans or more are at risk. After that, it uses the hazard of the older vintages of the same grade. The vintage curve is the mean of the grade curves, with the grade mix as weights.

# Purpose: Backtest four methods on the 2015 vintage with data up to December 2016 (dec-08).
# Input:   `vint`.
# Output:  Table `dec08`: estimate and error of each method.

AS_OF = "2016-12-01"
# Use only the data up to December 2016, as if we were at that date.
o16 = observe(vint[vint["issue_year"] <= 2015], AS_OF)
v15 = o16[o16["vintage"] == 2015]
actual_15 = vint.loc[vint["issue_year"] == 2015, "is_default"].mean()

# Method B (chain-ladder): grow the 2015 curve with the growth of the 2013 curve.
ref = o16[o16["vintage"] == 2013]  # The only vintage with 36 months of data in December 2016
ref_curve = np.array([((ref["event"] == 1) & (ref["time"] <= m)).mean() for m in range(1, 37)])
age15 = months_between(vint.loc[v15.index, "issue_date"], pd.Series(pd.Timestamp(AS_OF), index=v15.index))
chain = sum(len(g) * (g["event"] == 1).mean() * ref_curve[35] / ref_curve[int(a) - 1]
            for a, g in v15.groupby(age15)) / len(v15)

# Compare each method with the actual 36-month default rate.
dec08 = pd.DataFrame({"36-month default rate of the 2015 vintage": [
    (v15["event"] == 1).mean(), chain,
    vintage_forecast(o16, 2015, competing=False)[-1], vintage_forecast(o16, 2015)[-1], actual_15,
]}, index=["A. Current default rate", "B. Chain-ladder (2013 vintage)", "C. Kaplan-Meier by grade",
           "D. Kaplan-Meier by grade with cumulative incidence", "Actual (data up to March 2019)"])
dec08["Error (points)"] = (dec08.iloc[:, 0] - actual_15) * 100
N.update({f"d08_{k}": f"{v:.1%}" for k, v in zip("ABCDX", dec08.iloc[:, 0])})
fmt(dec08, {"36-month default rate of the 2015 vintage": "{:.1%}", "Error (points)": "{:+.1f}"})

	36-month default rate of the 2015 vintage	Error (points)
A. Current default rate	7.7%	-7.2
B. Chain-ladder (2013 vintage)	15.7%	+0.8
C. Kaplan-Meier by grade	20.2%	+5.3
D. Kaplan-Meier by grade with cumulative incidence	14.6%	-0.3
Actual (data up to March 2019)	14.9%	+0.0

Decision. Method D. Show method B as a check. The plan was method C. The evidence changed the decision.

Why. Method A gives 7.7%, about half of the actual 14.9%. Method C gives 20.2%, which is much too high. Method C treats a prepaid loan as a loan that can still default. About half of the loans in this data prepay, so this error is large.

Method D gives 14.6%. Method B gives 15.7%. Both are near the actual value. Method D also gives a curve for each grade, so it changes correctly when the grade mix of a vintage changes.

What would change the decision. If the mix of applicants changes much within a grade, add a Cox model with application features.

The chart applies method D to each vintage, with all data up to March 2019.

# Purpose: Plot the vintage curves and forecast the lifetime default rate of 2016–2018.
# Input:   `o19`, `vintage_forecast()`.
# Output:  Plot and table `vint_table`.

import chart_style as cs

cs.setup()  # Safe to call again: the house chart style

FOCUS = 2018  # The vintage of the action title
rows, ends = [], {}
fig, ax = plt.subplots(figsize=(8.5, 4.6))
for year in range(2013, 2019):
    vy = o19[o19["vintage"] == year]
    # Months of data for the youngest loan of the vintage.
    max_age = months_between(pd.Series(pd.Timestamp(f"{year}-12-01")), pd.Series(DATA_END)).iloc[0]
    months = np.arange(1, min(max_age, 36) + 1)
    # Solid line: actual cumulative default rate.
    actual = [((vy["event"] == 1) & (vy["time"] <= m)).mean() for m in months]
    fc = vintage_forecast(o19, year)
    # Blue: the vintage of the title. Navy: the young vintages. Gray: the closed vintages (context).
    color = cs.BLUE if year == FOCUS else (cs.NAVY if max_age < 36 else cs.GRAY)
    lw = 2.4 if year == FOCUS else 1.4
    ax.plot(months, actual, color=color, lw=lw)
    # Dashed line: forecast of method D, for vintages that are not complete.
    if max_age < 36:
        ax.plot(np.arange(len(months), 37), fc[len(months) - 1:], "--", color=color, lw=lw * 0.8)
        ax.plot(months[-1], actual[-1], "o", color=color, ms=4 if year != FOCUS else 6)
    ends[year] = (fc[-1] if max_age < 36 else actual[-1], color)
    rows.append({"Vintage": str(year), "Loans": len(vy), "Default rate to date": (vy["event"] == 1).mean(),
                 "Lifetime default rate (forecast)": fc[-1],
                 "Months of data (youngest loan)": int(max_age)})

# Direct labels at the line ends. Move labels apart if they are too near.
ys = sorted(ends.items(), key=lambda kv: kv[1][0])
gap, last = 0.0085, -1.0
for year, (y, color) in ys:
    y_lab = max(y, last + gap)
    last = y_lab
    ax.annotate(f"{year}  {y:.1%}", (36, y), xytext=(36.6, y_lab), textcoords="data", va="center",
                fontsize=9, color=color, fontweight="bold" if year == FOCUS else "normal")
foc = o19[o19["vintage"] == FOCUS]
td = (foc["event"] == 1).mean()
age_f = months_between(pd.Series(pd.Timestamp(f"{FOCUS}-12-01")), pd.Series(DATA_END)).iloc[0]
ax.annotate(f"{FOCUS} today: {td:.1%}", (age_f, ((foc["event"] == 1) & (foc["time"] <= age_f)).mean()),
            xytext=(12, 40), textcoords="offset points", color=cs.BLUE, fontsize=9,
            arrowprops={"arrowstyle": "-", "color": cs.BLUE, "lw": 0.8})
ax.set(xlabel="Months on book", ylabel="Cumulative default rate", xlim=(0, 41))
cs.percent_axis(ax, "y")
cs.action_title(fig, f"The {FOCUS} vintage shows {td:.1%} today but its lifetime forecast is "
                     f"{ends[FOCUS][0]:.1%}",
                "Cumulative default rate by months on book. Solid: actual. Dashed: forecast (method D). "
                "Gray: complete vintages.")
plt.show()

vint_table = pd.DataFrame(rows).set_index("Vintage")
N.update({"lt_2016": f"{vint_table.loc['2016', 'Lifetime default rate (forecast)']:.1%}",
          "lt_2018": f"{vint_table.loc['2018', 'Lifetime default rate (forecast)']:.1%}",
          "td_2018": f"{vint_table.loc['2018', 'Default rate to date']:.1%}"})
fmt(vint_table, {"Loans": "{:,.0f}", "Default rate to date": "{:.1%}",
                 "Lifetime default rate (forecast)": "{:.1%}"})

	Loans	Default rate to date	Lifetime default rate (forecast)	Months of data (youngest loan)
Vintage
2013	100,422	12.3%	12.4%	63
2014	162,570	13.7%	13.8%	51
2015	282,934	14.9%	14.9%	39
2016	318,277	15.0%	16.0%	27
2017	296,484	9.4%	14.3%	15
2018	304,292	2.7%	12.8%	3

How to read: Each line is one vintage. A solid line is the actual cumulative default rate. A dashed line is the forecast up to month 36. The label at the right end gives the lifetime default rate.

So what: The 2018 vintage shows only 2.7% to date, but the forecast is 12.8%. The 2016 vintage has the highest forecast: 16.0%. A report of the current default rate hides this risk from Finance.

Business note. The current default rate of a young vintage is much too low. The 2018 vintage shows 2.7% to date, but its lifetime forecast is 12.8%. Finance must plan the reserves with the forecast. The next question is if the price pays for these losses.

Pricing: NPV by grade

The loss forecast tells Finance how much each vintage will lose. The price must pay for this loss, the cost of funds and the other costs. Step 7 showed that the 2015 book lost money also with a better cutoff. This section finds where the price is too low.

Purpose. This step shows if the interest rate pays for the loss and the cost of funds in each grade. It supports this decision: which grades need repricing.

The table gives the realized NPV of each grade for two periods. The last column gives the increase of the interest rate that makes the NPV of the 2015 loans equal to zero. This estimate keeps the volume and the default rate the same. A price increase can change both, so a pricing test must measure them (case 03).

# Purpose: Compare the price, the default rate and the NPV of each grade: 2013–2014 against 2015.
# Input:   `df`, `ECON`.
# Output:  Table `npv_grade` with the rate increase for an NPV of zero.

# Two periods: 2013–2014 and 2015.
booked = df.assign(period=np.where(df["issue_year"] <= 2014, "2013–2014", "2015"))
by = booked.groupby(["grade", "period"]).agg(
    rate=("int_rate", "mean"), bad=("default", "mean"), npv=("npv", "mean"),
    amount=("funded_amnt", "mean"), loans=("npv", "size")).unstack("period")
npv_grade = pd.DataFrame({
    "Loans (2015)": by[("loans", "2015")],
    "Interest rate 2013–2014": by[("rate", "2013–2014")] / 100,
    "Interest rate 2015": by[("rate", "2015")] / 100,
    "Default rate 2013–2014": by[("bad", "2013–2014")],
    "Default rate 2015": by[("bad", "2015")],
    "NPV for each loan 2013–2014": by[("npv", "2013–2014")],
    "NPV for each loan 2015": by[("npv", "2015")],
})
# NPV changes by ECON slope × amount × (1 − default rate) for each point of interest rate.
npv_grade["Rate increase for zero NPV, 2015 (points)"] = (
    -by[("npv", "2015")] / (ECON[0][0] * by[("amount", "2015")] * (1 - by[("bad", "2015")]))).clip(lower=0)
npv_grade.index.name = "Grade"
# Numbers for the text.
rate_change = by[("rate", "2015")] - by[("rate", "2013–2014")]
need = npv_grade["Rate increase for zero NPV, 2015 (points)"]
N.update({
    "npv_total_15": money(test["npv"].sum() / 1e6),
    "npv_total_14": money(train["npv"].sum() / 1e6),
    "rate_down": ", ".join(rate_change.index[rate_change < 0]),
    "rate_drop_b": f"{-rate_change['B']:.1f}",
    "bad_up": ", ".join(by.index[by[("bad", "2015")] > by[("bad", "2013–2014")]]),
    "neg_grades": ", ".join(npv_grade.index[npv_grade["NPV for each loan 2015"] < 0]),
    "need_small": ", ".join(need.index[need <= 0.5]), "need_big": ", ".join(need.index[need >= 2]),
})
fmt(npv_grade, {"Loans (2015)": "{:,.0f}", "Interest rate 2013–2014": "{:.1%}", "Interest rate 2015": "{:.1%}",
                "Default rate 2013–2014": "{:.1%}", "Default rate 2015": "{:.1%}",
                "NPV for each loan 2013–2014": lambda v: money(v, "", 0), "NPV for each loan 2015": lambda v: money(v, "", 0),
                "Rate increase for zero NPV, 2015 (points)": "{:.1f}"})

	Loans (2015)	Interest rate 2013–2014	Interest rate 2015	Default rate 2013–2014	Default rate 2015	NPV for each loan 2013–2014	NPV for each loan 2015	Rate increase for zero NPV, 2015 (points)
Grade
A	70,107	7.5%	6.9%	5.1%	5.4%	$97	−$51	0.3
B	91,672	11.5%	10.0%	10.3%	11.9%	$344	−$16	0.1
C	77,363	14.5%	13.2%	16.5%	19.5%	$345	−$128	1.0
D	32,676	17.7%	16.6%	21.4%	26.2%	$255	−$311	2.7
E	9,426	20.7%	19.1%	25.9%	32.9%	$137	−$657	6.0
F	1,360	24.1%	23.5%	28.9%	42.5%	$95	−$987	12.9
G	244	25.7%	26.9%	35.6%	46.3%	$82	−$1,303	19.7

The 2013–2014 loans give a total NPV of $71.7M. The 2015 loans give −$32.9M. Two changes caused this result:

• The interest rate went down in these grades: A, B, C, D, E, F. For example, it went down by 1.4 points in grade B.
• The default rate went up in these grades: A, B, C, D, E, F, G.

With the assumptions of this notebook, these grades have a negative NPV in 2015: A, B, C, D, E, F, G. These grades need a rate increase of 0.5 points or less: A, B. These grades need 2 points or more: D, E, F, G. For the riskiest grades, a cutoff is a better tool than a price.

The chart gives the same result by sub-grade.

# Purpose: Plot the realized NPV for each loan by sub-grade, for the two periods.
# Input:   `booked` (from the cell before).
# Output:  Bar chart.

import chart_style as cs

cs.setup()  # Safe to call again: the house chart style

# Mean realized NPV by sub-grade and period.
sg = booked.groupby(["sub_grade", "period"])["npv"].mean().unstack("period").reindex(SUBGRADE_ORDER).dropna(how="all")
n_neg = int((sg["2015"] < 0).sum())
N.update({"sg_neg": f"{n_neg}", "sg_n": f"{len(sg)}",
          "sg_pos_1314": f"{int((sg['2013–2014'] > 0).sum())}"})

fig, ax = plt.subplots(figsize=(11, 4.2))
x = np.arange(len(sg))
# Gray: 2013–2014 (context). 2015: red for a loss, navy for a gain.
ax.bar(x - 0.2, sg["2013–2014"], 0.4, color=cs.LIGHT_GRAY)
ax.bar(x + 0.2, sg["2015"], 0.4, color=np.where(sg["2015"] < 0, cs.NEGATIVE, cs.NAVY))
ax.axhline(0, color=cs.INK, lw=0.8)
ax.set_xticks(x, sg.index, rotation=90, fontsize=8)
ax.set(ylabel="Realized NPV for each loan ($)", xlabel="Sub-grade (A1 = lowest risk)")
ax.yaxis.set_major_formatter(plt.FuncFormatter(lambda v, _: f"${v:,.0f}" if v >= 0 else f"−${-v:,.0f}"))
# Direct key for the two periods, in the empty space at the bottom left.
ax.text(0.02, 0.22, "■ 2013–2014 loans", transform=ax.transAxes, fontsize=9, color=cs.GRAY)
ax.text(0.02, 0.14, "■ 2015 loans, loss", transform=ax.transAxes, fontsize=9, color=cs.NEGATIVE)
ax.text(0.02, 0.06, "■ 2015 loans, gain", transform=ax.transAxes, fontsize=9, color=cs.NAVY)
cs.action_title(fig, f"2015 loans lost money in {n_neg} of {len(sg)} sub-grades",
                "Realized NPV for each loan by sub-grade. Gray: 2013–2014 loans. Red: 2015 loss. Navy: 2015 gain.")
plt.show()

How to read: Each pair of bars is one sub-grade, from the lowest risk (A1) to the highest risk (G5). The gray bar is the 2013–2014 loans. The colored bar is the 2015 loans. A bar below zero is a loss.

So what: In 2013–2014, 28 of 35 sub-grades made money. In 2015, 32 of 35 sub-grades lost money. The loss is not only in the risky sub-grades, so the problem is the price of the whole book, not the cutoff.

The cost of funds is an assumption (as-01). The next table gives the NPV for each loan of the 2015 loans for three values. The last column gives the cost of funds at which the NPV of each grade is zero.

# Purpose: Repeat the NPV by grade for three costs of funds. Find the breakeven cost of funds.
# Input:   `test`, `loan_npv()`.
# Output:  Table `sens_c`.

sens_c = pd.DataFrame({f"Cost of funds {c:.0%}": pd.Series(loan_npv(test, cof=c)).groupby(
    test["grade"].to_numpy()).mean() for c in (0.02, 0.03, 0.05)})
# Breakeven: the cost of funds at which the total NPV of the grade is zero.
sens_c["Breakeven cost of funds"] = [
    brentq(lambda c: loan_npv(test[test["grade"] == g], cof=c).sum(), 1e-4, 0.5)
    if loan_npv(test[test["grade"] == g], cof=1e-4).sum() > 0 else np.nan
    for g in sens_c.index]
sens_c.index.name = "Grade"
fmt(sens_c, {c: lambda v: money(v, "", 0) for c in sens_c.columns[:3]}).assign(**{
    "Breakeven cost of funds": sens_c["Breakeven cost of funds"].map(lambda x: "None" if np.isnan(x) else f"{x:.1%}")})

	Cost of funds 2%	Cost of funds 3%	Cost of funds 5%	Breakeven cost of funds
Grade
A	$140	−$51	−$424	2.7%
B	$146	−$16	−$330	2.9%
C	$19	−$128	−$413	2.1%
D	−$168	−$311	−$592	0.9%
E	−$514	−$657	−$935	None
F	−$879	−$987	−$1,196	None
G	−$1,207	−$1,303	−$1,490	None

If a grade has no breakeven value, its NPV is negative also with a cost of funds near zero.

Business note. At the 2015 prices, these grades lose money: A, B, C, D, E, F, G. Grades A, B need a small price increase. The riskiest grades need an increase that the market may not accept, so a cutoff is the better tool for them. A pricing test must measure the effect of a price on volume and on default rate. After the price, the last lever is the money that comes back after a default.

Recoveries

The cutoff and the price decide the loss before it happens. After a charge-off, the lender can still get back a part of the principal. The collection team has a fixed capacity. This section asks where that capacity gives the most dollars.

Purpose. This step shows which charged-off loans give the most money back. It supports this decision: where to put the collection resources.

The scope is the charged-off loans from the 2013–2015 vintages. The recovery rate uses the principal that the borrower did not pay at the charge-off.

# Purpose: Calculate the net recovery rate of the charged-off loans by segment.
# Input:   `df`.
# Output:  `co` (charged-off loans), table `seg`.

co = df[df["loan_status"] == "Charged Off"].copy()
# Principal that the borrower did not pay at the charge-off.
co["principal_at_co"] = co["funded_amnt"] - co["total_rec_prncp"]
# Recoveries minus the collection fee.
co["net_recovery"] = co["recoveries"] - co["collection_recovery_fee"]
co["recovery_rate"] = co["net_recovery"] / co["principal_at_co"]
co["months_paid"] = months_between(co["issue_date"], pd.to_datetime(co["last_pymnt_d"], format="%b-%Y")).fillna(0)

# Segments to compare.
segments = {
    "Grade": co["grade"],
    "Months paid before default": pd.cut(co["months_paid"], [-1, 6, 12, 24, 36],
                                         labels=["0–6", "7–12", "13–24", "25–36"]),
    "Principal at charge-off": pd.qcut(co["principal_at_co"], 4, labels=["Q1 (low)", "Q2", "Q3", "Q4 (high)"]),
    "Home ownership": co["home_ownership"].where(co["home_ownership"].isin(["RENT", "MORTGAGE", "OWN"])),
    "Vintage": co["issue_year"].astype(str),
}
# Recovery rate of a segment: sum of net recoveries / sum of principal.
seg = pd.concat({name: co.groupby(s, observed=True).agg(
    Loans=("recovery_rate", "size"),
    **{"Recovery rate": ("net_recovery", "sum"), "No recovery": ("recoveries", lambda r: (r == 0).mean())})
    for name, s in segments.items()})
seg["Recovery rate"] = seg["Recovery rate"] / pd.concat({name: co.groupby(s, observed=True)["principal_at_co"].sum()
                                                          for name, s in segments.items()})
fee_share = co["collection_recovery_fee"].sum() / co["recoveries"].sum()
N.update({"rr_all": f"{co['net_recovery'].sum() / co['principal_at_co'].sum():.1%}",
          "rr_min": f"{seg.loc[['Grade', 'Months paid before default', 'Principal at charge-off'], 'Recovery rate'].min():.1%}",
          "rr_max": f"{seg.loc[['Grade', 'Months paid before default', 'Principal at charge-off'], 'Recovery rate'].max():.1%}",
          "fee_share": f"{fee_share:.0%}", "co_n": f"{len(co):,}"})
fmt(seg, {"Loans": "{:,.0f}", "Recovery rate": "{:.1%}", "No recovery": "{:.0%}"})

		Loans	Recovery rate	No recovery
Grade	A	6,482	11.0%	34%
	B	20,579	10.9%	25%
	C	26,389	10.9%	22%
	D	16,069	11.2%	21%
	E	5,758	11.3%	21%
	F	1,323	11.9%	21%
	G	182	11.4%	21%
Months paid before default	0–6	9,567	10.2%	24%
	7–12	17,142	10.6%	21%
	13–24	32,648	11.1%	22%
	25–36	16,969	13.6%	29%
Principal at charge-off	Q1 (low)	19,196	11.1%	30%
	Q2	19,195	11.2%	23%
	Q3	19,195	10.9%	22%
	Q4 (high)	19,196	11.1%	20%
Home ownership	MORTGAGE	29,303	11.6%	23%
	OWN	8,532	11.2%	24%
	RENT	38,947	10.5%	25%
Vintage	2013	12,378	11.4%	8%
	2014	22,315	11.4%	25%
	2015	42,089	10.7%	28%

The 76,782 charged-off loans give back 11.0% of the principal at the charge-off. The rate stays between 10.2% and 13.6% across grade, months paid and principal. The collection fee takes 17% of the gross recoveries.

The share of loans with no recovery increases for the younger vintages. Their recoveries were not complete at the end of the data.

Which charged-off loans should the collectors work first?

Why this question matters. The collectors cannot work each account with the same effort. If the order is wrong, the team spends its time on accounts that give little money back.

Ways to do it. An order can use four levels of information. Each level adds information to the level before it:

Information that the order uses	Method	Advantage	Main risk
None	Random order, or the order of the charge-off date	No work.	It gives the baseline only.
The balance	A. Rank by the principal at charge-off	Simple. No model to maintain.	It does not use differences in the recovery rate.
The balance and the borrower	B. GBM that predicts the net recovery in $	It can use the borrower and the payment history.	A model to validate and to monitor.
The effect of each treatment	Uplift model of each treatment	It finds the accounts where an action changes the result.	The data has only one treatment for each loan. We screen it out for now.

Evidence. The code trains the GBM on the charged-off loans from 2013–2014. It tests both methods on the charged-off loans from 2015. The metric is the share of all net recovery dollars in the top 20% of loans. This metric matches the business question: if the team can work only one account in five, how many dollars does it reach?

# Purpose: Compare a rank by principal with a GBM for the collection priority (dec-09).
# Input:   `co`.
# Output:  Table `dec09`.

REC_FEATURES = ["principal_at_co", "months_paid", "funded_amnt", "int_rate", "annual_inc", "dti", "fico",
                "revol_util", "mort_acc", "emp_years", "inq_last_6mths", "open_acc", "total_acc"]
# Out-of-time: train on charged-off loans of 2013–2014, test on 2015.
co_tr, co_te = co[co["issue_year"] <= 2014], co[co["issue_year"] == 2015]
rec_gbm = HistGradientBoostingRegressor(max_iter=300, learning_rate=0.05, min_samples_leaf=200,
                                        random_state=SEED).fit(co_tr[REC_FEATURES], co_tr["net_recovery"])
pred_rec = rec_gbm.predict(co_te[REC_FEATURES])


# Business metric: the share of recovery dollars in the top 20% of loans.
def top_share(score, share=0.2):
    """Share of the net recovery dollars in the top 20% of loans by the score."""
    o = np.argsort(-np.asarray(score))
    k = int(share * len(o))
    return co_te["net_recovery"].to_numpy()[o][:k].sum() / co_te["net_recovery"].sum()


dec09 = pd.DataFrame([
    {"Method": "Random order", "Top 20% share": 0.2, "R²": 0.0},
    {"Method": "A. Rank by the principal at charge-off", "Top 20% share": top_share(co_te["principal_at_co"]),
     "R²": r2_score(co_te["net_recovery"], co_te["principal_at_co"]
                    * co_tr["net_recovery"].sum() / co_tr["principal_at_co"].sum())},
    {"Method": "B. GBM", "Top 20% share": top_share(pred_rec), "R²": r2_score(co_te["net_recovery"], pred_rec)},
]).set_index("Method")
N.update({"d09_a": f"{dec09.iloc[1, 0]:.1%}", "d09_b": f"{dec09.iloc[2, 0]:.1%}"})
fmt(dec09, {"Top 20% share": "{:.1%}", "R²": "{:.3f}"})

	Top 20% share	R²
Method
Random order	20.0%	0.000
A. Rank by the principal at charge-off	45.5%	0.239
B. GBM	46.2%	0.240

Decision. Method A: rank by the principal at charge-off.

Why. The top 20% of loans by principal give 45.5% of the net recovery dollars. The GBM gives 46.2%, almost the same. The recovery rate is almost flat across the segments, so the principal decides the dollars. A model adds cost but no value here.

The flat rate suggests that Lending Club sold most charged-off loans at a similar price. Thus the real choice for this portfolio is to sell a loan or to collect it in-house. The data cannot measure this, because each loan had only one treatment. A test with random treatments can measure it, with the uplift method of case 03.

What would change the decision. If the lender collects in-house, with different treatments, the recovery rate will vary more. Then a model of the uplift of each treatment becomes useful.

Business note. The recovery rate is about 11.0% of the principal in each segment. A model does not find better accounts than a rank by balance. The lever for this portfolio is the choice between a sale of the debt and in-house collection, and a test can measure it. The conclusion collects the results of all steps.

Conclusion

The notebook started with one question: can a new score approve more applicants at the champion bad rate, and add money? Steps 0 to 7 answered it. Part 2 then showed what a cutoff cannot fix.

The answer

Yes. At the champion bad rate of 12.5%, the challenger approves 85.7% of the 2015 applicants against 84.5% for the champion. On the 2015 loans, it adds $7.1M of realized NPV, and at least $5.1M for each cost assumption that we tested. But the 2015 book lost money at its prices, so the cutoff must go together with a pricing test.

Recommendation

1. Test the challenger cutoff. Send 10% of applications to the challenger at the champion bad rate (Step 7). Add a small random test below the cutoff to measure the applicants that the data does not contain (Step 7).
2. Test a higher price for the grades with a negative NPV. The 2015 prices did not pay for the risk: 32 of 35 sub-grades lost money (Part 2: Pricing).
3. Plan the losses with the lifetime default rate, not the current rate. The 2018 vintage shows 2.7% to date, but its forecast is 12.8% (Part 2: Loss forecast).
4. Do not build a recovery model. A rank by balance gives the same dollars. Test a sale of the debt against in-house collection (Part 2: Recoveries).

What a reviewer should challenge

These limits can change the result most. Each one has a control.

Limitation	Effect on the result	Control, or what to do next
The data has only booked loans. The champion is an assumption (as-04).	The true declined applicants can be riskier. The approval gain can be too high.	The test below the cutoff. Step 6 shows three champion approval rates, and the gain stays positive.
The costs are assumptions (as-01, as-03).	The NPV can change. The cutoff of the value rule changes much.	The sensitivity tables of Step 7 and Part 2: Pricing. The tornado chart in the Excel model.
The expected NPV uses the 2013–2014 economics (as-07).	Its level is too high for the 2015 loans.	Update the economics with each new vintage. Compare the expected NPV and the realized NPV each quarter.
The 2015 back-test is one period.	The economy and the applicants change after 2015.	The PSI guardrail and the kill criteria of the rollout plan.
The scorecard adds the effect of each feature, without interactions.	The GBM benchmark finds a little more risk: 0.8 more points of approval and $2.3M more NPV on the 2015 loans.	Compare the GBM with the scorecard on each new vintage. Examine the GBM again if it adds 2 KS points or more (Step 4c).
Lending Club is not a card portfolio.	Line assignment and revolving balances are not in the data.	The same method applies with a different target.

Other limits have a small effect. The dates of the cash flows are estimates (as-06), but the totals are actual values. The recoveries of young vintages are not complete. The Excel model uses the mean loan of each policy, so its NPV differs a little from the notebook.

Next steps

1. Business case deck (6–8 slides): the challenger cutoff with the pricing test as one recommendation.
2. Model risk review of the scorecard: the table of points, the reason code text and the GBM benchmark.
3. Design of the pricing test for the grades with a negative NPV, with the method of case 03.
4. Start the rollout plan of Step 7. It has a champion/challenger test on 10% of applications and a test below the cutoff. It also checks the PSI and the vintage curves each month.

Appendix A: data dictionary

The table gives each column of the Lending Club file, then each column that this notebook calculates. The definition of a file column is the text of the Lending Club data dictionary (LCDataDictionary.xlsx), without changes. The other columns come from this notebook:

• Bucket. “Feature” means that the model uses the column. Any other value is the bucket of the column tree (Step 2b and Step 4a): 1. Not permitted, 2. No usable data, 3. No mathematical value.
• Rule. The first rule of the selection funnel that removes the column.
• Detail. The evidence for the rule: the IV, the stronger column for a duplicate, the PSI or the missing share. For a candidate column, it also gives the result of the backward elimination.
• Other use. The use of the column outside the model, for example the target or the NPV.
• Missing. The share of missing values for the loans in the scope of the model (2013–2015). A dash means that the code does not read the column.

The first table counts the columns in each bucket. The second table shows the features first, then the buckets in order.

# Purpose: Count the columns of Appendix A in each bucket.
# Input:   `column_status`.
# Output:  Small table.

bucket_order = ["Feature", "1. Not permitted", "2. No usable data", "3. No mathematical value"]
summary = column_status.groupby("bucket")["column"].size().reindex(bucket_order).fillna(0).astype(int)
summary.rename("Columns").rename_axis("Bucket").to_frame()

	Columns
Bucket
Feature	28
1. Not permitted	63
2. No usable data	26
3. No mathematical value	41

# Purpose: Build Appendix A: the definition, bucket, rule and use of each column.
# Input:   `column_status`, `lc_dict`, `CALCULATED`, `ENGINEERED`, `df`.
# Output:  Table `dictionary`.

# Uses outside the model.
OTHER_USE = {
    "loan_status": "Target (Step 2a)", "issue_d": "Scope, vintage", "term": "Scope", "application_type": "Scope",
    "funded_amnt": "NPV", "installment": "NPV", "int_rate": "NPV, expected NPV, pricing",
    "grade": "Loss forecast, pricing", "sub_grade": "Champion policy (as-04)",
    "total_pymnt": "NPV", "recoveries": "NPV, recoveries", "collection_recovery_fee": "NPV, recoveries",
    "total_rec_prncp": "Recoveries, Excel model", "last_pymnt_d": "NPV, loss forecast, recoveries",
    "home_ownership": "Recoveries (segment)", "id": "Join of the two data files",
}
# Definitions of the calculated features (Step 2d).
CALC_TEXT = {
    "fico": "Middle of the FICO range at the application.",
    "emp_years": "Employment length in years (0 to 10).",
    "credit_hist_months": "Months from the first credit line to the issue date.",
    "loan_to_income": "Loan amount divided by the annual income. New ratio: capacity.",
    "loan_to_rev_limit": "Loan amount divided by the total revolving credit limit. New ratio: size of the loan against other credit.",
    "new_acc_share": "Accounts opened in the last 24 months divided by all accounts. New ratio: credit appetite.",
    "bal_to_limit": "Total current balance divided by the total credit limit. New ratio: utilization of all credit.",
}

dictionary = column_status.copy()
calc = dictionary["column"].isin(list(CALCULATED))
dictionary["Definition"] = [
    f"{CALC_TEXT.get(c, '')} Calculated from {CALCULATED[c]}." if c in CALCULATED else lc_dict[c]
    for c in dictionary["column"]]
dictionary["Other use"] = dictionary["column"].map(OTHER_USE).fillna("")
dictionary["Missing"] = [f"{df[c].isna().mean():.1%}" if c in df.columns else "—" for c in dictionary["column"]]
dictionary["Column"] = dictionary["column"] + np.where(
    dictionary["column"].isin(ENGINEERED), " (new ratio)", np.where(calc, " (calculated)", ""))
# Sort: features first, then by bucket and rule number.
dictionary["_b"] = dictionary["bucket"].map({b: i for i, b in enumerate(bucket_order)})
dictionary["_r"] = dictionary["rule"].str.extract(r"^(\d+)\.")[0].astype(float).fillna(0)
dictionary = (dictionary.sort_values(["_b", "_r", "column"])
              .rename(columns={"bucket": "Bucket", "rule": "Rule", "detail": "Detail"})
              [["Column", "Definition", "Bucket", "Rule", "Detail", "Other use", "Missing"]]
              .set_index("Column"))
N["n_dict_rows"] = f"{len(dictionary)}"
(dictionary.style.set_properties(**{"text-align": "left", "vertical-align": "top"})
 .set_properties(subset=["Definition"], **{"min-width": "320px"}))

	Definition	Bucket	Rule	Detail	Other use	Missing
Column
acc_open_past_24mths	Number of trades opened in past 24 months.	Feature	Feature	IV 0.057; wrapper removes it in round 3		0.0%
annual_inc	The self-reported annual income provided by the borrower during registration.	Feature	Feature	IV 0.088; wrapper keeps it		0.0%
bc_open_to_buy	Total open to buy on revolving bankcards.	Feature	Feature	IV 0.078; wrapper keeps it		1.0%
dti	A ratio calculated using the borrower’s total monthly debt payments on the total debt obligations, excluding mortgage and the requested LC loan, divided by the borrower’s self-reported monthly income.	Feature	Feature	IV 0.052; wrapper keeps it		0.0%
emp_years (calculated)	Employment length in years (0 to 10). Calculated from emp_length.	Feature	Feature	IV 0.020; wrapper keeps it		6.1%
fico (calculated)	Middle of the FICO range at the application. Calculated from fico_range_low, fico_range_high.	Feature	Feature	IV 0.131; wrapper keeps it		0.0%
home_ownership	The home ownership status provided by the borrower during registration or obtained from the credit report. Our values are: RENT, OWN, MORTGAGE, OTHER	Feature	Feature	IV 0.042; wrapper keeps it	Recoveries (segment)	0.0%
loan_amnt	The listed amount of the loan applied for by the borrower. If at some point in time, the credit department reduces the loan amount, then it will be reflected in this value.	Feature	Feature	IV 0.010; wrapper removes it in round 4		0.0%
loan_to_income (new ratio)	Loan amount divided by the annual income. New ratio: capacity. Calculated from loan_amnt, annual_inc.	Feature	Feature	IV 0.043; wrapper keeps it		0.0%
loan_to_rev_limit (new ratio)	Loan amount divided by the total revolving credit limit. New ratio: size of the loan against other credit. Calculated from loan_amnt, total_rev_hi_lim.	Feature	Feature	IV 0.034; wrapper keeps it		0.0%
mo_sin_old_il_acct	Months since oldest bank installment account opened	Feature	Feature	IV 0.021; wrapper keeps it		3.7%
mo_sin_old_rev_tl_op	Months since oldest revolving account opened	Feature	Feature	IV 0.034; wrapper removes it in round 4		0.0%
mo_sin_rcnt_rev_tl_op	Months since most recent revolving account opened	Feature	Feature	IV 0.034; wrapper removes it in round 3		0.0%
mo_sin_rcnt_tl	Months since most recent account opened	Feature	Feature	IV 0.038; wrapper removes it in round 3		0.0%
mort_acc	Number of mortgage accounts.	Feature	Feature	IV 0.047; wrapper removes it in round 2		0.0%
mths_since_recent_bc	Months since most recent bankcard account opened.	Feature	Feature	IV 0.029; wrapper keeps it		0.9%
mths_since_recent_inq	Months since most recent inquiry.	Feature	Feature	IV 0.033; wrapper keeps it		10.6%
new_acc_share (new ratio)	Accounts opened in the last 24 months divided by all accounts. New ratio: credit appetite. Calculated from acc_open_past_24mths, total_acc.	Feature	Feature	IV 0.096; wrapper keeps it		0.0%
num_actv_rev_tl	Number of currently active revolving trades	Feature	Feature	IV 0.021; wrapper keeps it		0.0%
num_tl_op_past_12m	Number of accounts opened in past 12 months	Feature	Feature	IV 0.049; wrapper keeps it		0.0%
percent_bc_gt_75	Percentage of all bankcard accounts > 75% of limit.	Feature	Feature	IV 0.035; wrapper removes it in round 4		1.1%
purpose	A category provided by the borrower for the loan request.	Feature	Feature	IV 0.020; wrapper keeps it		0.0%
revol_bal	Total credit revolving balance	Feature	Feature	IV 0.019; wrapper removes it in round 2		0.0%
revol_util	Revolving line utilization rate, or the amount of credit the borrower is using relative to all available revolving credit.	Feature	Feature	IV 0.018; wrapper removes it in round 4		0.0%
tot_hi_cred_lim	Total high credit/credit limit	Feature	Feature	IV 0.078; wrapper removes it in round 2		0.0%
total_acc	The total number of credit lines currently in the borrower's credit file	Feature	Feature	IV 0.010; wrapper removes it in round 2		0.0%
total_bc_limit	Total bankcard high credit/credit limit	Feature	Feature	IV 0.072; wrapper keeps it		0.0%
verification_status	Indicates if income was verified by LC, not verified, or if the income source was verified	Feature	Feature	IV 0.010; wrapper removes it in round 2		0.0%
collection_recovery_fee	post charge off collection fee	1. Not permitted	3. Known only after the issue date		NPV, recoveries	0.0%
debt_settlement_flag	Flags whether or not the borrower, who has charged-off, is working with a debt-settlement company.	1. Not permitted	3. Known only after the issue date			—
debt_settlement_flag_date	The most recent date that the Debt_Settlement_Flag has been set	1. Not permitted	3. Known only after the issue date			—
deferral_term	Amount of months that the borrower is expected to pay less than the contractual monthly payment amount due to a hardship plan	1. Not permitted	3. Known only after the issue date			—
hardship_amount	The interest payment that the borrower has committed to make each month while they are on a hardship plan	1. Not permitted	3. Known only after the issue date			—
hardship_dpd	Account days past due as of the hardship plan start date	1. Not permitted	3. Known only after the issue date			—
hardship_end_date	The end date of the hardship plan period	1. Not permitted	3. Known only after the issue date			—
hardship_flag	Flags whether or not the borrower is on a hardship plan	1. Not permitted	3. Known only after the issue date			—
hardship_last_payment_amount	The last payment amount as of the hardship plan start date	1. Not permitted	3. Known only after the issue date			—
hardship_length	The number of months the borrower will make smaller payments than normally obligated due to a hardship plan	1. Not permitted	3. Known only after the issue date			—
hardship_loan_status	Loan Status as of the hardship plan start date	1. Not permitted	3. Known only after the issue date			—
hardship_payoff_balance_amount	The payoff balance amount as of the hardship plan start date	1. Not permitted	3. Known only after the issue date			—
hardship_reason	Describes the reason the hardship plan was offered	1. Not permitted	3. Known only after the issue date			—
hardship_start_date	The start date of the hardship plan period	1. Not permitted	3. Known only after the issue date			—
hardship_status	Describes if the hardship plan is active, pending, canceled, completed, or broken	1. Not permitted	3. Known only after the issue date			—
hardship_type	Describes the hardship plan offering	1. Not permitted	3. Known only after the issue date			—
last_credit_pull_d	The most recent month LC pulled credit for this loan	1. Not permitted	3. Known only after the issue date			—
last_fico_range_high	The upper boundary range the borrower’s last FICO pulled belongs to.	1. Not permitted	3. Known only after the issue date			—
last_fico_range_low	The lower boundary range the borrower’s last FICO pulled belongs to.	1. Not permitted	3. Known only after the issue date			—
last_pymnt_amnt	Last total payment amount received	1. Not permitted	3. Known only after the issue date			—
last_pymnt_d	Last month payment was received	1. Not permitted	3. Known only after the issue date		NPV, loss forecast, recoveries	0.1%
loan_status	Current status of the loan	1. Not permitted	3. Known only after the issue date		Target (Step 2a)	0.0%
next_pymnt_d	Next scheduled payment date	1. Not permitted	3. Known only after the issue date			—
orig_projected_additional_accrued_interest	The original projected additional interest amount that will accrue for the given hardship payment plan as of the Hardship Start Date. This field will be null if the borrower has broken their hardship payment plan.	1. Not permitted	3. Known only after the issue date			—
out_prncp	Remaining outstanding principal for total amount funded	1. Not permitted	3. Known only after the issue date			—
out_prncp_inv	Remaining outstanding principal for portion of total amount funded by investors	1. Not permitted	3. Known only after the issue date			—
payment_plan_start_date	The day the first hardship plan payment is due. For example, if a borrower has a hardship plan period of 3 months, the start date is the start of the three-month period in which the borrower is allowed to make interest-only payments.	1. Not permitted	3. Known only after the issue date			—
pymnt_plan	Indicates if a payment plan has been put in place for the loan	1. Not permitted	3. Known only after the issue date			—
recoveries	post charge off gross recovery	1. Not permitted	3. Known only after the issue date		NPV, recoveries	0.0%
settlement_amount	The loan amount that the borrower has agreed to settle for	1. Not permitted	3. Known only after the issue date			—
settlement_date	The date that the borrower agrees to the settlement plan	1. Not permitted	3. Known only after the issue date			—
settlement_percentage	The settlement amount as a percentage of the payoff balance amount on the loan	1. Not permitted	3. Known only after the issue date			—
settlement_status	The status of the borrower’s settlement plan. Possible values are: COMPLETE, ACTIVE, BROKEN, CANCELLED, DENIED, DRAFT	1. Not permitted	3. Known only after the issue date			—
settlement_term	The number of months that the borrower will be on the settlement plan	1. Not permitted	3. Known only after the issue date			—
total_pymnt	Payments received to date for total amount funded	1. Not permitted	3. Known only after the issue date		NPV	0.0%
total_pymnt_inv	Payments received to date for portion of total amount funded by investors	1. Not permitted	3. Known only after the issue date			—
total_rec_int	Interest received to date	1. Not permitted	3. Known only after the issue date			0.0%
total_rec_late_fee	Late fees received to date	1. Not permitted	3. Known only after the issue date			0.0%
total_rec_prncp	Principal received to date	1. Not permitted	3. Known only after the issue date		Recoveries, Excel model	0.0%
funded_amnt	The total amount committed to that loan at that point in time.	1. Not permitted	4. Output of the Lending Club policy		NPV	0.0%
grade	LC assigned loan grade	1. Not permitted	4. Output of the Lending Club policy		Loss forecast, pricing	0.0%
installment	The monthly payment owed by the borrower if the loan originates.	1. Not permitted	4. Output of the Lending Club policy		NPV	0.0%
int_rate	Interest Rate on the loan	1. Not permitted	4. Output of the Lending Club policy		NPV, expected NPV, pricing	0.0%
sub_grade	LC assigned loan subgrade	1. Not permitted	4. Output of the Lending Club policy		Champion policy (as-04)	0.0%
annual_inc_joint	The combined self-reported annual income provided by the co-borrowers during registration	1. Not permitted	5. Joint applications only			—
dti_joint	A ratio calculated using the co-borrowers' total monthly payments on the total debt obligations, excluding mortgages and the requested LC loan, divided by the co-borrowers' combined self-reported monthly income	1. Not permitted	5. Joint applications only			—
revol_bal_joint	Sum of revolving credit balance of the co-borrowers, net of duplicate balances	1. Not permitted	5. Joint applications only			—
sec_app_chargeoff_within_12_mths	Number of charge-offs within last 12 months at time of application for the secondary applicant	1. Not permitted	5. Joint applications only			—
sec_app_collections_12_mths_ex_med	Number of collections within last 12 months excluding medical collections at time of application for the secondary applicant	1. Not permitted	5. Joint applications only			—
sec_app_earliest_cr_line	Earliest credit line at time of application for the secondary applicant	1. Not permitted	5. Joint applications only			—
sec_app_fico_range_high	FICO range (low) for the secondary applicant	1. Not permitted	5. Joint applications only			—
sec_app_fico_range_low	FICO range (high) for the secondary applicant	1. Not permitted	5. Joint applications only			—
sec_app_inq_last_6mths	Credit inquiries in the last 6 months at time of application for the secondary applicant	1. Not permitted	5. Joint applications only			—
sec_app_mort_acc	Number of mortgage accounts at time of application for the secondary applicant	1. Not permitted	5. Joint applications only			—
sec_app_mths_since_last_major_derog	Months since most recent 90-day or worse rating at time of application for the secondary applicant	1. Not permitted	5. Joint applications only			—
sec_app_num_rev_accts	Number of revolving accounts at time of application for the secondary applicant	1. Not permitted	5. Joint applications only			—
sec_app_open_acc	Number of open trades at time of application for the secondary applicant	1. Not permitted	5. Joint applications only			—
sec_app_open_act_il	Number of currently active installment trades at time of application for the secondary applicant	1. Not permitted	5. Joint applications only			—
sec_app_revol_util	Ratio of total current balance to high credit/credit limit for all revolving accounts	1. Not permitted	5. Joint applications only			—
verification_status_joint	Indicates if the co-borrowers' joint income was verified by LC, not verified, or if the income source was verified	1. Not permitted	5. Joint applications only			—
addr_state	The state provided by the borrower in the loan application	1. Not permitted	6. Fair lending risk			0.0%
emp_title	The job title supplied by the Borrower when applying for the loan.*	1. Not permitted	6. Fair lending risk			—
zip_code	The first 3 numbers of the zip code provided by the borrower in the loan application.	1. Not permitted	6. Fair lending risk			—
desc	Loan description provided by the borrower	2. No usable data	1. Not about the borrower			—
disbursement_method	The method by which the borrower receives their loan. Possible values are: CASH, DIRECT_PAY	2. No usable data	1. Not about the borrower			—
funded_amnt_inv	The total amount committed by investors for that loan at that point in time.	2. No usable data	1. Not about the borrower			—
id	A unique LC assigned ID for the loan listing.	2. No usable data	1. Not about the borrower		Join of the two data files	0.0%
initial_list_status	The initial listing status of the loan. Possible values are – W, F	2. No usable data	1. Not about the borrower			—
member_id	A unique LC assigned Id for the borrower member.	2. No usable data	1. Not about the borrower			—
policy_code	publicly available policy_code=1 new products not publicly available policy_code=2	2. No usable data	1. Not about the borrower			—
title	The loan title provided by the borrower	2. No usable data	1. Not about the borrower			—
url	URL for the LC page with listing data.	2. No usable data	1. Not about the borrower			—
application_type	Indicates whether the loan is an individual application or a joint application with two co-borrowers	2. No usable data	2. Scope column		Scope	0.0%
issue_d	The month which the loan was funded	2. No usable data	2. Scope column		Scope, vintage	0.0%
term	The number of payments on the loan. Values are in months and can be either 36 or 60.	2. No usable data	2. Scope column		Scope	0.0%
all_util	Balance to credit limit on all trades	2. No usable data	8. Not collected in the training years	Missing up to 100% in 2013–2014		97.3%
il_util	Ratio of total current balance to high credit/credit limit on all install acct	2. No usable data	8. Not collected in the training years	Missing up to 100% in 2013–2014		97.7%
inq_fi	Number of personal finance inquiries	2. No usable data	8. Not collected in the training years	Missing up to 100% in 2013–2014		97.3%
inq_last_12m	Number of credit inquiries in past 12 months	2. No usable data	8. Not collected in the training years	Missing up to 100% in 2013–2014		97.3%
max_bal_bc	Maximum current balance owed on all revolving accounts	2. No usable data	8. Not collected in the training years	Missing up to 100% in 2013–2014		97.3%
mths_since_rcnt_il	Months since most recent installment accounts opened	2. No usable data	8. Not collected in the training years	Missing up to 100% in 2013–2014		97.4%
open_acc_6m	Number of open trades in last 6 months	2. No usable data	8. Not collected in the training years	Missing up to 100% in 2013–2014		97.3%
open_act_il	Number of currently active installment trades	2. No usable data	8. Not collected in the training years	Missing up to 100% in 2013–2014		97.3%
open_il_12m	Number of installment accounts opened in past 12 months	2. No usable data	8. Not collected in the training years	Missing up to 100% in 2013–2014		97.3%
open_il_24m	Number of installment accounts opened in past 24 months	2. No usable data	8. Not collected in the training years	Missing up to 100% in 2013–2014		97.3%
open_rv_12m	Number of revolving trades opened in past 12 months	2. No usable data	8. Not collected in the training years	Missing up to 100% in 2013–2014		97.3%
open_rv_24m	Number of revolving trades opened in past 24 months	2. No usable data	8. Not collected in the training years	Missing up to 100% in 2013–2014		97.3%
total_bal_il	Total current balance of all installment accounts	2. No usable data	8. Not collected in the training years	Missing up to 100% in 2013–2014		97.3%
total_cu_tl	Number of finance trades	2. No usable data	8. Not collected in the training years	Missing up to 100% in 2013–2014		97.3%
earliest_cr_line	The month the borrower's earliest reported credit line was opened	3. No mathematical value	7. Replaced by a calculated column	Replaced by credit_hist_months		0.0%
emp_length	Employment length in years. Possible values are between 0 and 10 where 0 means less than one year and 10 means ten or more years.	3. No mathematical value	7. Replaced by a calculated column	Replaced by emp_years		6.1%
fico_range_high	The upper boundary range the borrower’s FICO at loan origination belongs to.	3. No mathematical value	7. Replaced by a calculated column	Replaced by fico		0.0%
fico_range_low	The lower boundary range the borrower’s FICO at loan origination belongs to.	3. No mathematical value	7. Replaced by a calculated column	Replaced by fico		0.0%
acc_now_delinq	The number of accounts on which the borrower is now delinquent.	3. No mathematical value	9. Weak (IV < 0.01)	IV 0.000; wrapper removes it in round 1		0.0%
bal_to_limit (new ratio)	Total current balance divided by the total credit limit. New ratio: utilization of all credit. Calculated from tot_cur_bal, tot_hi_cred_lim.	3. No mathematical value	9. Weak (IV < 0.01)	IV 0.005; wrapper keeps it		0.0%
chargeoff_within_12_mths	Number of charge-offs within 12 months	3. No mathematical value	9. Weak (IV < 0.01)	IV 0.000; wrapper removes it in round 1		0.0%
collections_12_mths_ex_med	Number of collections in 12 months excluding medical collections	3. No mathematical value	9. Weak (IV < 0.01)	IV 0.000; wrapper removes it in round 1		0.0%
delinq_2yrs	The number of 30+ days past-due incidences of delinquency in the borrower's credit file for the past 2 years	3. No mathematical value	9. Weak (IV < 0.01)	IV 0.001; wrapper removes it in round 1		0.0%
delinq_amnt	The past-due amount owed for the accounts on which the borrower is now delinquent.	3. No mathematical value	9. Weak (IV < 0.01)	IV 0.000; wrapper removes it in round 1		0.0%
mths_since_last_delinq	The number of months since the borrower's last delinquency.	3. No mathematical value	9. Weak (IV < 0.01)	IV 0.002; wrapper removes it in round 3		49.8%
mths_since_last_major_derog	Months since most recent 90-day or worse rating	3. No mathematical value	9. Weak (IV < 0.01)	IV 0.004; wrapper removes it in round 3		71.9%
mths_since_last_record	The number of months since the last public record.	3. No mathematical value	9. Weak (IV < 0.01)	IV 0.003; wrapper removes it in round 4		82.4%
mths_since_recent_bc_dlq	Months since most recent bankcard delinquency	3. No mathematical value	9. Weak (IV < 0.01)	IV 0.002; wrapper removes it in round 2		74.3%
mths_since_recent_revol_delinq	Months since most recent revolving delinquency.	3. No mathematical value	9. Weak (IV < 0.01)	IV 0.002; wrapper removes it in round 2		64.6%
num_accts_ever_120_pd	Number of accounts ever 120 or more days past due	3. No mathematical value	9. Weak (IV < 0.01)	IV 0.003; wrapper removes it in round 1		0.0%
num_actv_bc_tl	Number of currently active bankcard accounts	3. No mathematical value	9. Weak (IV < 0.01)	IV 0.004; wrapper removes it in round 1		0.0%
num_bc_sats	Number of satisfactory bankcard accounts	3. No mathematical value	9. Weak (IV < 0.01)	IV 0.001; wrapper removes it in round 1		0.0%
num_bc_tl	Number of bankcard accounts	3. No mathematical value	9. Weak (IV < 0.01)	IV 0.006; wrapper removes it in round 3		0.0%
num_il_tl	Number of installment accounts	3. No mathematical value	9. Weak (IV < 0.01)	IV 0.007; wrapper removes it in round 4		0.0%
num_op_rev_tl	Number of open revolving accounts	3. No mathematical value	9. Weak (IV < 0.01)	IV 0.004; wrapper removes it in round 1		0.0%
num_rev_accts	Number of revolving accounts	3. No mathematical value	9. Weak (IV < 0.01)	IV 0.002; wrapper removes it in round 3		0.0%
num_sats	Number of satisfactory accounts	3. No mathematical value	9. Weak (IV < 0.01)	IV 0.000; wrapper removes it in round 2		0.0%
num_tl_120dpd_2m	Number of accounts currently 120 days past due (updated in past 2 months)	3. No mathematical value	9. Weak (IV < 0.01)	IV 0.001; wrapper removes it in round 1		3.8%
num_tl_30dpd	Number of accounts currently 30 days past due (updated in past 2 months)	3. No mathematical value	9. Weak (IV < 0.01)	IV 0.000; wrapper removes it in round 1		0.0%
num_tl_90g_dpd_24m	Number of accounts 90 or more days past due in last 24 months	3. No mathematical value	9. Weak (IV < 0.01)	IV 0.000; wrapper removes it in round 1		0.0%
open_acc	The number of open credit lines in the borrower's credit file.	3. No mathematical value	9. Weak (IV < 0.01)	IV 0.000; wrapper removes it in round 1		0.0%
pct_tl_nvr_dlq	Percent of trades never delinquent	3. No mathematical value	9. Weak (IV < 0.01)	IV 0.001; wrapper removes it in round 4		0.0%
pub_rec	Number of derogatory public records	3. No mathematical value	9. Weak (IV < 0.01)	IV 0.002; wrapper removes it in round 1		0.0%
pub_rec_bankruptcies	Number of public record bankruptcies	3. No mathematical value	9. Weak (IV < 0.01)	IV 0.002; wrapper removes it in round 1		0.0%
tax_liens	Number of tax liens	3. No mathematical value	9. Weak (IV < 0.01)	IV 0.000; wrapper removes it in round 1		0.0%
tot_coll_amt	Total collection amounts ever owed	3. No mathematical value	9. Weak (IV < 0.01)	IV 0.001; wrapper removes it in round 2		0.0%
total_bal_ex_mort	Total credit balance excluding mortgage	3. No mathematical value	9. Weak (IV < 0.01)	IV 0.009; wrapper removes it in round 2		0.0%
total_il_high_credit_limit	Total installment high credit/credit limit	3. No mathematical value	9. Weak (IV < 0.01)	IV 0.007; wrapper keeps it		0.0%
avg_cur_bal	Average current balance of all accounts	3. No mathematical value	10. Duplicate of a stronger column	Duplicate of tot_hi_cred_lim (IV 0.072); wrapper keeps it		0.0%
bc_util	Ratio of total current balance to high credit/credit limit for all bankcard accounts.	3. No mathematical value	10. Duplicate of a stronger column	Duplicate of percent_bc_gt_75 (IV 0.029); wrapper removes it in round 3		1.1%
credit_hist_months (calculated)	Months from the first credit line to the issue date. Calculated from earliest_cr_line, issue_d.	3. No mathematical value	10. Duplicate of a stronger column	Duplicate of mo_sin_old_rev_tl_op (IV 0.022); wrapper removes it in round 3		0.0%
inq_last_6mths	The number of inquiries in past 6 months (excluding auto and mortgage inquiries)	3. No mathematical value	10. Duplicate of a stronger column	Duplicate of mths_since_recent_inq (IV 0.028); wrapper keeps it		0.0%
num_rev_tl_bal_gt_0	Number of revolving trades with balance >0	3. No mathematical value	10. Duplicate of a stronger column	Duplicate of num_actv_rev_tl (IV 0.020); wrapper removes it in round 1		0.0%
tot_cur_bal	Total current balance of all accounts	3. No mathematical value	10. Duplicate of a stronger column	Duplicate of tot_hi_cred_lim (IV 0.064); wrapper removes it in round 2		0.0%
total_rev_hi_lim	Total revolving high credit/credit limit	3. No mathematical value	10. Duplicate of a stronger column	Duplicate of total_bc_limit (IV 0.057); wrapper removes it in round 2		0.0%

Reader guides: decisions, assumptions and glossary

This appendix holds the reference material of the notebook. Click a title to open it.

NoteHow to read this notebook

• Read the text to follow the logic. Click Show code to see the code for a step.
• The notebook follows the steps of a model project, from Step 0 to Step 7. Part 2 uses the same data for three more questions across the life of a loan.
• Each step starts with a gray block. This block says what the step before showed, and the question of this step.
• A blue block records a decision. It starts with a question. Then it gives the possible methods, the evidence, the decision and the reason.
• A gray-blue block (Technical note) gives the method in detail for a data scientist. A business reader can skip it.
• A green block is a business note. It gives the result of a step for a manager.
• Appendix A gives the meaning of each column in the data.

Each team can read only the part that it needs.

Part	Question	Team
Step 0	What is the business problem?	All
Steps 1–3	Which data can answer it?	All
Steps 4–6	Which applicants will default? How good is the score?	Credit risk
Step 7	Where do we put the cutoff? Is the proposal good for the business?	Credit risk, Finance, deal economics
Part 2: Loss forecast	What is the lifetime default rate of each vintage?	Loss forecasting
Part 2: Pricing	Does the interest rate pay for the loss and the cost of funds in each grade?	Product and pricing
Part 2: Recoveries	Which charged-off loans give the most money back?	Collections and recoveries

NoteDecision log

The decision blocks in the text show only the question. This log gives each decision an ID for a model risk review. Click the question to go to the decision.

ID	Question	Decision	Step
dec-01	Which loans give a default rate that we can trust?	36-month loans from 2013–2015 with a full outcome window	2
dec-02	Which split gives a test result that is near to production?	Out-of-time split: train 2013–2014, test 2015	3
dec-10	Which columns should the model use?	A selection funnel with fixed rules (a filter method). Backward elimination is a check, and a GBM on all columns is a reference. Each column that we do not use has a reason.	4
dec-03	Can the challenger use the Lending Club sub-grade?	No. Use only the data from the application and the credit bureau.	4
dec-04	Which type of model gives the score?	Scorecard (logistic regression with WoE). The GBM is a benchmark. The rule before the evidence: use the GBM only if it adds 2 KS points or more. It adds less, so we keep the plan.	4
dec-05	How do we handle the default rate in the training data?	No resampling. Recalibrate the PD to the recent default rate.	4
dec-06	Where do we put the cutoff of the challenger?	Rule A: same bad rate as the champion, maximum approvals. The plan was the maximum expected NPV. The evidence changed the decision.	7
dec-07	How do we learn the outcome of applicants that no policy approves?	Accept the gap now. Add a small test below the cutoff to the rollout plan.	7
dec-08	How do we know the final default rate of a vintage that is still open?	Kaplan-Meier by grade with cumulative incidence. The plan was Kaplan-Meier only.	Part 2
dec-09	Which charged-off loans should the collectors work first?	Rank by the principal at charge-off. A GBM adds no value.	Part 2

NoteAssumption register

ID	Assumption	Value	Source	Effect if the value is wrong
as-01	Cost of funds	3% per year	Assumption: a bank funding cost plus a charge for capital. Confirm with Finance.	Large. Part 2: Pricing gives the cost of funds at which each grade has an NPV of zero. Step 7 shows the effect on the cutoff.
as-02	Servicing cost	1% of each payment	The Lending Club servicing fee to investors	Small effect on NPV.
as-03	Acquisition cost for each new loan	$300	Assumption: marketing and underwriting cost. Confirm with Finance.	Large effect on the cutoff of rule C (Step 7). The sensitivity check uses $150 and $450.
as-04	Champion policy	Rank applicants by sub-grade (A1 is the best). Approve the best sub-grades up to about 85% of applicants.	Assumption. The data has only booked loans, so the champion must decline some of them.	The swap-set changes. The method stays the same.
as-05	Default rate of declined applicants	Not known	The data has only approved loans	The approval gain can be too high. See the test below the cutoff.
as-06	Dates of the cash flows of each loan	Installment each month until the last payment. The rest in the month of the last payment. Recoveries 12 months after the last payment.	The data gives only the totals and the date of the last payment	Small. The dates change the discount, not the amounts.
as-07	Economics for the expected NPV	NPV for each $1 from the 2013–2014 loans: a straight line on the interest rate without default, a mean by grade with default	Step 7a	The level of the expected NPV is too high for the 2015 loans. The order of applicants stays correct.

NoteGlossary

The words in this table are technical names. The notebook uses each name only with the meaning below.

Technical name	Meaning in this notebook
Default	A loan with the status “Charged Off”, “Default” or “Late (31-120 days)” at the end of the data.
Bad rate	The number of defaults divided by the number of approved loans.
PD (probability of default)	The probability that a loan becomes a default before the end of its term.
Champion	The current approval policy.
Challenger	The new approval policy that we test against the champion.
Cutoff	The score value that separates approved applicants from declined applicants.
Swap-set	The applicants that one policy approves and the other policy declines. “Swap-in” applicants are new approvals. “Swap-out” applicants are new declines.
Pseudo reject inference	A test of a new policy on loans that the champion declines but that the lender booked. We know the outcomes of these loans.
Out-of-time split	A split where the test data comes from a later period than the training data.
Vintage	All loans that the lender issued in the same year.
Months on book	The number of months from the issue date of a loan.
WoE (weight of evidence)	A change of each feature into the log ratio of the share of defaults to the share of non-defaults in each bin. A higher WoE means a higher risk.
IV (information value)	A measure of how well one feature separates defaults from non-defaults. Less than 0.02 is weak.
GBM (gradient boosting machine)	A model that adds many small decision trees. Each tree corrects the errors of the previous trees.
KS	The maximum difference between the cumulative score distributions of defaults and non-defaults.
Gini	2 × AUC − 1.
AUC	Area under the ROC curve. It shows how well a score puts defaults above non-defaults. 0.5 is random. 1.0 is perfect.
PSI (population stability index)	A measure of the change in a score distribution between two samples. Less than 0.1 is stable.
Calibration	The match between the predicted PD and the actual default rate.
Recalibration	A change of the level of the PD that keeps the order of the applicants.
Log-odds	ln(PD / (1 − PD)).
Reason codes	The features that increase the risk of an applicant most. The lender gives them in an adverse action notice.
Reject inference	A method that estimates the outcome of declined applicants.
NPV (net present value)	The value today of all future cash flows of a loan, after the cost of funds.
Realized NPV	The NPV from the actual cash flows of a loan. We know it only at the end of the loan.
Expected NPV	The NPV that we expect on the day of the application, from the PD and the interest rate.
Prepayment	A full payment of the loan before the end of its term.
Charge-off	The lender stops the collection of payments and records the loan as a loss.
Recovery rate	Recoveries minus the collection fee, divided by the principal at charge-off.
Kaplan-Meier	A method that estimates the share of loans without default at each month on book. It uses loans that are not yet closed.
Hazard	The share of the open loans at the start of a month that have an event (default or prepayment) in that month.
Competing event	An event that stops another event. A prepaid loan cannot default.
Cumulative incidence	The cumulative default rate when prepayment is a competing event.
Cumulative default rate	The share of the loans of a vintage with a default up to a month on book.
Lifetime default rate	The cumulative default rate at month 36.
Chain-ladder	A method that grows the cumulative default rate of a young vintage with the growth of an older vintage.
Uplift	The change of an outcome that a treatment causes for a customer.
Guardrail	A metric that the test monitors each month. It must stay inside a limit.
Kill criteria	The conditions that stop a test before its end.
Tornado chart	A chart that shows the change of a result when each input goes up or down.
Breakeven	The value of an input at which the NPV is zero.
Feature	A column that the model uses as an input.
Scorecard	A logistic regression on WoE features, with the coefficients changed to points. The lender adds the points to get the score.
Regulation B (Reg B)	The US rule under the Equal Credit Opportunity Act. It requires the lender to give the main reasons for each decline.
SR 11-7	The guidance of the US Federal Reserve on model risk management. It asks for an independent validation of each model.
Temporal leakage	Information from a later period that goes into the training data. The test result then looks better than production.
k-fold cross-validation	A method that splits the data into k parts. It trains on k − 1 parts and tests on the last part, k times.
Rolling time windows	A validation method that trains on one period and tests on the next period. Then it moves both periods forward and repeats.
Class weight	A weight that makes each default count more than each non-default in the training of a model.
SMOTE	A method that adds synthetic defaults to the training data. It makes new rows between near defaults.
KNN (k-nearest neighbors)	A model that gives each applicant the mean outcome of the k most similar applicants in the training data.
SVM (support vector machine)	A model that finds the boundary with the largest margin between defaults and non-defaults.
Naive Bayes	A model that assumes that the features are independent for a given outcome.
Selection funnel	A list of rules that removes columns in a fixed order. Each removed column gets the reason of the first rule that removes it.
Spearman correlation	The correlation of the ranks of two columns. 1 or −1 means that one column gives the same order as the other.
Business note	A short result of a step for a manager.
Technical note	Detail of a method for a data scientist. A business reader can skip it.

Appendix B: reproducibility

# Purpose: Print the versions of the packages, for reproducibility.
# Input:   None.
# Output:  Text.

import sys
import sklearn
print("Python", sys.version.split()[0])
print("numpy", np.__version__, "| pandas", pd.__version__,
      "| scikit-learn", sklearn.__version__, "| duckdb", duckdb.__version__)

Python 3.13.16
numpy 2.5.3 | pandas 3.0.5 | scikit-learn 1.9.1 | duckdb 1.5.6

Appendix C: functions used in the analysis

Steps 3 to 7 use the functions in the file case01_helpers.py. They give the metrics, the WoE encoder with the scorecard, the GBM benchmark, the policy rules and the reason codes. A reviewer can read the full source below, without a separate file.

# Purpose: Show the source of the helper module in the notebook.
# Input:   File case01_helpers.py (next to this notebook).
# Output:  The source as a Python code block, or a note if the file does not exist.

from pathlib import Path

helpers_file = Path("case01_helpers.py")
if helpers_file.exists():
    fence = "`" * 3  # A literal fence here would end the cell in the .qmd file
    print(fence + "python\n" + helpers_file.read_text(encoding="utf-8").rstrip("\n") + "\n" + fence)
else:
    print("*The file `case01_helpers.py` does not exist. The helper functions are in the cells of Step 3.*")

"""Helper functions for case 01, Steps 3 to 7.

# Purpose: One place for the tools that Steps 3-7 use: the metrics of the evaluation framework,
#          the WoE encoder with the logistic regression, the GBM, the policy rules and the reason codes.
# Input:   configure(cat_all=..., seed=...) one time, then configure(train=...) after the split.
#          The notebook gives the categorical columns, the seed and the training data.
# Output:  Functions and constants. The notebook loads them with `from case01_helpers import *`.

The configuration objects (CAT_LEVELS, GBM_PARAMS) are dicts that configure() changes in place.
Thus the names that the notebook imports always show the current configuration.
"""
import numpy as np
import pandas as pd
from sklearn.ensemble import HistGradientBoostingClassifier
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import roc_auc_score

__all__ = [
    "configure", "HAND_PICKED", "SUBGRADE_ORDER", "CAT_LEVELS", "GBM_PARAMS",
    "ks_stat", "score_metrics", "psi", "approval_at_bad_rate",
    "WoEEncoder", "split_types", "fit_woe_lr", "pd_woe_lr",
    "gbm_matrix", "fit_gbm", "subgrade_rank", "approve_top", "logit", "expit",
    "REASON_TEXT", "reason_effects", "describe_reason",
    "typical_woe", "scorecard_points", "scorecard_reason_effects",
]

# --- Configuration (set by configure()) ---
_CAT_ALL = []        # Names of the categorical columns (from Step 2)
CAT_LEVELS = {}      # Category levels of the training data, for each categorical column
GBM_PARAMS = dict(learning_rate=0.05, max_iter=500, max_leaf_nodes=31, min_samples_leaf=200,
                  l2_regularization=1.0, early_stopping=True, validation_fraction=0.1, random_state=None)

# The hand-picked list of the first version of this notebook (the baseline of the column selection).
HAND_PICKED = [
    "loan_amnt", "emp_years", "annual_inc", "dti", "delinq_2yrs", "fico",
    "credit_hist_months", "inq_last_6mths", "mths_since_last_delinq", "open_acc",
    "pub_rec", "revol_bal", "revol_util", "total_acc", "mort_acc",
    "pub_rec_bankruptcies", "acc_open_past_24mths", "bc_util",
    "num_tl_op_past_12m", "mo_sin_old_rev_tl_op", "tot_cur_bal", "total_rev_hi_lim",
    "home_ownership", "verification_status", "purpose",
]
# Lending Club sub-grades from the best (A1) to the worst (G5).
SUBGRADE_ORDER = [f"{g}{i}" for g in "ABCDEFG" for i in range(1, 6)]


def configure(cat_all=None, seed=None, train=None):
    """Set the categorical columns, the seed of the GBM and the category levels of the training data."""
    if cat_all is not None:
        _CAT_ALL[:] = list(cat_all)
    if seed is not None:
        GBM_PARAMS["random_state"] = seed
    if train is not None:
        # A level that is not in the training data becomes missing in the test data.
        CAT_LEVELS.clear()
        CAT_LEVELS.update({c: sorted(train[c].dropna().unique()) for c in _CAT_ALL})


# --- Branch 1 of the evaluation framework: ranking ---
def ks_stat(y, score):
    """Return the KS: the maximum distance between the cumulative score distributions of defaults
    and non-defaults."""
    order = np.argsort(-np.asarray(score))
    y_sorted = np.asarray(y)[order]
    cum_bad = np.cumsum(y_sorted) / y_sorted.sum()
    cum_good = np.cumsum(1 - y_sorted) / (1 - y_sorted).sum()
    return np.max(np.abs(cum_bad - cum_good))


def score_metrics(y, score):
    """Return the AUC, the Gini (2 x AUC - 1) and the KS of a score (a higher score = a higher risk)."""
    auc = roc_auc_score(y, score)
    return {"AUC": auc, "Gini": 2 * auc - 1, "KS": ks_stat(y, score)}


# --- Branch 3: stability ---
def psi(expected, actual, bins=10):
    """Return the population stability index of a score between two samples.
    The bins are the deciles of the expected (reference) sample."""
    edges = np.unique(np.quantile(expected, np.linspace(0, 1, bins + 1)))
    edges[0], edges[-1] = -np.inf, np.inf
    e = np.histogram(expected, edges)[0] / len(expected)
    a = np.histogram(actual, edges)[0] / len(actual)
    e, a = np.clip(e, 1e-6, None), np.clip(a, 1e-6, None)
    return np.sum((a - e) * np.log(a / e))


# --- Branch 4: business value ---
def approval_at_bad_rate(y, risk, max_bad_rate):
    """Return the largest approval share where the bad rate of the approved loans stays at or below
    the limit. The applicants go in order of risk, from the lowest."""
    order = np.argsort(risk)
    cum_bad = np.cumsum(np.asarray(y)[order]) / np.arange(1, len(order) + 1)
    ok = np.where(cum_bad <= max_bad_rate)[0]
    return (ok.max() + 1) / len(order) if len(ok) else 0.0


# --- WoE encoder and logistic regression (the scorecard) ---
class WoEEncoder:
    """Bins each feature on the training data and replaces each bin with its WoE.

    WoE = ln(share of defaults in the bin / share of non-defaults in the bin).
    A higher WoE means a higher risk. Missing values get their own bin.
    Numeric bins: deciles of the training data. Categories with less than 1% of the loans join "OTHER".
    """

    def __init__(self, num_features, cat_features, n_bins=10, min_share=0.01):
        self.num, self.cat, self.n_bins, self.min_share = num_features, cat_features, n_bins, min_share

    def _bin(self, col, x):
        """Return the bin label of each value of one column."""
        if col in self.num:
            b = np.digitize(x, self.edges_[col]).astype(object)
            b[np.isnan(x.astype(float))] = "missing"
            return pd.Series(b, index=x.index).astype(str)
        x = x.fillna("missing").astype(str)
        return x.where(x.isin(self.keep_[col]), "OTHER")

    def fit(self, X, y):
        """Return the encoder with the bins, the WoE and the IV of each column of the training data."""
        self.edges_, self.keep_, self.woe_, self.iv_ = {}, {}, {}, {}
        n_bad, n_good = y.sum(), (1 - y).sum()
        for col in self.num + self.cat:
            x = X[col]
            if col in self.num:
                q = np.nanquantile(x.astype(float), np.linspace(0, 1, self.n_bins + 1)[1:-1])
                self.edges_[col] = np.unique(q)
            else:
                share = x.fillna("missing").astype(str).value_counts(normalize=True)
                self.keep_[col] = share[share >= self.min_share].index
            t = pd.crosstab(self._bin(col, x), y)
            # Add 0.5 to each count, so that a bin without defaults has a finite WoE.
            bad = (t.get(1, 0) + 0.5) / n_bad
            good = (t.get(0, 0) + 0.5) / n_good
            woe = np.log(bad / good)
            self.woe_[col] = woe
            self.iv_[col] = float(np.sum((bad - good) * woe))
        return self

    def transform(self, X, cols=None):
        """Return the WoE of each value. A bin that the training data does not have gets 0."""
        cols = cols or list(self.woe_)
        return pd.DataFrame({c: self._bin(c, X[c]).map(self.woe_[c]).fillna(0.0).to_numpy()
                             for c in cols}, index=X.index)


def split_types(cols):
    """Return two lists: the numeric columns and the categorical columns."""
    return [c for c in cols if c not in _CAT_ALL], [c for c in cols if c in _CAT_ALL]


def fit_woe_lr(tr, cols, **lr_args):
    """Return (encoder, columns, logistic regression), fitted on the WoE of the training data."""
    enc = WoEEncoder(*split_types(cols)).fit(tr, tr["default"])
    lr = LogisticRegression(C=1.0, max_iter=1000, **lr_args).fit(enc.transform(tr, cols), tr["default"])
    return enc, cols, lr


def pd_woe_lr(model, X):
    """Return the PD of the logistic regression with WoE for each row of X."""
    enc, cols, lr = model
    return lr.predict_proba(enc.transform(X, cols))[:, 1]


# --- GBM: one set of settings for all GBMs in the notebook ---
def gbm_matrix(X, cols):
    """Return the numeric matrix for the GBM. Categories become codes with the levels of the training data."""
    if any(c not in CAT_LEVELS for c in split_types(cols)[1]):
        raise RuntimeError("Call configure(train=train) before gbm_matrix().")
    out = X[cols].copy()
    for c in split_types(cols)[1]:
        levels = CAT_LEVELS[c]
        # A level that is not in the training data becomes missing (code -1, then NaN).
        codes = pd.Categorical(out[c].where(out[c].isin(levels)), categories=levels).codes
        out[c] = np.where(codes < 0, np.nan, codes)
    return out.astype(float)


def fit_gbm(tr, cols, **extra_params):
    """Return a GBM with GBM_PARAMS (plus extra_params), fitted on the training data."""
    model = HistGradientBoostingClassifier(**{**GBM_PARAMS, **extra_params},
                                           categorical_features=[c in _CAT_ALL for c in cols])
    return model.fit(gbm_matrix(tr, cols), tr["default"])


# --- Policies ---
def subgrade_rank(X):
    """Return the rank of the sub-grade of each row: A1 = 0, G5 = 34."""
    return X["sub_grade"].map({g: i for i, g in enumerate(SUBGRADE_ORDER)}).to_numpy()


def approve_top(risk, share):
    """Return True for the share of applicants with the lowest risk."""
    return risk <= np.quantile(risk, share)


# --- Recalibration (Step 5) ---
def logit(p):
    """Return the log-odds ln(p / (1 - p))."""
    return np.log(p / (1 - p))


def expit(z):
    """Return the probability 1 / (1 + exp(-z)): the inverse of logit()."""
    return 1 / (1 + np.exp(-z))


# --- Branch 5: reason codes ---
# Plain-English meaning of each feature that can become a reason code.
# Each entry: (what the feature measures, text if a high value adds risk, text if a low value adds risk).
REASON_TEXT = {
    # Application data
    "loan_amnt": ("Amount of the loan", "Large loan amount", "Small loan amount"),
    "annual_inc": ("Annual income", "High income", "Low income"),
    "emp_years": ("Years with the current employer", "Long time with the current employer",
                  "Short time with the current employer"),
    "dti": ("Monthly debt payments as a share of income", "High debt payments compared with income",
            "Low debt payments compared with income"),
    "home_ownership": ("Home ownership status", "Home ownership status", "Home ownership status"),
    "verification_status": ("Income verification status", "Income verification status",
                            "Income verification status"),
    "purpose": ("Purpose of the loan", "Purpose of the loan", "Purpose of the loan"),
    # Engineered ratios (Step 2)
    "loan_to_income": ("Loan amount as a share of income", "Loan amount is high compared with income",
                       "Loan amount is low compared with income"),
    "revol_bal_to_income": ("Card and revolving balances as a share of income",
                            "High revolving balances compared with income",
                            "Low revolving balances compared with income"),
    "bal_to_limit": ("Total balances as a share of total credit limits", "Balances are near the credit limits",
                     "Balances are far below the credit limits"),
    "open_to_total_acc": ("Open accounts as a share of all accounts", "High share of accounts still open",
                          "Low share of accounts still open"),
    "new_acc_share": ("Accounts opened in the last 24 months as a share of all accounts",
                      "Many of the accounts are new", "Few of the accounts are new"),
    "loan_to_rev_limit": ("Loan amount as a share of the total revolving credit limit",
                          "Loan amount is high compared with the credit limits",
                          "Loan amount is low compared with the credit limits"),
    "installment_to_income": ("Monthly payment of the loan as a share of income",
                              "High monthly payment compared with income",
                              "Low monthly payment compared with income"),
    # Credit score and length of credit history
    "fico": ("FICO score", "High FICO score", "Low FICO score"),
    "credit_hist_months": ("Length of the credit history", "Long credit history", "Short credit history"),
    "mo_sin_old_rev_tl_op": ("Age of the oldest revolving account", "Old revolving accounts",
                             "No old revolving accounts"),
    "mo_sin_old_il_acct": ("Age of the oldest installment account", "Old installment accounts",
                           "No old installment accounts"),
    # Recent credit activity
    "inq_last_6mths": ("Credit inquiries in the last 6 months", "Many recent credit inquiries",
                       "Few recent credit inquiries"),
    "mths_since_recent_inq": ("Months since the most recent credit inquiry", "No recent credit inquiry",
                              "Recent credit inquiry"),
    "acc_open_past_24mths": ("Accounts opened in the last 24 months", "Many accounts opened recently",
                             "Few accounts opened recently"),
    "num_tl_op_past_12m": ("Accounts opened in the last 12 months", "Many accounts opened in the last year",
                           "Few accounts opened in the last year"),
    "mo_sin_rcnt_tl": ("Months since the most recent new account", "No recent new account",
                       "Recent new account"),
    "mo_sin_rcnt_rev_tl_op": ("Months since the most recent new revolving account",
                              "No recent new revolving account", "Recent new revolving account"),
    "mths_since_recent_bc": ("Months since the most recent new bank card", "No recent new bank card",
                             "Recent new bank card"),
    # Use of revolving credit and bank cards
    "revol_bal": ("Balance on revolving accounts", "High balance on revolving accounts",
                  "Low balance on revolving accounts"),
    "revol_util": ("Share of the revolving credit limit in use", "High use of revolving credit",
                   "Low use of revolving credit"),
    "bc_util": ("Share of the bank card limit in use", "High use of bank card credit",
                "Low use of bank card credit"),
    "bc_open_to_buy": ("Unused credit on bank cards", "Much unused credit on bank cards",
                       "Little unused credit on bank cards"),
    "percent_bc_gt_75": ("Share of bank cards above 75% of the limit", "Many bank cards near the limit",
                         "Few bank cards near the limit"),
    "total_bc_limit": ("Total credit limit on bank cards", "High credit limit on bank cards",
                       "Low credit limit on bank cards"),
    "total_rev_hi_lim": ("Total revolving credit limit", "High revolving credit limit",
                         "Low revolving credit limit"),
    "max_bal_bc": ("Maximum balance on one bank card", "High balance on one bank card",
                   "Low balance on one bank card"),
    # Size of the credit file and balances
    "tot_hi_cred_lim": ("Total credit limit on all accounts", "High total credit limit",
                        "Low total credit limit"),
    "tot_cur_bal": ("Total balance on all accounts", "High total balance", "Low total balance"),
    "avg_cur_bal": ("Average balance for each account", "High average balance", "Low average balance"),
    "total_bal_ex_mort": ("Total balance without the mortgage", "High balances other than the mortgage",
                          "Low balances other than the mortgage"),
    "total_il_high_credit_limit": ("Total limit on installment accounts", "High installment credit limit",
                                   "Low installment credit limit"),
    "mort_acc": ("Number of mortgage accounts", "Many mortgage accounts", "No mortgage account"),
    "open_acc": ("Number of open accounts", "Many open accounts", "Few open accounts"),
    "total_acc": ("Number of accounts in the credit file", "Many accounts in the credit file",
                  "Few accounts in the credit file"),
    "num_actv_rev_tl": ("Number of active revolving accounts", "Many active revolving accounts",
                        "Few active revolving accounts"),
    "num_actv_bc_tl": ("Number of active bank cards", "Many active bank cards", "Few active bank cards"),
    "num_bc_sats": ("Number of bank cards in good standing", "Many bank cards in good standing",
                    "Few bank cards in good standing"),
    "num_bc_tl": ("Number of bank card accounts", "Many bank card accounts", "Few bank card accounts"),
    "num_il_tl": ("Number of installment accounts", "Many installment accounts", "Few installment accounts"),
    "num_op_rev_tl": ("Number of open revolving accounts", "Many open revolving accounts",
                      "Few open revolving accounts"),
    "num_rev_accts": ("Number of revolving accounts", "Many revolving accounts", "Few revolving accounts"),
    "num_rev_tl_bal_gt_0": ("Revolving accounts with a balance", "Many revolving accounts with a balance",
                            "Few revolving accounts with a balance"),
    "num_sats": ("Number of accounts in good standing", "Many accounts in good standing",
                 "Few accounts in good standing"),
    # Payment problems
    "delinq_2yrs": ("Late payments of 30 days or more in the last 2 years", "Recent late payments",
                    "No recent late payments"),
    "mths_since_last_delinq": ("Months since the last late payment", "No recent late payment",
                               "Recent late payment"),
    "mths_since_recent_bc_dlq": ("Months since the last late payment on a bank card",
                                 "No recent late payment on a bank card", "Recent late payment on a bank card"),
    "mths_since_recent_revol_delinq": ("Months since the last late payment on a revolving account",
                                       "No recent late payment on a revolving account",
                                       "Recent late payment on a revolving account"),
    "mths_since_last_major_derog": ("Months since the last serious late payment",
                                    "No recent serious late payment", "Recent serious late payment"),
    "num_accts_ever_120_pd": ("Accounts that were ever 120 days late", "Accounts with serious late payments",
                              "No accounts with serious late payments"),
    "num_tl_30dpd": ("Accounts 30 days late now", "Accounts late now", "No accounts late now"),
    "num_tl_120dpd_2m": ("Accounts 120 days late now", "Accounts seriously late now",
                         "No accounts seriously late now"),
    "num_tl_90g_dpd_24m": ("Accounts 90 days late in the last 24 months", "Serious late payments recently",
                           "No serious late payments recently"),
    "pct_tl_nvr_dlq": ("Share of accounts never late", "Most accounts never late",
                       "Many accounts with late payments"),
    "acc_now_delinq": ("Accounts late now", "Accounts late now", "No accounts late now"),
    "delinq_amnt": ("Amount past due now", "Amount past due now", "No amount past due"),
    "chargeoff_within_12_mths": ("Charge-offs in the last 12 months", "Recent charge-off",
                                 "No recent charge-off"),
    "collections_12_mths_ex_med": ("Collections in the last 12 months (not medical)", "Recent collections",
                                   "No recent collections"),
    "tot_coll_amt": ("Total amount in collection", "Amount in collection", "No amount in collection"),
    # Public records
    "pub_rec": ("Derogatory public records", "Derogatory public records", "No derogatory public records"),
    "pub_rec_bankruptcies": ("Bankruptcies in the public record", "Bankruptcy in the public record",
                             "No bankruptcy in the public record"),
    "tax_liens": ("Tax liens", "Tax liens", "No tax liens"),
    "mths_since_last_record": ("Months since the last public record", "No recent public record",
                               "Recent public record"),
}


def describe_reason(col, value, typical, categorical=False):
    """Return (meaning, direction, borrower sentence) for one reason code.
    If the column is not in REASON_TEXT, the column name becomes the text."""
    meaning, high_text, low_text = REASON_TEXT.get(col, (col, f"High value of {col}", f"Low value of {col}"))
    if categorical:
        return meaning, "Different from typical", f"{meaning}: {value}"
    if pd.isna(value):
        return meaning, "Missing", f"{meaning}: no data in the credit file"
    if pd.isna(typical) or value == typical:
        return meaning, "Same as typical", meaning
    return (meaning, "Higher than typical", high_text) if value > typical else \
           (meaning, "Lower than typical", low_text)


def typical_woe(model, typical):
    """Return the WoE of the typical approved applicant for each feature of the scorecard.
    `typical` holds one raw value for each feature: the median of a number, the most common category."""
    enc, cols, _ = model
    row = pd.DataFrame({c: [typical[c]] for c in cols})
    for c in enc.num:
        row[c] = row[c].astype(float)
    return enc.transform(row, cols).iloc[0]


def scorecard_points(model, X_rows, woe_typical):
    """Return a DataFrame (rows x features) of reason points on the log-odds scale:
    coefficient x (WoE of the applicant - WoE of the typical approved applicant).
    A positive value means that the feature increases the PD of the applicant."""
    enc, cols, lr = model
    w = enc.transform(X_rows, cols)
    return (w - woe_typical[cols]) * lr.coef_[0]


def scorecard_reason_effects(model, X_rows, woe_typical):
    """Return a DataFrame (rows x features): the PD decrease if one feature takes the WoE of the
    typical approved applicant. The order of the features is the same as the order of the points."""
    pts = scorecard_points(model, X_rows, woe_typical)
    z = logit(pd_woe_lr(model, X_rows))
    return pd.DataFrame(expit(z)[:, None] - expit(z[:, None] - pts.to_numpy()),
                        index=X_rows.index, columns=pts.columns)


def reason_effects(model, X_rows, typical, features):
    """Return a DataFrame (rows x features): the PD decrease if one feature takes the typical value.
    A positive value means that the actual value of the feature increases the PD of the applicant.
    This method works for any model (for example the GBM benchmark). The scorecard uses its points."""
    base = model.predict_proba(X_rows)[:, 1]
    out = {}
    for c in features:
        changed = X_rows.copy()
        changed[c] = typical[c]
        out[c] = base - model.predict_proba(changed)[:, 1]
    return pd.DataFrame(out, index=X_rows.index)