Convert the Notebook

The data scientist who handed me that notebook told me it was “basically done, it just needs to be deployed.” It ran top to bottom and printed a believable number, so I understood why they thought so. It took me the better part of a day to explain why “runs in my kernel” and “can be deployed” are not the same sentence. The notebook wasn’t the product. It was the proof that a product was possible, and the thing the data scientist put in front of stakeholders to make that case. Turning the one into the other is this lesson. And there’s a quiet cost in that conversion worth naming up front. The notebook was also how the work got explained to people. Its charts and its printed number were a translation of the model into something a human could read, and a package explains itself to no one. That translation job doesn’t disappear when the cells become functions. It moves downstream, to the API contract, the dashboard, the report, and keeping it alive across the handoff is part of the engineer’s job, not the data scientist’s.

The previous lesson built the target shape: a module you can import, an entry point you run, an install that resolves from anywhere. Now you have the raw material to pour into it. Conversion isn’t a rewrite. It’s moving each piece of the notebook to the place in that shape where it belongs, and watching a specific fragility disappear as you do.

The notebook you are handed

Here’s the starting point: the notebook the data scientist handed over, and it’s a real one, not a toy. It loads the Lending Club loans, sizes up the label imbalance and the missingness, cuts the book by purpose and term, tries and discards two feature sets, builds a third with mean-filled gaps and one-hot categoricals, splits and fits a logistic regression, checks the AUC is stable across folds, bakes the model off against a decision tree and xgboost (and keeps the logistic, since it wins on these features and credit can read its coefficients), and prints a test ROC-AUC around 0.70. It ends with “looks good, ship it.”

# Loan default model — handoff from data science
#
# Default-risk model on the Lending Club book. This is the **v3** notebook — the v1
# and v2 feature experiments are kept below for the record. Runs top to bottom — just
# hit **Run All**. Picks features, trains a logistic regression, prints the AUC.
# Looks solid; should be ready to deploy.
#
# *Handoff notes: decision threshold still TBD with the credit team; retraining
# cadence TBD.*

import importlib.util
import pathlib

import pandas as pd
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import roc_auc_score
from sklearn.model_selection import cross_val_score, train_test_split
from sklearn.tree import DecisionTreeClassifier
from xgboost import XGBClassifier


# This notebook is the "before" handoff — it predates the package, so it loads the data
# itself. It must not `import de_refs`: this directory builds a package named
# `ml_pipeline` (the published distribution's name), which would shadow the real one
# `de_refs` imports. So locate the vendored CSV by the installed package's path.
def load_loans() -> pd.DataFrame:
    spec = importlib.util.find_spec("de_refs")
    data_dir = pathlib.Path(spec.origin).parent / "data"
    return pd.read_csv(data_dir / "lending_club.csv")


df = load_loans()
df.head()

# how big is this thing, and what did pandas make of each column?
print(df.shape)
df.dtypes

# how imbalanced is the label?
df["bad_loan"].value_counts(normalize=True)

# Roughly 18% of the book goes bad, so plain accuracy will read inflated here — a
# model that predicts "fine" for everyone scores ~0.82 doing nothing. AUC is the
# number to report.

# quick look at the numeric columns
df[["loan_amnt", "int_rate", "annual_inc", "dti", "revol_util"]].describe()

# what is actually missing, and how much?
df.isna().mean().sort_values(ascending=False).head(10)

# does the book behave like a loan book? rate and default by purpose
df.groupby("purpose")[["int_rate", "bad_loan"]].mean().sort_values("bad_loan")

# term mix — the 60-month loans should be the riskier bucket
print(df["term"].value_counts())
df.groupby("term")["bad_loan"].mean()

# anything obviously collinear before we throw these at a linear model?
df[["loan_amnt", "int_rate", "annual_inc", "dti", "revol_util"]].corr().round(2)

# Feature experiments
#
# v1 and v2 kept for the record — v3 below is the keeper.

# v1 — two features, quick and dirty (dropna, just want a floor)
sub = df[["loan_amnt", "int_rate", "bad_loan"]].dropna()
X1 = sub[["loan_amnt", "int_rate"]]
y1 = sub["bad_loan"]
X1_tr, X1_te, y1_tr, y1_te = train_test_split(X1, y1, test_size=0.2, random_state=42)
m1 = LogisticRegression(max_iter=1000).fit(X1_tr, y1_tr)
roc_auc_score(y1_te, m1.predict_proba(X1_te)[:, 1])

# v2 — add the debt-burden numerics. barely moves.
sub2 = df[["loan_amnt", "int_rate", "dti", "revol_util", "bad_loan"]].dropna()
X2 = sub2[["loan_amnt", "int_rate", "dti", "revol_util"]]
y2 = sub2["bad_loan"]
X2_tr, X2_te, y2_tr, y2_te = train_test_split(X2, y2, test_size=0.2, random_state=42)
m2 = LogisticRegression(max_iter=1000).fit(X2_tr, y2_tr)
roc_auc_score(y2_te, m2.predict_proba(X2_te)[:, 1])

# v2 barely moves it. **v3**: keep all five numerics, fill the gaps with the column
# mean instead of dropping rows, and one-hot `home_ownership` + `purpose`. Best so
# far — going with this.

# a handful of numeric features, fill the gaps with the column mean
num_cols = ["loan_amnt", "int_rate", "annual_inc", "dti", "revol_util"]
df[num_cols] = df[num_cols].fillna(df[num_cols].mean())

# one-hot a couple of categoricals
df = pd.get_dummies(df, columns=["home_ownership", "purpose"])
feature_cols = num_cols + [
    c for c in df.columns if c.startswith(("home_ownership_", "purpose_"))
]

X = df[feature_cols]
y = df["bad_loan"]
X.shape, y.shape

# paranoia: no NaNs left, and the label rate did not move?
print(X.isna().sum().sum())
print(y.mean())

X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42
)

%%time
model = LogisticRegression(max_iter=1000)
model.fit(X_train, y_train)

proba = model.predict_proba(X_test)[:, 1]
auc = roc_auc_score(y_test, proba)
auc

# is that number stable, or did we get a lucky split?
cross_val_score(LogisticRegression(max_iter=1000), X, y, cv=3, scoring="roc_auc")

# which features carry it? (raw coefficients, unscaled — good enough for a look)
pd.Series(model.coef_[0], index=feature_cols).sort_values()

# Model bake-off — is logistic leaving much on the table?
#
# Same split, same features: the linear model against a single decision tree and
# xgboost.

%%time
candidates = {
    "logistic": LogisticRegression(max_iter=1000),
    "decision tree": DecisionTreeClassifier(max_depth=6, random_state=42),
    "xgboost": XGBClassifier(random_state=42),
}
for name, m in candidates.items():
    m.fit(X_train, y_train)
    m_auc = roc_auc_score(y_test, m.predict_proba(X_test)[:, 1])
    print(f"{name:14s} test AUC: {m_auc:.3f}")

# Neither tree model beats the linear one here — with five numerics and two
# one-hot groups the signal is mostly linear, and an untuned booster does not fix
# a feature problem. Keeping **logistic**: best number on this data, and credit
# can read which way each feature pushes a score straight off the coefficients.

# what would a 0.5 cutoff actually flag? (rough look — threshold is TBD)
pd.crosstab(y_test, proba > 0.5)

# sanity: score one borrower end to end
model.predict_proba(X_test.iloc[[0]])

# TODO before deploy(?): pick the threshold with the credit team, revisit boosting
# with real tuning and richer features, look at calibration. Parking those — the
# AUC is there and it is stable across folds.

print(f"rows: {len(df)}")
print(f"features: {len(feature_cols)}")
print(f"test AUC: {auc:.3f}")
print("looks good — ship it")

Nothing here is wrong as data science. The AUC is real, the model trains, and on the author’s machine the whole thing runs end to end every time. If parts of the notebook’s own reasoning are new to you (why AUC instead of accuracy on an imbalanced book, what the cross-validation check buys, how to read the model bake-off), don’t stop to master them here. Each gets its full treatment later: metrics and splits in M3, Numerical & Statistical Foundations, and model comparison in M4, Classical Machine Learning. This lesson’s job is the conversion, so where each cell belongs matters more right now than what each cell computes. The problem is everything the notebook quietly assumes, and each assumption maps to a piece of the package shape that’s missing:

• It runs in one order, in one kernel. The training cell works only because every cell above it already ran, in sequence, mutating the same df in a shared namespace. df is one-hot encoded in place by the feature cell. Run the training cell against a fresh df and the feature columns don’t exist. Restart the kernel and run any cell out of order and you get a KeyError or a NameError. The notebook isn’t a program. It’s a sequence of side effects on a shared namespace, and that namespace is the only thing holding it together. A module has no such hidden state. Its names come from its own top level, in a defined order, every import.
• There is no way to call it. The feature munging, the split, and the fit are loose statements at the top level, not functions. Nothing else, not a test, a scheduler, an API, or a retraining job, can ask it to build features or train without rerunning the whole notebook from the top. The fix is the module boundary from the previous lesson: named functions another file can import and call.
• The work fires the moment the code runs. Loading the data, encoding the features, and training the model all happen the instant their cells execute. There’s no separation between “define how to build features and train” and “actually do it,” so you can’t borrow one piece without triggering the entire pipeline. The fix is the __main__ guard: definitions at the top level, the run gated behind the guard.

Let me be concrete about how that hidden state actually bites, because you’ll debug it before you learn to distrust it. A notebook is stateful. The kernel keeps every variable in memory after a cell runs, which is genuinely useful. You load a large dataset once and keep re-running the model cell without reloading. Here’s the cost, the other edge of that same blade: because the state persists, you can run the cells in any order, and the notebook will happily use a df left behind by a cell you ran twenty minutes ago and have since edited. The execution-count labels in the margins ([12], [7], [31]) are the tell. When they aren’t a clean ascending 1, 2, 3, the notebook’s current output was produced by a run order that no longer exists in the code, and nobody, including the author, can say for certain what produced the number on screen. Here’s the classic version. A variable still resolves because it lives in the kernel, even though you deleted the cell that defined it. The code reads as broken but runs fine, right up until a fresh kernel turns it into a NameError. The single most useful habit this lesson is really teaching is to mistrust any notebook result until Restart Kernel and Run All reproduces it from a clean state, because that’s the only run order a module can have. A package can’t drift this way. Its names come from its own top level, in a fixed order, on every import.

There’s also a real modeling bug hiding in the feature cell. df[num_cols].fillna(df[num_cols].mean()) computes the mean over the whole frame, including the rows that become the test set, so a test statistic leaks into training. And we’re going to leave it exactly as it is. Conversion changes the shape of code, not its behavior. If the package computed a different AUC than the notebook, the conversion would be wrong. The leak is carried forward unchanged here and fixed in M3, Numerical & Statistical Foundations, where the fill is learned on the training split only. Naming it now is the point: the conversion preserves it faithfully, and the package’s AUC has to match the notebook’s to prove the move was clean.

The conversion is those three structural fixes, in order. Each one moves a part of the notebook into the layout and closes the gap it left. The scrolly below starts from the mess — the out-of-order cells and the shared df they all mutate — and walks the whole move, cell by cell: every piece of the notebook, where it lands in the package, and the tangle of run-order wires straightening out as each one finds its file. Watch the disorder disappear, because that disappearance is the point, not just the tidier result. It’s the same decomposition your module project asks you to perform yourself on the Adult / Census Income data.

Move the logic into a module

The first move is to take what the notebook does, build features then split-and-fit, and lift each piece out of the loose cells into a named function in a module. This is the module boundary from the previous lesson: feature-building and training become capabilities another file can import and call, rather than statements that only mean something in run order.

"""The notebook's logic, converted into importable functions.

This is the *after* of the conversion lesson: the data scientist's notebook (build
features, split, fit, score) collapsed from loose run-order cells into named functions
you can import and call. The two functions are dataset-agnostic — you pass the frame and
say which columns are numeric and which categorical — so they apply to any tabular set.
They are exactly the published `dutchengineer-ml-pipeline` library; `main()` here is a
thin driver that runs them on the loan data.

Behavior is identical to the notebook — same columns, same split seed, same model, same
test AUC — because a faithful conversion preserves behavior; it only changes the *shape*.
The full-frame mean fill leaks test statistics into training; it is carried forward
unchanged (M3 fixes it on the training split only).
"""

import pandas as pd
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import roc_auc_score
from sklearn.model_selection import train_test_split

from ml_pipeline.data import load as load_loans

SEED = 42


def build_features(
    df: pd.DataFrame, numeric: list[str], categorical: list[str]
) -> tuple[pd.DataFrame, list[str]]:
    """Munge any frame into (X, feature_names) given its numeric/categorical columns."""
    df = df.copy()
    df[numeric] = df[numeric].fillna(df[numeric].mean())
    df = pd.get_dummies(df, columns=categorical)
    dummy_cols = [
        c for c in df.columns if c.startswith(tuple(f"{p}_" for p in categorical))
    ]
    return df[numeric + dummy_cols], numeric + dummy_cols


def train_and_score(x: pd.DataFrame, y: pd.Series) -> float:
    """Split, fit a logistic regression, and return the held-out ROC-AUC."""
    x_train, x_test, y_train, y_test = train_test_split(
        x, y, test_size=0.2, random_state=SEED
    )
    model = LogisticRegression(max_iter=1000).fit(x_train, y_train)
    return float(roc_auc_score(y_test, model.predict_proba(x_test)[:, 1]))


def main() -> None:
    # The driver: load one dataset, name its columns, run the general functions on it.
    df = load_loans()
    x, feature_cols = build_features(
        df,
        numeric=["loan_amnt", "int_rate", "annual_inc", "dti", "revol_util"],
        categorical=["home_ownership", "purpose"],
    )
    auc = train_and_score(x, df["bad_loan"])
    print(f"rows: {len(df)}")
    print(f"features: {len(feature_cols)}")
    print(f"test AUC: {auc:.3f}")


if __name__ == "__main__":
    main()

Look at what changed and what didn’t. The logic is identical to the notebook: the same fillna(mean) (leak and all), the same one-hot encoding, the same random_state=42 split, the same LogisticRegression(max_iter=1000), and therefore the same test AUC of 0.70. What moved is where the boundaries are. build_features and train_and_score are now names defined in one place, callable from anywhere, each taking its input as an argument rather than reaching into a df that some earlier cell happened to leave in the namespace.

Conversion is also triage, and knowing what to drop is half the skill. A real notebook is full of things that earned their place during exploration and have no place in a module, and the handoff notebook above has all of them: the df.head() and df.describe() cells you used to understand the data, the correlation table, the cells where two feature sets were tried and a third kept, the cross-validation stability check, the model bake-off that justified the choice, the coefficient peek, the 0.5-cutoff crosstab, the single-borrower sanity check, the stray print(df.shape) debugging. None of it is wrong. It’s how the analysis got done. But it’s communication scaffolding, not the pipeline, and carrying it into the module would mean re-running every experiment on every scoring run. The rule is to keep only the lines that transform data on the path from raw input to the trained model, and leave the exploration behind in the notebook (or a separate analysis one), which is exactly the advice the performance literature gives: the moment a function is worth keeping, lift it out of the notebook into a module and leave the looking-around where it belongs. Two notebook-only constructs deserve specific attention because they fail silently when moved. IPython magics (the %%time on the training cell above, %matplotlib inline, !pip install ...) aren’t Python and won’t run under python -m. And relative file paths like pd.read_csv("data/loans.csv") resolve against the notebook’s directory, but against the caller’s working directory once the code is a module. The same path that “just worked” in the notebook becomes a FileNotFoundError under a scheduler launched from elsewhere. Both are reasons the loader belongs behind a function that takes its path explicitly, not a bare top-level read.

One change is more than cosmetic: build_features takes the numeric and categorical column lists as arguments instead of hardcoding the notebook’s loan columns. That’s what turns it from “the loan feature step” into a reusable function that applies to any tabular dataset. The same two functions work on Lending Club, the Adult/Census data your project uses, or anything else. The dataset-specific knowledge (which columns, which label) moves out of the function and into the caller. The notebook’s “run the feature cell, then the X/y cell, then the training cell, in that order” coupling is gone, because there are no cells. There are functions, and a function brings its inputs with it.

Notice what was not fixed. build_features still imputes with the full-frame mean, so the leak rode along into the package untouched. And that’s correct for this step. The package’s job here is to reproduce the notebook’s behavior in a shape you can build on. Change the model while changing the structure and you could no longer tell a conversion bug from a modeling change. The AUC printing 0.70, identical to the notebook, is the proof the move was clean. The leak is now sitting in a single named function where M3 can find it and fix it in one place, which is itself an argument for the conversion: a bug spread across notebook cells is a bug you can’t fix once.

The notebook trained the same model and got the same AUC. Why is moving that work into functions, rather than leaving the cells that already work, the thing that makes it deployable, testable, and retrainable?

Move the run behind an entry point

The functions are importable, but the notebook still has a second habit baked in: it does the work the moment it’s read. In a module that’s a trap. Importing the file to borrow build_features would also run the split and the training. The previous lesson’s if __name__ == "__main__": guard is exactly the fix, and it’s worth watching the mechanism that makes it work.

"""Lesson 1.2 Show — what `__main__` means and why the guard matters.

`__name__` is `"__main__"` only when the file is run directly; on import it is the
module's name. The `if __name__ == "__main__":` guard is what lets `python -m pkg`
run the scorer end to end while `import pkg` stays silent — the same module is both
importable and runnable, with no work firing just because someone imported it.
"""


def run_scorer() -> str:
    return "scorer ran"


print(f"__name__ at import/run time is: {__name__!r}")

if __name__ == "__main__":
    # Only this branch fires under `python 02_main_entry_point.py` or `python -m`.
    print(run_scorer())

Run that file directly and it prints '__main__'. The same top-level line, reached by an import instead, would print the module’s dotted name. That asymmetry is the whole point. When Python runs a file directly, it sets that module’s __name__ to "__main__", so the guarded block fires. When something imports the file, __name__ is the real module name, the guard is false, and the block stays quiet. You get the file’s functions without its job running. Keep your definitions at the top level and your run-calls inside the guard, and the same file is safe to import for its parts and runnable as a program: the two things the notebook couldn’t be at once.

This is the move that retires the notebook’s worst handoff property. In the notebook, there was no way to reuse one piece without running everything. Here, from yourpkg import build_features costs you nothing, and python -m yourpkg does the full load-features-train-score run on purpose.

Run it as a package

With the logic in modules and the run behind the guard, the converted package is a real one. The importable core is the two dataset-agnostic functions. __main__ is a thin driver that loads one dataset, names its columns, and runs them, so python -m yourpkg builds features, trains the model, and prints the same 0.70 AUC the notebook did, while from yourpkg import build_features gives you the reusable function with nothing firing. That’s the deliverable the data scientist thought they already had, and the difference between the two is everything this lesson did.

The matching AUC isn’t a nice-to-have. It’s the test that the conversion was correct, and it’s worth being disciplined about how you check it. This kind of move, changing the structure of the code without changing what it computes, is a refactor, and the only way to know a refactor was clean is to compare the output before and after against a fixed input. Here the comparison is concrete: the notebook’s random_state=42 and the package’s random_state=42 mean the same split, the same fit, and therefore a number you can match digit for digit, not “roughly 0.70.” If the package prints 0.70 and the notebook printed 0.70, the move preserved behavior. If it prints 0.68, you didn’t find a better model, you introduced a conversion bug: a column dropped in the move, a different fill, a split that no longer lines up. The gap is pointing right at it. This is why the leak was carried forward untouched. Change the model and the structure in one step and you’re left unable to tell which change moved the number. One change at a time, with a fixed seed and a number you can diff, is the whole safety net of a refactor, and it’s the same net you’ll rely on every time you touch working ML code for the rest of this curriculum.

This exact package is published on GitHub. You can pip install git+https://github.com/adutchengineer/ml-pipeline-starter and from ml_pipeline import build_features, train_and_score in any project. That’s the proof the conversion worked: the notebook’s logic became a library anyone can install and apply to their own data, instead of a file trapped in one kernel. (The source is at github.com/adutchengineer/ml-pipeline-starter.) The same install also gives you from ml_pipeline.datasets import load_loans — the loan data these lessons run on, fetched from its source rather than a file you have to find. That is the data-access seam a real system keeps separate from its modeling code, and it is what the later lessons load from.

The conversion is the pattern you’ll repeat for the rest of the course. A piece of working-but-fragile code arrives, and the engineering is moving it into a shape where each fragility has a place to go. The notebook is now a package. The lessons that follow take this package and put it under version control, make its imports survive a move, validate its inputs, and pin its environment, each one closing another gap between “it works” and “someone else can rely on it.”