Gate the Model and Version It

My pipeline tested code and data, and it still let me ship a model that was strictly worse. The new model passed every check, auto-promoted, and caught fewer than one in ten actual defaults. Its accuracy looked higher because it predicted “fully paid” more often, and accuracy was the number the gate watched. Then I tried to roll back and found I could not: I had the new model file, but the previous model’s training data, code commit, and hyperparameters had been overwritten by the same script that trained the new one. “Roll back” turned into “retrain from memory and hope,” which ate three days I did not have, and on a small team those were three days nobody else could cover.

The last lesson hardened the data half of the pipeline. The leakage test from that lesson caught a feature like recoveries, non-zero only after a default, before it could read the answer into the model. That stage made the data trustworthy. It did nothing about the model the trusted data trains. A green check that asserts “the data is clean” and “the code runs” still passes a model that has never once identified a default, because nothing in CI compared the new model against anything. This lesson closes that gap with two pieces that are not deferrable even on a three-person team: a model-quality gate that asserts an improvement over a baseline on the metric that survives class imbalance, and a version record that binds every promoted artifact to the data, code, and config that produced it so rollback is a one-line operation instead of a three-day rebuild.

Three stages compose to close it. First, a gate that asserts the right number against an explicit baseline. Second, a version that is the artifact plus its lineage, made tamper-evident by a content hash. Third, a promotion that is a guarded, reversible state transition rather than a file copy. Each stage exists because the one before it is worthless without it: a gate decides whether a model may ship, versioning makes the shipped model addressable and rebuildable, and the gated transition is what versioning makes possible.

A model test asserts behaviour against a baseline, on held-out data

One decision this section turns on is narrow and load-bearing: which single number does the gate assert on? A competent engineer reaches for accuracy, because accuracy is what every tutorial reports and it reads as “how often the model is right.” That instinct is exactly what ratified a strictly-worse model in the incident above. Under class imbalance, where one class is far rarer than the other as defaults are among loans, accuracy, recall, and PR-AUC disagree by enough that the wrong pick passes a model that has never identified a single default.

Watch the wrong model first. Here is a “model test” of the kind that looked green in the incident: it trains, it checks that something came back, and it reports the accuracy.

def test_model_trains() -> None:
    model = train(X_train, y_train)
    assert model is not None          # passes on a model that predicts one class
    acc = model.score(X_test, y_test)
    assert acc > 0.80                 # passes on the do-nothing predictor

Both assertions hold for a model that predicts “fully paid” for every loan. The first holds because train() returns an object, and a degenerate model is still an object. The second holds because the do-nothing predictor scores 1 − default_rate, and on this dataset that is 0.817, comfortably over the 0.80 bar. The test framework reports PASS because no assertion FAILED, and PASS here means “nothing asserted false,” not “model is good.” A test that calls train() and only checks the result is not None asserts nothing about quality at all.

Assert on the metric that survives the class balance, not the one that flatters it

A model gate is not “does training run.” It is “does this model clear a bar chosen in advance, measured on data it never trained on, against an explicit baseline.” The non-obvious part is that the bar and the metric are the real decision. A metric that weights every example equally cannot see a rare class, so under imbalance it rewards a model for ignoring the very thing it was built to catch. The gate has to encode the business cost in the metric, then compare against a known reference (the majority-class predictor, or the current production model’s score) so that “improvement” is a measured gain over something concrete rather than a number that looks big in isolation.

The mechanism is arithmetic, and the arithmetic is why the instinct fails. Accuracy is

$$\text{accuracy} = \frac{TP + TN}{TP + TN + FP + FN}$$

where $TP$ and $TN$ are correct positive and negative predictions and $FP$, $FN$ are the two error types. When the positive class is rare, the $TN$ term is large and dominates the ratio, so the rare positives barely move the score. A predictor that always says “fully paid” gets every negative right and every positive wrong, scoring exactly $1 - \text{positive rate}$. Recall drops the $TN$ term entirely:

$$\text{recall} = \frac{TP}{TP + FN}$$

It asks only “of the actual defaults, how many did we catch?” The do-nothing model catches zero, so its recall is $0$ and it cannot hide. That single change, dropping the term that imbalance inflates, is the whole reason recall sees what accuracy cannot.

Below are the numbers the gate would actually compute on this dataset, so run them before trusting either metric. Watch the accuracy of the do-nothing model clear a bar that its recall fails cold.

import numpy as np


def confusion_counts(y_true: np.ndarray, y_pred: np.ndarray) -> dict[str, int]:
    tp = int(np.sum((y_pred == 1) & (y_true == 1)))
    tn = int(np.sum((y_pred == 0) & (y_true == 0)))
    fp = int(np.sum((y_pred == 1) & (y_true == 0)))
    fn = int(np.sum((y_pred == 0) & (y_true == 1)))
    return {"tp": tp, "tn": tn, "fp": fp, "fn": fn}


def accuracy(c: dict[str, int]) -> float:
    return (c["tp"] + c["tn"]) / sum(c.values())


def recall(c: dict[str, int]) -> float:
    denom = c["tp"] + c["fn"]
    return c["tp"] / denom if denom else 0.0


def main() -> None:
    # Lending Club default rate ~0.183 -- same loader the M3/M4 lessons use.
    # Place exactly 18.3% positives so the do-nothing accuracy is exactly 1 - 0.183,
    # rather than a sampled rate that lands a few thousandths off.
    rng = np.random.default_rng(0)
    n = 10_000
    n_pos = round(0.183 * n)  # 1830
    y_true = np.zeros(n, dtype=int)
    y_true[:n_pos] = 1
    rng.shuffle(y_true)

    # The degenerate model: always predict "fully paid" (0).
    y_donothing = np.zeros(n, dtype=int)
    c = confusion_counts(y_true, y_donothing)
    print("do-nothing accuracy:", round(accuracy(c), 3))
    print("do-nothing recall:  ", round(recall(c), 3))


if __name__ == "__main__":
    main()

The do-nothing model scores about 0.817 accuracy and 0.0 recall on the same predictions. A gate on accuracy with a bar near 0.80 blesses it; a gate on recall rejects it without ambiguity. The accuracy figure is not wrong, because it is precisely $1 - 0.183$; it measures the wrong thing, because the large true-negative count from correctly labelling every non-default drowns the rare positives the model exists to find. Why does the recall denominator make this model impossible to flatter, and what would the recall of an “always predict default” model be instead?

Any real model must clear a floor: the majority-class predictor that always returns the most prevalent class, whose accuracy equals that class’s prevalence. On this dataset that floor is 0.817. A logistic model on the real features lands around 0.70 ROC-AUC, which is a genuine improvement in ranking defaults even though its accuracy may sit near the same 0.817 floor: two models can share one accuracy while differing sharply in how well they separate the classes. ROC-AUC, the area under the curve of true-positive rate against false-positive rate, collapses that ranking quality across every threshold into one number, so it does not depend on the single chosen cutoff. That threshold-freedom is also its limit: it says nothing about how the model behaves at the one operating point it will actually serve at. It is the right gate metric and the wrong only metric.

Each candidate metric is honest in a different regime, which is why the choice is a decision and not a default. Compare them before picking one for the gate.

The gate, then, is an explicit assertion: compute the chosen metric on held-out data, compare it against a recorded baseline, and fail the build if the candidate does not beat it. This is the same imbalance-aware metric the foundations module established for this dataset, now wired as a precondition for shipping rather than a number printed at the end of training. Without that comparison the gate asserts nothing, and a gate that asserts nothing is the green check that let the worse model through.

import numpy as np


def accuracy(y_true: np.ndarray, y_pred: np.ndarray) -> float:
    return 0.0  # placeholder -- replace


def recall(y_true: np.ndarray, y_pred: np.ndarray) -> float:
    return 0.0  # placeholder -- replace


def main() -> None:
    rng = np.random.default_rng(0)
    n = 10_000
    y_true = (rng.random(n) < 0.183).astype(int)

    # Candidate: catches a few defaults, but mostly predicts "fully paid".
    # It flags as default only when a noisy score crosses a high cutoff.
    score = rng.random(n) + 0.15 * y_true
    y_pred = (score > 0.95).astype(int)

    print("accuracy:", round(accuracy(y_true, y_pred), 3))
    print("recall:  ", round(recall(y_true, y_pred), 3))


if __name__ == "__main__":
    main()

import numpy as np


def accuracy(y_true: np.ndarray, y_pred: np.ndarray) -> float:
    return float(np.mean(y_true == y_pred))


def recall(y_true: np.ndarray, y_pred: np.ndarray) -> float:
    tp = int(np.sum((y_pred == 1) & (y_true == 1)))
    fn = int(np.sum((y_pred == 0) & (y_true == 1)))
    denom = tp + fn
    return tp / denom if denom else 0.0


def main() -> None:
    rng = np.random.default_rng(0)
    n = 10_000
    y_true = (rng.random(n) < 0.183).astype(int)
    score = rng.random(n) + 0.15 * y_true
    y_pred = (score > 0.95).astype(int)

    print("accuracy:", round(accuracy(y_true, y_pred), 3))
    print("recall:  ", round(recall(y_true, y_pred), 3))


if __name__ == "__main__":
    main()

A model version is the artifact PLUS what produced it

The gate from the last section decides whether a model may ship. That decision is worthless if nothing can later say which model passed, or rebuild it after the data and code have moved on. The mid-level mental model here is that a model is a .pkl on disk: the file is the model, and having the file means having the model. That belief holds right up until an incident, when the rollback attempt discovers that “I have the file” does not mean “I can recreate this model.”

Here is the breaking condition, concrete: the training script from the incident, the one that overwrote its own history.

def retrain_and_save() -> None:
    df = load_latest_training_data()      # whatever the table holds today
    model = fit(df, params=current_params)
    joblib.dump(model, "model.pkl")       # overwrites the previous model in place
    # the data snapshot, the params, and the code commit are not recorded anywhere

This script survives a rollback request as a file and fails it as a version. The .pkl is there, but the previous model’s data snapshot, code commit, and hyperparameters were overwritten by the same run that trained the new one. The model.pkl still loads and runs, yet nothing answers “what produced the model that was serving last week?”, and without that answer, rollback is reconstruction.

A version is the artifact bound to the inputs that produced it, made tamper-evident by a content hash

A trained model is a deterministic function of its inputs:

$$\text{model} = f(\text{training_data},\ \text{code},\ \text{config},\ \text{seed})$$

The artifact is only $f$'s output. Storing the output alone severs the function from its inputs: $f$ can still be evaluated on new data, but it cannot be reproduced, audited for how it was built, or rebuilt once the inputs have moved. A version is the artifact plus a manifest recording those four inputs. A content hash of the training data, a digest computed from the bytes where the same bytes always produce the same digest, turns the binding into a verifiable claim (“trained on dataset a1b2…”) instead of a filename taken on trust.

Each input fails differently when it is missing, which is why the manifest needs all four rather than a convenient subset. Lose the data hash and there is no proof of which snapshot trained the model, and ML training data mutates constantly: a nightly retrain reads a different table than yesterday’s. Lose the code commit and a since-changed feature transform silently produces a different model from the “same” rebuild, with no error to warn you. Lose the config and the rebuild lands at a different point in model space because the hyperparameters that shaped the fit are gone. Lose the seed and even byte-identical inputs give a different fit for any stochastic learner, because the random shuffle and initialisation differ run to run. The manifest is the set of four things without which “reproduce version 3” degrades to “retrain from memory.”

What makes the data binding trustworthy rather than merely recorded is the content hash. A filename can be reused; bytes cannot lie. Here is the manifest written next to the artifact, with the data hash computed from the training bytes.

import hashlib
import json
from dataclasses import dataclass, field


def data_hash(raw_bytes: bytes) -> str:
    return hashlib.sha256(raw_bytes).hexdigest()


@dataclass
class ManifestSpec:
    """The inputs a model manifest records — one object, not a 7-arg parameter list."""

    artifact_path: str
    training_bytes: bytes
    git_commit: str
    metric_name: str
    metric_value: float
    params: dict[str, object] = field(default_factory=dict)
    seed: int = 0

    def to_manifest(self) -> dict[str, object]:
        return {
            "artifact": self.artifact_path,
            "data_sha256": data_hash(self.training_bytes),
            "git_commit": self.git_commit,
            "params": self.params,
            "metric": {self.metric_name: round(self.metric_value, 4)},
            "seed": self.seed,
        }


def main() -> None:
    # Two "training datasets": identical bytes, then one byte changed.
    data_v1 = b"loan_id,annual_inc,default\n1,52000,0\n2,18000,1\n"
    data_v1_again = b"loan_id,annual_inc,default\n1,52000,0\n2,18000,1\n"
    data_v2 = b"loan_id,annual_inc,default\n1,52000,0\n2,18500,1\n"

    m1 = ManifestSpec(
        "model_v1.pkl", data_v1, "9f3a1c", "roc_auc", 0.70, {"C": 1.0}
    ).to_manifest()
    m2 = ManifestSpec(
        "model_v2.pkl", data_v2, "9f3a1c", "roc_auc", 0.71, {"C": 1.0}
    ).to_manifest()

    print("v1 data hash:", m1["data_sha256"][:12])
    print("same bytes  :", data_hash(data_v1_again)[:12], "(identical)")
    print("one byte chg:", m2["data_sha256"][:12], "(different)")
    print(json.dumps(m1, indent=2))


if __name__ == "__main__":
    main()

The hash of identical bytes matches and the hash of one-byte-changed data does not, which is exactly the property that makes “trained on dataset X” a checkable claim. The manifest sits next to the .pkl as a small JSON record, and it costs nothing to write: a few fields at the end of the training run. Why does hashing the bytes, rather than recording the filename or a row count, catch a silently-edited training set that a filename never would?

This is the one piece of rollout machinery not to defer on a small team, and the reasoning is about which property comes for free. The three-day rollback bit at a size of three people, and it will bite at any size; the property that fixes it, artifact bound to lineage, is available from a JSON manifest written next to the .pkl on the very first promoted model. Reproducibility needs no hosted server or platform team. What changes with scale is only where the manifest is stored, not whether one is produced.

A registry such as MLflow stores versioned models and links each version to the run that produced it (params, metrics, source commit, and artifact), giving an addressable, immutable version-with-lineage rather than a bare file. It is the natural home for the manifest once the directory stops scaling. The dependency lock from the packaging module already pins the code-and-dependency half of reproducibility; the manifest is the data-and-config half. Together they are what makes the next section possible: a promotion that moves between addressable versions instead of overwriting a file.

import hashlib

# `hashlib` is the tool the solution needs; the starter does not use it yet.
# Referenced here so the import stays at the top and the file lints clean.
_ = hashlib


def build_manifest(
    training_bytes: bytes, git_commit: str, auc: float
) -> dict[str, object]:
    # BUG: the hash is hardcoded, so every model claims the same data lineage.
    return {
        "data_sha256": "deadbeef",  # placeholder -- replace with a real hash
        "git_commit": git_commit,
        "metric": {"roc_auc": round(auc, 4)},
    }


def main() -> None:
    data_a = b"loan_id,annual_inc\n1,52000\n"
    data_b = b"loan_id,annual_inc\n1,99000\n"

    ma = build_manifest(data_a, "9f3a1c", 0.70)
    mb = build_manifest(data_b, "9f3a1c", 0.71)
    print("a:", ma["data_sha256"][:12])
    print("b:", mb["data_sha256"][:12])
    print("distinct lineage:", ma["data_sha256"] != mb["data_sha256"])


if __name__ == "__main__":
    main()

import hashlib


def build_manifest(
    training_bytes: bytes, git_commit: str, auc: float
) -> dict[str, object]:
    return {
        "data_sha256": hashlib.sha256(training_bytes).hexdigest(),
        "git_commit": git_commit,
        "metric": {"roc_auc": round(auc, 4)},
    }


def main() -> None:
    data_a = b"loan_id,annual_inc\n1,52000\n"
    data_b = b"loan_id,annual_inc\n1,99000\n"

    ma = build_manifest(data_a, "9f3a1c", 0.70)
    mb = build_manifest(data_b, "9f3a1c", 0.71)
    print("a:", ma["data_sha256"][:12])
    print("b:", mb["data_sha256"][:12])
    print("distinct lineage:", ma["data_sha256"] != mb["data_sha256"])


if __name__ == "__main__":
    main()

Promotion is a gated state transition, not a copy

Promotion is a step in a process, moving a model from “trained” to “serving,” and the cost of skipping the step’s structure is exactly the opening incident. The mid-level model is that promotion is a copy: the new model is good, so move it into the production location. cp model.pkl prod/ looks like the step. It omits the two things that make the step safe.

Watch the copy fail at both of them.

# "Promote" the new model.
shutil.copy("model.pkl", "prod/model.pkl")   # 1: asks no questions — the gate is never enforced
                                              # 2: overwrites prod/model.pkl — rollback has no target

A copy asks no questions, so the quality gate from the first section is never enforced at the moment that matters: a strictly-worse model copies as cleanly as a better one. And the copy overwrites the file that was serving, so “go back” has no target. Both failures are silent at copy time and both surface during an incident, which is the worst moment to learn that promotion was never guarded and rollback was never possible.

Promotion is a guarded state transition, and rollback is the same transition run backward

Moving a model to serving is a deliberate, reversible state change guarded by the model gate, not a file copy. A staged registry replaces the copy with a state machine in which the only legal path into production runs through the gate, and the previously-serving version is moved aside rather than destroyed. That is what turns rollback into an operation that exists instead of a rebuild you attempt under pressure.

The load-bearing indirection is the production alias: a mutable, named reference that points at a particular version. Callers never request a filename or a version number; they request “the production model,” and the registry resolves that alias to whatever version it currently points at. Promotion is therefore a single atomic operation: re-point the alias. No moment exists where two versions are half-live, and no file is destroyed. Rollback is the identical operation in the other direction, re-pointing the alias at the archived version. Without the state machine, “roll back to last week’s model” is “reconstruct last week’s model,” which is the three-day incident.

The state machine has three lifecycle states and a guarded edge between them. Step through it before reading the transition rules, because the structure forces the rules.

Walk the legal transitions as a process and the rules are forced rather than chosen. The staging → production edge is guarded: the only way across is a promote(version) call that requires the version to have cleared the gate first; skip the guard and you have a copy command again. The same call moves the previously-production version to archived rather than deleting it, and that step is what gives rollback a target. The archived → production rollback is not a new mechanism; it is the same alias re-point in reverse against a registry that kept the old version intact.

Here is the state machine as code, with the gate enforced as a precondition and the prior version archived rather than overwritten. Watch the promote call refuse a model that has not cleared the gate, and watch rollback recover the exact archived version.

class ModelRegistry:
    def __init__(self) -> None:
        self.versions: dict[str, dict[str, object]] = {}
        self.alias: dict[str, str] = {}  # alias name -> version id
        self.archived: list[str] = []

    def register(self, version_id: str, roc_auc: float) -> None:
        self.versions[version_id] = {"roc_auc": roc_auc, "stage": "staging"}

    def gate_passes(self, version_id: str, baseline_auc: float) -> bool:
        # The precondition: a candidate must beat the current champion's metric.
        return self.versions[version_id]["roc_auc"] > baseline_auc

    def promote(self, version_id: str, baseline_auc: float) -> str:
        if not self.gate_passes(version_id, baseline_auc):
            return (
                "BLOCKED " + version_id + ": did not beat baseline " + str(baseline_auc)
            )
        prior = self.alias.get("production")
        if prior is not None:
            self.versions[prior]["stage"] = "archived"  # moved aside, not deleted
            self.archived.append(prior)
        self.versions[version_id]["stage"] = "production"
        self.alias["production"] = version_id  # atomic re-point
        return "PROMOTED " + version_id + " (archived " + str(prior) + ")"

    def rollback(self) -> str:
        if not self.archived:
            return "no archived version to roll back to"
        target = self.archived.pop()
        current = self.alias["production"]
        self.versions[current]["stage"] = "archived"
        self.versions[target]["stage"] = "production"
        self.alias["production"] = target  # same re-point, backward
        return "ROLLED BACK to " + target


def main() -> None:
    reg = ModelRegistry()
    reg.register("v3", roc_auc=0.70)
    reg.promote("v3", baseline_auc=0.65)  # first champion
    reg.register("v4_bad", roc_auc=0.62)
    print(reg.promote("v4_bad", baseline_auc=0.70))  # the incident model -- blocked
    reg.register("v4_good", roc_auc=0.73)
    print(reg.promote("v4_good", baseline_auc=0.70))  # passes, v3 archived
    print("production now:", reg.alias["production"])
    print(reg.rollback())  # re-point back to v3
    print("production now:", reg.alias["production"])


if __name__ == "__main__":
    main()

The strictly-worse v4_bad is refused by promote because the gate is a precondition, not a step someone might remember to run. The good model promotes and archives v3 in the same call, and rollback re-points the alias to the archived v3 in one line, an operation that exists only because v3 was kept. What in this code makes rollback impossible if promote had instead overwritten the production entry, and which line is the difference?

Skipping this structure costs asymmetrically and only shows up under pressure. A missing guard lets a bad model into production silently, and a missing archive turns a one-line rollback into a from-memory rebuild, both during an incident. That asymmetry is the whole argument: the structure costs a few lines at promotion time and saves three days at the worst possible moment.

class Registry:
    def __init__(self) -> None:
        self.production: str | None = None
        self.archived: list[str] = []

    def promote(self, version_id: str) -> str:
        # BUG: overwrites production without keeping the prior version.
        self.production = version_id
        return "PROMOTED " + version_id

    def rollback(self) -> str:
        # BUG: nothing was ever archived, so there is no target.
        return "no target"  # placeholder -- make rollback work


def main() -> None:
    reg = Registry()
    reg.promote("v3")
    reg.promote("v4")
    print("production:", reg.production)
    print(reg.rollback())
    print("production:", reg.production)


if __name__ == "__main__":
    main()

class Registry:
    def __init__(self) -> None:
        self.production: str | None = None
        self.archived: list[str] = []

    def promote(self, version_id: str) -> str:
        if self.production is not None:
            self.archived.append(self.production)  # keep the prior champion
        self.production = version_id
        return "PROMOTED " + version_id

    def rollback(self) -> str:
        if not self.archived:
            return "no target"
        target = self.archived.pop()
        if self.production is not None:
            self.archived.append(self.production)
        self.production = target
        return "ROLLED BACK to " + target


def main() -> None:
    reg = Registry()
    reg.promote("v3")
    reg.promote("v4")
    print("production:", reg.production)
    print(reg.rollback())
    print("production:", reg.production)


if __name__ == "__main__":
    main()

Summary

• A model gate asserts an improvement over an explicit baseline on a metric that survives class imbalance: accuracy reports 1 − default_rate (0.817 here) for a do-nothing model, so the gate must use recall or ROC-AUC, which drop the dominating true-negative term.
• A test framework reports PASS when no assertion FAILS; a “test” that calls train() and checks the result is not None is green on a degenerate model. The gate must compare against a recorded baseline or it asserts nothing.
• A model version is the artifact plus a manifest of the four inputs that produced it (data hash, code commit, config, seed), because each one fails differently when missing and model = f(data, code, config, seed) cannot be rebuilt from the output alone.
• A content hash of the training bytes makes “trained on dataset X” a verifiable claim rather than a filename you trust; the same bytes always hash the same, changed bytes never do.
• Promotion is a guarded, reversible state transition, not a copy: the production alias re-points atomically, the prior champion is archived not deleted, and rollback is the identical re-point backward, which is why it is one line instead of a three-day rebuild.

Check your understanding:

• Without looking back: a candidate model scores 0.83 accuracy and 0.05 recall on a dataset with an 0.183 default rate. Should the gate pass it, and which number tells you?
• What four inputs must the manifest record, and what specifically breaks at rebuild time if the seed is the one that is missing?
• Promotion overwrote the production file instead of archiving the prior version. The new model is bad. Walk the exact sequence that turns “roll back” into “rebuild from memory”: which step was skipped, and where?