Retraining: The Loop That Can Hurt You

I wired a drift monitor straight into a retrain once, and it taught my model a broken upstream feed as if it were the truth. The monitor fired on a feature whose distribution had moved, so the loop did the obvious thing: it retrained on the recent window, ran the model-quality gate from the last lesson, watched the gate pass, and auto-promoted. The retrained model was strictly worse than the one it replaced. The shift was not the world changing. An upstream feed had started reporting a money field in cents instead of dollars, and the loop had treated that corruption as ground truth. The gate could not catch it, because the held-out set it judged against was carved from the same corrupted window, so the model scored well on exactly the corruption it had just learned. By the time I read the dashboards, the bad model’s own predictions had begun seeding the next training set, and the loop had compounded the bug at machine speed across several generations before I looked.

That loop is the most dangerous thing in this module. The previous lessons built the pieces it runs on: a green CI pipeline (Lesson 1), data tests that fail the build on a bad-data commit (Lesson 2), and a model-quality gate plus a versioned registry with one-line rollback (Lesson 3). M10 added drift detection: the PSI and KS detectors that watch the live input distribution against a reference window, the signal that catches the loan-default scorer decaying when its inputs walk away from training. The seductive wiring connects them directly: drift → retrain → gate → auto-promote. This lesson is about why that exact wiring is the single most destructive thing you can build, and what belongs between each arrow. We start by treating a drift alarm as a hypothesis instead of an order, then walk the poisoning loop one step at a time to see why the gate ratifies the bug instead of catching it, and finally write the runbook whose real content is the list of conditions under which you do not retrain.

Three load-bearing terms carry the argument, so each is grounded on first use here even if an earlier lesson defined it. Drift is the input distribution having moved relative to a reference window. The gate is Lesson 3’s check that a candidate model must clear a chosen metric over a baseline on held-out data before it is allowed to serve. A held-out set is data the model never trained on, used to judge it. A retrain trigger is the event, a drift alarm or a schedule, that starts a new training run. The whole lesson turns on what happens between the trigger and the gate.

Drift is a trigger, not a verdict

The mental model that ships the incident is the one a competent engineer holds on first contact with drift monitoring: the distribution moved, so the world changed, so retrain on the new data. It reads as a closed syllogism. It is not, because the first step is wrong: a drift alarm does not tell you the world changed. It tells you two distributions are far apart, and a broken upstream feed produces exactly the same distance as a genuine shift in the population. The two causes demand opposite actions, and the alarm cannot tell you which one you have.

Treat a drift alarm as a hypothesis to diagnose, not an instruction to obey

A drift detector measures that the input distribution moved; it is structurally incapable of telling you why. The reason is mechanical: the common drift statistics are functions of only the two distributions, so any two inputs that produce the same distributional gap produce the same number. PSI (Population Stability Index) bins the reference distribution into fixed buckets, records the fraction of reference data in each bin, drops the current data into those same bins, and sums a per-bin term that grows with both the size of the move and the log-ratio of the two fractions, large whenever mass relocated between bins, blind to why it moved. KS (Kolmogorov–Smirnov) takes the maximum vertical gap between the two cumulative distribution functions, point by point, again a pure measure of how far apart two distributions are, with no notion of cause. A real income shift and a units bug both relocate mass across the same bins, so they produce the same elevated statistic. That blindness is not a limitation to engineer around; it is what a distance measure is.

The standard PSI term per bin is below. The book corpus confirms the mechanism (bins, then the log of the divided percentages), so this is presented as the standard PSI definition, not as a quoted closed form.

$$\text{PSI} = \sum_{i=1}^{B} \left( c_i - r_i \right) \cdot \ln!\left( \frac{c_i}{r_i} \right)$$

Here $B$ is the number of bins, $r_i$ is the fraction of the reference distribution that fell in bin $i$, and $c_i$ is the fraction of the current window in that same bin. Every input to that sum is a property of the two distributions and nothing else: there is no term for why $c_i$ differs from $r_i$. A feed that started reporting cents as dollars moves mass into higher bins; a real surge in applicant incomes moves mass into the same higher bins. Same $c_i$, same PSI, opposite root cause.

So the diagnostic step that belongs between the alarm and the retrain is the one question the statistic cannot answer: is the new data trustworthy? The diagnostic is not a bigger model and not a longer reference window; it is Lesson 2’s data contract, run against the recent window. Check the raw min, max, dtype, and declared ranges of the drifted feature: a real shift stays inside the contract (incomes moved but are still plausible positive dollars), a units corruption blows through it (incomes are 100× too large, or negative). That single check routes the alarm to the right response.

The two responses are genuinely opposite, which is why collapsing them into one unconditional retrain is the bug. The choice is below.

The named failure mode is the incident that opened this lesson: a drift monitor fired on a feature whose distribution had “shifted,” the shift was an upstream unit change rather than customer behaviour, and retraining on it would have baked the corruption in. The boundary that was violated is the assumption that drift implies the world changed: the alarm crossed from “the data is different” to “the data is correct-but-different” with nothing checking the difference. The diagnostic that separated the two causes was checking the raw min and max against the Lesson 2 contract, not a bigger model and not a longer reference window.

Here is the mechanism made concrete. Two completely different events, a real shift in the applicant population and a cents-as-dollars corruption, are fed to the same PSI computation, and the statistic comes out elevated for both, proving it cannot distinguish them. Watch the two PSI values land in the same range while only the contract check separates the causes.

import numpy as np


def psi(ref: np.ndarray, cur: np.ndarray, n_bins: int = 10) -> float:
    # Fixed bin edges from the reference distribution's quantiles.
    edges = np.quantile(ref, np.linspace(0, 1, n_bins + 1))
    edges[0], edges[-1] = -np.inf, np.inf
    r = np.histogram(ref, edges)[0] / len(ref)
    c = np.histogram(cur, edges)[0] / len(cur)
    r = np.clip(r, 1e-6, None)  # avoid log(0) / divide-by-zero
    c = np.clip(c, 1e-6, None)
    return float(np.sum((c - r) * np.log(c / r)))


def contract_ok(values: np.ndarray, lo: float = 0.0, hi: float = 1_000_000.0) -> bool:
    # Lesson 2's data contract: plausible positive dollars, in range.
    return bool(np.all(values >= lo) and np.all(values <= hi))


def main() -> None:
    rng = np.random.default_rng(7)

    # Reference window: annual_inc in plausible dollars.
    reference = rng.normal(70_000, 20_000, 5_000).clip(10_000, 250_000)

    # Cause A -- a REAL shift: a new higher-income applicant segment enters.
    real_shift = rng.normal(95_000, 22_000, 5_000).clip(10_000, 300_000)

    # Cause B -- a DATA BUG: half the rows reported cents as dollars (100x too large).
    bug = reference.copy()
    corrupt_idx = rng.choice(len(bug), size=len(bug) // 2, replace=False)
    bug[corrupt_idx] = bug[corrupt_idx] * 100  # cents reported as dollars

    print(f"PSI, real shift: {psi(reference, real_shift):.3f}  -> drift fires")
    print(f"PSI, units bug : {psi(reference, bug):.3f}  -> drift fires")
    print()
    print(f"contract holds, real shift: {contract_ok(real_shift)}")
    print(f"contract holds, units bug : {contract_ok(bug)}")


if __name__ == "__main__":
    main()

Both PSI values clear any sane drift threshold: the statistic fired for a real shift and a corruption alike, exactly as a pure distance measure must. The contract check is the only line that disagrees: it stays True for the real shift and flips to False for the units bug, because the corrupted incomes blew past the plausible-dollars ceiling. The detector said “something moved”; the contract said “and one of them is garbage.” Wiring the detector straight to a retrain throws away that second sentence.

Why does PSI fire for the units bug rather than ignoring it as an obvious outlier? Because PSI scores how much bin mass relocated, not whether the raw values are plausible: moving half the rows up by 100× shifts a large fraction of mass into the top bins, which is exactly the kind of move PSI is built to catch. The statistic measures rearrangement, not validity, which is precisely why validity has to be checked separately.

This is the first arrow in the dangerous wiring, and it is where the loop is supposed to branch: the detector’s job is to route to a check or a human, not to fire a retrain. The next section shows what happens when that branch is missing and the corrupted window reaches the trainer.

import numpy as np

# dti is a ratio: in a healthy feed it sits between 0 and ~60 (percent).
recent_window = np.array([12.0, 18.5, 9.0, 33.0, 21.0, 4400.0, 27.5, 15.0])


def diagnose_drift(values: np.ndarray, lo: float = 0.0, hi: float = 60.0) -> str:
    # TODO: check the contract; route to "fix the pipe" or "consider retrain".
    return "TODO: diagnose"  # placeholder so the starter runs


def main() -> None:
    print(diagnose_drift(recent_window))


if __name__ == "__main__":
    main()

import numpy as np

recent_window = np.array([12.0, 18.5, 9.0, 33.0, 21.0, 4400.0, 27.5, 15.0])


def diagnose_drift(values: np.ndarray, lo: float = 0.0, hi: float = 60.0) -> str:
    out_of_range = values[(values < lo) | (values > hi)]
    if out_of_range.size > 0:
        return (
            f"DATA-QUALITY FAILURE: {out_of_range.size} value(s) violate the "
            f"contract [{lo}, {hi}] (e.g. {out_of_range[0]}). "
            "Freeze retraining, fix the upstream feed."
        )
    return "GENUINE SHIFT: all values in contract. A retrain may be warranted."


def main() -> None:
    print(diagnose_drift(recent_window))


if __name__ == "__main__":
    main()

Retraining on poisoned data is worse than not retraining

There is a comforting story about why the dangerous wiring is actually safe: the Lesson 3 gate makes auto-promotion safe, because a bad model would fail the gate, so drift → retrain → gate → promote is self-protecting. The gate is a real quality check that already caught a strictly-worse model once, so trusting it to catch the next one feels earned. It is the most expensive assumption in this module, because the gate that caught the worse model in Lesson 3 is the same gate that ratifies the corrupted model here, and the difference is entirely in where the held-out data came from.

An automated loop with no out-of-loop reference amplifies whatever it is fed

A quality gate only certifies a candidate if the data it judges against is drawn from the distribution you actually care about. The moment the gate’s held-out set comes from the same source as the training data, the gate stops measuring correctness and starts measuring internal consistency, and a self-consistent corruption is maximally internally consistent. The model learns the corrupted pattern from the training split and is then scored against a held-out split that exhibits the same pattern, so it scores high. The gate passes not despite the corruption but because of it. The only thing that breaks the loop is a reference the loop cannot touch: a clean, time-fixed held-out set that the retrain never trains on and never regenerates.

Walk the loop one step at a time, because the trap lives in the sequence, not in any single step.

The mechanism in step three is the whole lesson, so it earns a precise statement. A held-out set gives a valid estimate of generalization only when it is representative of the data the model will actually face. A held-out set carved from a corrupted window is not representative of the real world; it is representative of the corruption. So the gate run against it measures whether the model fits the corruption, which it does perfectly, because that is what it was just trained to do. The pattern repeats the imbalance trap from Lesson 3: the book says a held-out estimate is misleading when the set is unrepresentative; the addition here is the loop manufactures the unrepresentative set itself, so the gate is not merely wrong by accident; it is wrong by construction, and it gets more confidently wrong each cycle.

The compounding in steps four and five is a degenerate feedback loop: a model’s own predictions influence the feedback it later trains on, so the system’s outputs generate its future inputs. With no human input in the loop, the model learns from its own mistakes as well as its correct predictions, and accuracy can decrease across retrain cycles rather than improve. Each cycle runs faster than a human can review, so by the time anyone notices, several generations have each ratified the prior one’s corruption, and the drift back toward a clean reference now looks like more drift to retrain away. This is the non-obvious cost that inverts the usual instinct: a human in the loop is slow precisely where slow is the safe property, because the loop’s speed is what converts a recoverable data incident into a compounding model incident.

Now make the split visible. The corruption modeled here is a self-consistent one: an upstream bug that flipped the sign of a feature across the whole recent window, the kind of systematic error a unit change or a reversed join produces (the specific corruption is illustrative scaffolding, not a sourced threshold). The simulation runs two retrain cycles and prints the in-loop gate passing each cycle while the true accuracy against a fixed clean reference collapses. The gap between “gate says good” and “actually good” is the entire lesson.

import numpy as np
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import accuracy_score

rng = np.random.default_rng(0)

# A fixed CLEAN reference set, frozen in time -- the loop never touches it.
n_ref = 2_000
X_ref = rng.normal(0, 1, (n_ref, 4))
true_w = np.array([1.5, -1.0, 0.8, -0.5])
y_ref = (X_ref @ true_w + rng.normal(0, 0.5, n_ref) > 0).astype(int)


def make_corrupt_window(n: int) -> tuple[np.ndarray, np.ndarray]:
    # Honest labels from the TRUE relationship on clean features...
    X_clean = rng.normal(0, 1, (n, 4))
    y = (X_clean @ true_w + rng.normal(0, 0.5, n) > 0).astype(int)
    # ...but an upstream bug flipped the sign of feature 0 across the whole window.
    # The corruption is consistent, so it is learnable AND present in any split.
    X_seen = X_clean.copy()
    X_seen[:, 0] = -X_seen[:, 0]
    return X_seen, y


def in_loop_gate(model: LogisticRegression, X: np.ndarray, y: np.ndarray) -> float:
    # Hold-out carved from the SAME (corrupted) window.
    split = len(X) // 2
    model.fit(X[:split], y[:split])
    return accuracy_score(y[split:], model.predict(X[split:]))


def main() -> None:
    for cycle in range(1, 3):
        X_win, y_win = make_corrupt_window(1_500)
        model = LogisticRegression(max_iter=500)
        gate = in_loop_gate(model, X_win, y_win)
        # Served on CLEAN production features the model never saw the right sign of.
        true_acc = accuracy_score(y_ref, model.predict(X_ref))
        print(f"cycle {cycle}: in-loop gate={gate:.3f}  true(clean ref)={true_acc:.3f}")


if __name__ == "__main__":
    main()

The in-loop gate score sits high, above 0.9, while true accuracy against the clean reference collapses below 0.5, worse than a coin flip. The gate scores the model against a hold-out whose feature carries the same sign flip the model learned, so the learned-wrong rule looks right; served on correctly-signed production features, the same rule is backwards. Two numbers, both honest measurements, point in opposite directions: the gate is not lying, it is answering a different question than the one being asked. If this loop auto-promotes on the gate alone, it ships the failing model every time and reports the passing number.

Why does the in-loop gate not simply detect that the model is bad, given that it is, after all, measuring accuracy? Because accuracy is only as truthful as the data it is computed against, and the hold-out carries the identical corruption the model trained on. A measurement against a poisoned reference certifies poison.

The fix is one reference and one checkpoint. Add the clean reference and the gate run against it fails the candidate the in-loop gate passed.

import numpy as np
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import accuracy_score


def main() -> None:
    rng = np.random.default_rng(0)

    n_ref = 2_000
    X_ref = rng.normal(0, 1, (n_ref, 4))
    true_w = np.array([1.5, -1.0, 0.8, -0.5])
    y_ref = (X_ref @ true_w + rng.normal(0, 0.5, n_ref) > 0).astype(int)

    # The corrupted recent window: honest labels, but feature 0's sign is flipped.
    n = 1_500
    X_clean = rng.normal(0, 1, (n, 4))
    y_win = (X_clean @ true_w + rng.normal(0, 0.5, n) > 0).astype(int)
    X_win = X_clean.copy()
    X_win[:, 0] = -X_win[:, 0]

    candidate = LogisticRegression(max_iter=500).fit(X_win, y_win)

    # Two gates, two references. The bar: beat a majority-class baseline by a margin.
    in_loop = accuracy_score(y_win, candidate.predict(X_win))
    clean = accuracy_score(y_ref, candidate.predict(X_ref))
    baseline = max(np.mean(y_ref), 1 - np.mean(y_ref))

    print(f"in-loop gate (same window)   : {in_loop:.3f}  -> PASS")
    print(f"clean-reference gate         : {clean:.3f}")
    print(f"baseline (majority predictor): {baseline:.3f}")
    print(f"clean gate verdict: {'PASS' if clean > baseline + 0.05 else 'FAIL'}")


if __name__ == "__main__":
    main()

The clean-reference gate scores the corrupted candidate below the majority baseline and returns FAIL, halting the loop on the exact candidate the in-loop gate waved through. The reference works because it is frozen in time and the retrain pipeline is forbidden from regenerating it, so it stays a valid judge no matter what the recent window does. The second layer is the human-in-the-loop checkpoint: a deliberate review-and-authorize step before promotion, so the loop cannot promote a model trained on its own influenced data without a person signing off.

The named failure mode, walked all the way out: drift from a units corruption triggered the retrain, the new model passed its Lesson 3 gate because the held-out set was also corrupted, it auto-promoted, and its predictions began seeding the next training set. The loop had been built to propagate the bug at machine speed. The boundary it violated is the representativeness assumption every held-out estimate rests on: the gate trusted a reference the loop itself had poisoned. The fix is a clean, time-fixed reference set the loop cannot corrupt, plus a human approval gate before promotion.

import numpy as np
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import accuracy_score

# `accuracy_score` is the tool the solution scores with; referenced here so the
# import stays at the top and the starter lints clean before you wire it in.
_ = accuracy_score

rng = np.random.default_rng(3)

# Clean reference (frozen, never retrained on).
X_ref = rng.normal(0, 1, (1_500, 3))
w = np.array([1.2, -0.9, 0.6])
y_ref = (X_ref @ w + rng.normal(0, 0.4, 1_500) > 0).astype(int)

# Corrupted window the candidate trained on: feature 0's sign is flipped.
X_clean = rng.normal(0, 1, (1_000, 3))
y_win = (X_clean @ w + rng.normal(0, 0.4, 1_000) > 0).astype(int)
X_win = X_clean.copy()
X_win[:, 0] = -X_win[:, 0]
candidate = LogisticRegression(max_iter=500).fit(X_win, y_win)


def clean_gate(
    model: LogisticRegression,
    X_ref: np.ndarray,
    y_ref: np.ndarray,
    margin: float = 0.05,
) -> bool:
    # TODO: score against the CLEAN reference, compare to the majority baseline.
    return True  # placeholder so the starter runs (and is wrong on purpose)


def main() -> None:
    print("promote?", clean_gate(candidate, X_ref, y_ref))


if __name__ == "__main__":
    main()

import numpy as np
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import accuracy_score

rng = np.random.default_rng(3)

X_ref = rng.normal(0, 1, (1_500, 3))
w = np.array([1.2, -0.9, 0.6])
y_ref = (X_ref @ w + rng.normal(0, 0.4, 1_500) > 0).astype(int)

X_clean = rng.normal(0, 1, (1_000, 3))
y_win = (X_clean @ w + rng.normal(0, 0.4, 1_000) > 0).astype(int)
X_win = X_clean.copy()
X_win[:, 0] = -X_win[:, 0]
candidate = LogisticRegression(max_iter=500).fit(X_win, y_win)


def clean_gate(
    model: LogisticRegression,
    X_ref: np.ndarray,
    y_ref: np.ndarray,
    margin: float = 0.05,
) -> bool:
    score = accuracy_score(y_ref, model.predict(X_ref))
    baseline = max(np.mean(y_ref), 1 - np.mean(y_ref))
    print(
        f"clean score={score:.3f}  baseline={baseline:.3f}  bar={baseline + margin:.3f}"
    )
    return bool(score > baseline + margin)


def main() -> None:
    print("promote?", clean_gate(candidate, X_ref, y_ref))


if __name__ == "__main__":
    main()

The runbook: when not to retrain

A drift trigger just fired. Retrain, or hold? The naive runbook answers this by documenting how to retrain: the steps to pull the recent window, fit, gate, promote. That runbook retrains into every trap in this lesson, because it has encoded the action without the gate on the action. The decision the trigger actually demands is restraint, and restraint has to be written down as explicit pre-conditions or it does not happen at machine speed.

A retraining policy is mostly a list of conditions under which you hold

The point of a runbook is not the steps to retrain; it is the explicit pre-conditions that must be true before the trigger is allowed to act. The judgment being tested is restraint, and the asymmetry is what makes restraint the staff move: the cost of a needless retrain is a possible regression you then have to detect and roll back, while the cost of skipping a needless retrain is usually nothing, because the existing model keeps working. A mature retraining strategy establishes explicit policies for when to retrain rather than retraining reflexively on every drift alarm; automated retraining is a maturity stage reached deliberately, gated by policy, not the default wiring you start with.

Each no-retrain condition is a different way the recent data fails to justify a retrain, and each maps to a concrete pre-condition you check before the trigger fires. The three below are not three tools; they are three independent guards, any one of which can veto the retrain.

The runbook converts each guard into a check the trigger must clear. should_retrain(trigger) returns False, with a printed reason, when the minimum-sample gate fails, when the drift is attributable to a known bad feed, or when the label-latency window has not closed. The trigger becomes a retrain only when none of the guards object. Automated data validation and model validation are required stages here, not optional bolt-ons; they are what gate the pipeline before it trains and promotes the next version.

The decision function below encodes all three guards on a Lending Club-shaped trigger. Each guard prints why it held, so the runbook is auditable after the fact.

from dataclasses import dataclass


@dataclass
class Trigger:
    new_labeled_examples: int
    new_minority_examples: int
    drift_cause: str  # "real_shift" | "data_quality_failure" | "unknown"
    incident_active: bool
    labels_settled: bool  # has the label-latency window closed?


def should_retrain(t: Trigger, min_minority: int = 50) -> bool:
    # Guard 1 -- minimum-sample gate (the rare class is what we must learn).
    if t.new_minority_examples < min_minority:
        print(
            f"HOLD: only {t.new_minority_examples} new defaults (< {min_minority}); would overfit noise."
        )
        return False
    # Guard 2 -- freeze during incident / known bad feed.
    if t.incident_active or t.drift_cause == "data_quality_failure":
        print(
            "HOLD: drift attributable to a bad feed / active incident; freeze and fix the pipe."
        )
        return False
    # Guard 3 -- label-latency gate.
    if not t.labels_settled:
        print(
            "HOLD: label-latency window not closed; recent labels are unsettled guesses."
        )
        return False
    print("RETRAIN: no guard objects.")
    return True


def main() -> None:
    print("too few defaults:")
    should_retrain(Trigger(800, 22, "real_shift", False, True))
    print()
    print("bad feed:")
    should_retrain(Trigger(5_000, 900, "data_quality_failure", False, True))
    print()
    print("unsettled labels:")
    should_retrain(Trigger(5_000, 900, "real_shift", False, False))
    print()
    print("all clear:")
    should_retrain(Trigger(5_000, 900, "real_shift", False, True))


if __name__ == "__main__":
    main()

Only one of the four triggers survives every guard, so the default outcome of a drift alarm is now “do nothing,” not “retrain”: three triggers hold, and only the last (enough minority examples, a real shift, no incident, settled labels) clears every guard and becomes a retrain. That inverted default is the correct one for a loop that runs faster than anyone watching it. The runbook is where Section 1’s diagnosis becomes a written, checkable policy and Section 2’s out-of-loop discipline becomes a refusal the loop honors.

from dataclasses import dataclass


@dataclass
class Trigger:
    new_minority_examples: int
    drift_cause: str
    labels_settled: bool


def guard_min_sample(t: Trigger) -> tuple[bool, str]:
    # TODO: return False with a reason when there are too few minority examples.
    return True, "ok"  # placeholder


def guard_bad_feed(t: Trigger) -> tuple[bool, str]:
    return True, "ok"  # placeholder


def guard_label_latency(t: Trigger) -> tuple[bool, str]:
    return True, "ok"  # placeholder


def main() -> None:
    t = Trigger(
        new_minority_examples=12,
        drift_cause="data_quality_failure",
        labels_settled=False,
    )
    for g in (guard_min_sample, guard_bad_feed, guard_label_latency):
        print(g(t))


if __name__ == "__main__":
    main()

from dataclasses import dataclass


@dataclass
class Trigger:
    new_minority_examples: int
    drift_cause: str
    labels_settled: bool


def guard_min_sample(t: Trigger, floor: int = 50) -> tuple[bool, str]:
    if t.new_minority_examples < floor:
        return (
            False,
            f"too few defaults ({t.new_minority_examples}); retrain would overfit the small sample.",
        )
    return True, "ok"


def guard_bad_feed(t: Trigger) -> tuple[bool, str]:
    if t.drift_cause == "data_quality_failure":
        return (
            False,
            "drift is a broken feed; retrain would bake the corruption in as ground truth.",
        )
    return True, "ok"


def guard_label_latency(t: Trigger) -> tuple[bool, str]:
    if not t.labels_settled:
        return (
            False,
            "labels unsettled; retrain would learn placeholder outcomes, not real ones.",
        )
    return True, "ok"


def main() -> None:
    t = Trigger(
        new_minority_examples=12,
        drift_cause="data_quality_failure",
        labels_settled=False,
    )
    for g in (guard_min_sample, guard_bad_feed, guard_label_latency):
        ok, reason = g(t)
        print(f"{'PASS' if ok else 'HOLD'}: {reason}")


if __name__ == "__main__":
    main()

This closes the module’s loop. The next lesson, Roll Out Safely, and Keep It That Way, takes a model that should ship and ships it without betting all traffic at once. This lesson was about not shipping the model that should never have been trained.

Summary

• A drift alarm is a distance measurement between two distributions, so a real-world shift and a broken upstream feed produce the same statistic: the detector cannot tell you which you have, and the two demand opposite responses. The diagnostic that separates them is the Lesson 2 data contract run on the recent window, not a bigger model.
• The Lesson 3 gate does not protect an auto-retrain loop, because the gate’s held-out set is carved from the same recent window as the training data; a self-consistent corruption is present in both splits, so the model scores high on exactly the corruption it learned. The gate passes because of the corruption, not despite it.
• A degenerate feedback loop, the model’s own predictions seeding its next training labels, compounds the corruption across generations faster than a human can review, and the loop’s speed is what converts a recoverable data incident into a compounding model one.
• The fix is one out-of-loop reference (a clean, time-fixed held-out set the loop never trains on and never regenerates) plus a human-in-the-loop checkpoint before promotion.
• A retraining runbook is mostly a list of conditions under which you hold: too few new minority examples, drift traced to a bad feed, or labels that have not settled. The cost of a needless retrain is a regression you must detect and roll back; the cost of skipping one is usually nothing.

Check your understanding:

• Why is “drift detected → retrain” the wrong wiring, and what belongs between the two arrows?
• Without scrolling up: why does the Lesson 3 model-quality gate fail to catch a retrain on corrupted data, and what single reference fixes that?
• Name two situations where drift has fired but retraining is the wrong response, and say what to do instead in each.