Roll Out Safely, and Keep It That Way

The previous lesson made the retrain decision a guarded thing: drift fires a trigger, the candidate is trained on an out-of-loop clean reference rather than on whatever recent data happened to arrive, and the loop can refuse to fire when the recent window is poisoned. Lesson 3 before it made promotion a single atomic alias re-point: fast, clean, one command. Then a model that cleared every offline gate hit production and produced garbage, because the serving pipeline computed its features differently than the training pipeline did. The alias swap had been instant, so the broken model hit every request the moment it shipped, and with one person watching, the whole product was down while the rollback was scrambled together by hand. The speed that had been built was the problem.

This lesson confronts the cost of the speed Lesson 3 built. A clean promotion with no staged exposure is a clean way to break everything at once, and the failure that breaks it, a model that passed every offline gate and still fails live, is one no offline test can see by construction. The plan is a staged rollout: run the candidate in shadow so it sees the live stream but influences nothing, then canary a bounded slice of real traffic, then go to full, with the Lesson-3 alias as the rollback at every stage. The lesson closes on the thing that keeps the deploy fast and recoverable for the second model and the hundredth: standardization, every model forced through the same path so no deploy is ever the first of its kind.

Shadow: test on real traffic with zero blast radius

The mental model that ships the outage is that the offline gate already answered the question. The candidate cleared recall over baseline on a held-out set, schema and leakage checks are green, so by Lesson 3’s logic it is ready: the alias is one re-point from sending it every request. “Passes offline” reads as “works,” and the staged rollout looks like ceremony on top of a decision already made.

That model breaks on a boundary it cannot see. The offline held-out set was produced by the training pipeline; production traffic is built by the serving pipeline, which is different code that can disagree on a default fill, a null rule, a join, or a feature whose freshness lags at serve time. ML Production Systems names this directly: “Training–serving skew is the difference between the data preprocessing that is done during training and the preprocessing that is done during serving,” and it lists the cause first as “a discrepancy between how you handle data in the training and serving pipelines (often caused by different code used for training and serving).” The offline gate, by construction, only ever saw features the training pipeline computed, so a divergence that only appears when features are computed the serving way is invisible to every offline test that exists. Here is that skew with no model and no infrastructure: two code paths computing what looks like the same feature.

import numpy as np

# The training pipeline filled a missing annual_inc with the TRAIN-SET MEDIAN,
# a constant frozen at fit time.
train_median = 65000.0
def train_features(annual_inc: float | None) -> float:
    return train_median if annual_inc is None else annual_inc

# The serving pipeline filled missing income with 0 — a different team, different
# code, same "handle the null" intent. Both paths are internally valid.
def serve_features(annual_inc: float | None) -> float:
    return 0.0 if annual_inc is None else annual_inc

# A real serve request arrives with income missing. The model trained on ~65k for
# this row; serving hands it 0. No exception, no schema error — the column is present
# and numeric. The feature is simply wrong by 65,000.
print("train path:", train_features(None))   # 65000.0
print("serve path:", serve_features(None))   #     0.0

Both functions run, both return a float, both pass any schema check that asks “is annual_inc a present number.” The candidate scored against the train-path features offline; in production it scores against the serve-path features, and a borrower whose income field was blank is fed a number off by sixty-five thousand dollars. Reliable Machine Learning states the more general version of the trap: “The code for generating features is often different in the training stack from the serving stack… it is easy for the code that generates a given feature to get out of sync,” and a serving request fails for a reason offline validation cannot cover. This is train/serve skew, a feature-plumbing failure rather than a model failure, caught here from the serving side, where the drift module and the leakage test could only ever see it from the training side.

The correct model is that “passes offline” and “behaves correctly on the live serving features” are different claims, and the only honest test of the second is to feed the candidate real production inputs while letting it affect nothing. That is exactly what a shadow deployment is: the candidate runs in parallel with the production model, receives the identical live input, and computes a prediction that is logged for comparison and then discarded, never returned to the user, never written downstream. Designing ML Systems puts shadow first among deploy methods: “Shadow deployment might be the safest way to deploy your model… route it to both models to make predictions, but only serve the existing model’s prediction to the user.” Because the candidate’s output is never acted on, its blast radius (the share of real decisions a bad candidate can damage) is zero, and the skew above surfaces as a measurable divergence rate instead of an incident. The diagram traces the fan-out one request at a time.

The fan-out buys the one comparison offline testing structurally cannot: candidate against production on the real serving stream, so skew shows up as a divergence rate here rather than as a wrong first decision later. Here is a minimal shadow comparator over a window of requests: production and candidate score the same live inputs, and the candidate’s divergence is summarized without anyone acting on it.

import statistics


def production_score(annual_inc: float) -> float:
    # The deployed model, scoring against TRAINING-style features (median fill).
    return min(1.0, max(0.0, 0.9 - annual_inc / 200_000))


def candidate_score(annual_inc: float) -> float:
    # The candidate, identical logic -- the SKEW is upstream, in how income was filled.
    return min(1.0, max(0.0, 0.9 - annual_inc / 200_000))


def main() -> None:
    # The live serving stream: some rows arrive with income missing. The serving pipeline
    # fills missing income with 0; training filled it with the median (65000).
    train_median = 65_000.0
    raw_stream: list[float | None] = [80_000, None, 45_000, None, 120_000, 30_000]

    divergences: list[float] = []
    for raw in raw_stream:
        prod_feat = train_median if raw is None else raw  # production path
        cand_feat = 0.0 if raw is None else raw  # serving path the candidate sees
        prod = production_score(prod_feat)
        cand = candidate_score(cand_feat)  # logged, NOT served
        divergences.append(abs(cand - prod))

    print("max divergence:  ", round(max(divergences), 3))
    print("mean divergence: ", round(statistics.mean(divergences), 3))
    print(
        "rows that diverged >0.1:",
        sum(d > 0.1 for d in divergences),
        "of",
        len(divergences),
    )


if __name__ == "__main__":
    main()

The two scoring functions are byte-for-byte identical, yet the candidate diverges on exactly the rows where income was missing: the divergence is the skew, isolated to the feature-fill disagreement, and it is visible without a single real decision riding on the candidate. The cost is the reason shadow is not free to leave running: it doubles inference work, because every request now produces two predictions. Designing ML Systems states the price plainly, that shadow “doubles the number of predictions your system has to generate, which generally means doubling your inference compute cost.” That trade-off is what decides whether shadow is worth standing up at all, and at a few hundred predictions a day the honest answer is often to read the candidate’s outputs by hand instead.

def shadow_divergences(
    raw_incomes: list[float | None],
    train_median: float,
    threshold: float,
) -> int:
    """Return how many rows diverge by more than `threshold`.

    Production fills missing income with `train_median`; serving fills it with 0.
    Both score with the same model: score = clamp(0.9 - income / 200_000).
    """

    def score(income: float) -> float:
        return min(1.0, max(0.0, 0.9 - income / 200_000))

    count = 0
    for raw in raw_incomes:
        prod_feat = train_median if raw is None else raw
        serve_feat = 0.0 if raw is None else raw  # the candidate sees the serving fill
        # TODO: score both, compare the absolute difference to threshold, count it
        _ = (score, prod_feat, serve_feat)
        count += 0  # placeholder
    return count


def main() -> None:
    print(shadow_divergences([90_000, None, 50_000, None], 65_000.0, 0.1))


if __name__ == "__main__":
    main()

def shadow_divergences(
    raw_incomes: list[float | None],
    train_median: float,
    threshold: float,
) -> int:
    """Return how many rows diverge by more than `threshold`."""

    def score(income: float) -> float:
        return min(1.0, max(0.0, 0.9 - income / 200_000))

    count = 0
    for raw in raw_incomes:
        prod_feat = train_median if raw is None else raw
        serve_feat = 0.0 if raw is None else raw
        if abs(score(serve_feat) - score(prod_feat)) > threshold:
            count += 1
    return count


def main() -> None:
    print(shadow_divergences([90_000, None, 50_000, None], 65_000.0, 0.1))  # 2


if __name__ == "__main__":
    main()

Canary then full: limit blast radius by limiting exposure

Shadow proved the candidate runs correctly on live inputs, so the tempting next move is to re-point the alias and go to full: the candidate is validated, ship it. That reasoning treats shadow as the last check it needs to be. It is not, because shadow has a blind spot built into its own design.

Shadow logs predictions but changes nothing: its decisions never happen, so any second-order effect of those decisions is structurally invisible to it. A decision that shifts user behavior, trips a downstream system, or changes the label distribution cannot show up in a shadow log, because the decision was never made. Going shadow → full skips the only stage that measures the candidate’s effect rather than its outputs, and exposes every request to that unmeasured effect at once. Here is the gap, made concrete: shadow can only ever compare numbers, so an effect that requires the decision to fire is zero in every shadow log by construction.

# Shadow logs the candidate's APPROVE/DENY decision but never acts on it.
# Suppose the candidate approves a class of risky loans the production model denied.
# The second-order effect — those approved borrowers later defaulting — needs the
# loan to actually be issued. Shadow never issued it, so the effect is unobservable.

shadow_log = [
    {"decision": "approve", "served": False},  # candidate said approve, but nobody acted
    {"decision": "approve", "served": False},
]
defaults_observed = sum(1 for r in shadow_log if r["served"] and r.get("defaulted"))
print("defaults attributable to the candidate:", defaults_observed)  # 0, always

Every served flag is False, so the count is structurally zero: shadow cannot observe a downstream consequence of a decision it never let happen. The fix is to let a bounded fraction of real decisions happen and watch what they do. A canary release routes a small share of real traffic to the candidate, which now actually serves those requests, and live metrics on that slice gate the move to full. Designing ML Systems gives the four steps: “Deploy the candidate model alongside the existing model. The candidate model is called the canary… A portion of the traffic is routed to the candidate model… If its performance is satisfactory, increase the traffic… If not, abort the canary and route all the traffic back to the existing model.” The name is older than software: the SRE Book notes that “the first stages of a rollout are usually called ‘canaries’ — an allusion to canaries carried by miners into a coal mine to detect dangerous” gases. Rollback is the Lesson-3 alias re-point, so only the canary fraction was ever exposed.

The arithmetic is the entire safety argument. A bad model at a 5% canary harms 5% of traffic, and recovery is re-pointing the alias to send that 5% back to production; the same bad model promoted straight to full harms 100%, with no smaller blast radius to fall back to. Each stage answers a question the previous one structurally cannot, and the ordering is forced by what each stage risks: shadow asks “does it run on live inputs?” at zero risk, canary asks “does it behave correctly when its decisions affect users?” at bounded risk, full asks “does it hold at scale?” The code below makes the blast-radius arithmetic explicit across the three exposures.

def blast_radius(total_requests: int, exposure: float, bad_rate: float) -> int:
    """Requests a bad candidate damages = total * exposure * its bad-decision rate."""
    return round(total_requests * exposure * bad_rate)


def main() -> None:
    total = 10_000  # requests over the rollout window
    bad_rate = 0.30  # the candidate is wrong on 30% of decisions (skew slipped through)

    for stage, exposure in [("shadow", 0.0), ("canary 5%", 0.05), ("full", 1.0)]:
        damaged = blast_radius(total, exposure, bad_rate)
        print(f"{stage:>10}: {damaged:>5} damaged decisions  (exposure {exposure:.0%})")


if __name__ == "__main__":
    main()

Shadow damages zero because its exposure is zero; the canary caps the damage at the slice it was granted; full has no ceiling below 100%. The judgment is not whether to canary; it is the fraction, and the fraction is a tunable with a real failure mode at each end.

The principle for the fraction is that the canary slice must be large enough, in absolute requests over the window being watched, to move a metric that can be trusted; otherwise the gate is a coin flip. Observability Engineering tells the canonical story against forgetting this: a team “deployed to 1% of traffic, and then checked our dashboards.” Then “Everything looked fine…and it was fine, until we hit 70% and every request started queueing up trying to hit the same row lock on the database.” The 1% slice was a “structurally unrepresentative sample” with “error rates too small to pick out of the noise.” The fraction is the parameter; here is the response curve.

The canary is not a softer version of full; it is the only stage where the candidate’s real decisions are made and the damage is bounded, which is why the second-order failures that shadow cannot see and offline tests cannot see both surface here, cheaply. Note that canary is a different exposure model from blue/green: ML Production Systems describes blue/green as deploying “a new version… to the ‘Green’ environment, which acts as a staging setup where a series of tests are conducted” before a full swap of traffic, rolling back by redirecting to blue. Blue/green swaps everyone at once after offline tests; canary ramps a percentage and watches live decisions. They are not interchangeable, and treating canary as “blue/green but gradual” loses the point that canary’s whole value is the bounded live exposure blue/green does not have.

def damaged_decisions(total: int, exposure: float, bad_rate: float) -> int:
    """Decisions a bad candidate damages at this exposure.

    A bad candidate that is wrong on `bad_rate` of decisions, given `exposure`
    of `total` requests, damages how many?
    """
    # TODO: total requests routed to the candidate, times its bad-decision rate
    return 0  # placeholder


def main() -> None:
    total = 10_000
    bad = 0.30
    print("canary 5%:", damaged_decisions(total, 0.05, bad))
    print("full:     ", damaged_decisions(total, 1.00, bad))


if __name__ == "__main__":
    main()

def damaged_decisions(total: int, exposure: float, bad_rate: float) -> int:
    """Decisions a bad candidate damages at this exposure."""
    return round(total * exposure * bad_rate)


def main() -> None:
    total = 10_000
    bad = 0.30
    print("canary 5%:", damaged_decisions(total, 0.05, bad))  # 150
    print("full:     ", damaged_decisions(total, 1.00, bad))  # 3000


if __name__ == "__main__":
    main()

Standardization: the discipline that makes the speed durable

By this point the pipeline tests code, tests data, gates and versions the model, refuses unsafe retrains, and rolls out shadow → canary → full. Each piece works, so the obvious conclusion is that the deploy is now fast and safe: the hard parts are built, and from here it is a matter of running them. That conclusion mistakes the steps for the thing that makes a deploy fast.

A deploy is fast when there is nothing left to re-decide: every step is automated, tested, and identical to last time. The non-obvious half is that uniformity is also what keeps the recovery paths alive. The rollback that runs on every canary failure is the rollback that works in an incident; the rollback that exists in the repo but has never actually run is the one that takes a day while production is down. Here is the failure that teaches this: the first time one “urgent” model is allowed to skip the canary.

# The standard path. Every model goes through it, so the rollback branch runs on
# every canary failure — it is exercised constantly and known to work.
def deploy(model, *, skip_canary: bool = False):
    register(model)
    shadow(model)
    if skip_canary:
        promote_to_full(model)        # <-- the "skip it this once, it is urgent" branch
        return
    if not canary(model):             # this branch fires on most normal deploys...
        rollback()                    # ...so THIS rollback is exercised and trusted
        return
    promote_to_full(model)

# The urgent model skips the canary and ships broken to 100% of traffic. Now rollback
# must run — but it has not actually executed in weeks, because every NORMAL deploy
# auto-rolled-back at the canary stage, never from full. The cold branch takes most of
# a day to run correctly while production serves garbage.
deploy(urgent_model, skip_canary=True)

The skip_canary=True branch did two kinds of damage at once. The urgent model shipped to everyone with no bounded stage to catch it, and the rollback-from-full path, distinct from the rollback-from-canary path that runs constantly, turned out to be cold, untested, and slow exactly when it was needed. The exception did not save time; it converted a five-minute staged rollout into a day of debugging an unexercised path. The correct model is that reliability comes from repetition, not from the steps themselves: every model goes through the same path every time, so no deploy is ever the first of its kind and every recovery branch is exercised on every release. The SRE principle behind this is explicit in Building Secure and Reliable Systems, which advises “cutting and rolling out releases regularly… each release contains fewer changes, which are therefore less likely to require rollback,” and the value compounds because the release and rollback machinery stays warm only if it runs constantly.

The fragility here is social, not technical. The standard path does not erode because the code rots; it erodes the first time someone decides one model is special and routes it differently. Kubernetes Up and Running makes the same observation from the ops side: follow the release process “completely for every release, no matter how big or how small. Many outages have been caused by people accelerating releases.” On a small team this matters more, not less: there is no release engineer and no deploy-checklist owner, so the only thing keeping a deploy repeatable is that the path is automated and identical, and the person debugging the novel deploy at 2 a.m. is the same person who built it. The execution is a single entry point every model is forced through, with no parameter that lets a caller skip a stage. The structure below is the whole point: one door, the same stages, every time.

[any new model] as model
[gate + register\n(L3)] as gate
[shadow] as shadow
[canary slice] as canary
[full traffic] as full
[alias re-point\n(L3 rollback)] as rollback
model --> gate : the ONLY entry point
gate --> shadow
shadow --> canary
canary --> full : metrics pass
canary --> rollback : metrics fail
full --> rollback : regression caught
rollback --> gate : back through the same door

There is no edge from model to full and no skip_canary parameter, so the rollback edges from canary and full are traversed on real deploys often enough to stay trusted. Here is the standard path as one function with no skip parameter: the stages are forced, and “urgent” gets the same door faster, not a different door.

def standardized_deploy(model_name: str, canary_metric_ok: bool) -> str:
    """One entry point. Every model: gate -> register -> shadow -> canary -> full.
    There is no parameter to skip a stage. Rollback is the same alias re-point every time.
    """
    stages: list[str] = []
    stages.append(f"gate+register({model_name})")  # L3
    stages.append("shadow")  # 5.1 -- zero blast radius
    stages.append("canary")  # 5.2 -- bounded blast radius
    if not canary_metric_ok:
        stages.append("ROLLBACK via alias re-point")  # exercised on every failed canary
        return " -> ".join(stages)
    stages.append("full")
    return " -> ".join(stages)


def main() -> None:
    print("normal model:", standardized_deploy("xgb_v7", canary_metric_ok=True))
    print("bad model:   ", standardized_deploy("xgb_v8", canary_metric_ok=False))
    print("urgent model:", standardized_deploy("hotfix_v9", canary_metric_ok=True))


if __name__ == "__main__":
    main()

The urgent model takes the identical path as the normal one, with no faster door, only the same door run without hesitation, and the bad model’s rollback is the same alias re-point that fires on every routine canary failure, which is why it is fast. Practical MLOps and the ML Interviews Book both make the same point: standardizing every model onto one repeatable, CI-gated, data-validated, model-gated, registry-versioned, canary-rolled template is what turns deployment from a per-project ordeal into a repeatable step, and it is the mechanism behind a months-long deploy becoming a three-day one and staying there. The value is not the list of stages; it is that they are the same stages every time, which is what makes the speed durable rather than a one-time win.

def standardized_deploy(model_name: str, canary_metric_ok: bool) -> list[str]:
    """Return the ordered list of stages this model passed through.

    There must be NO path to "full" that skips "shadow" or "canary".
    A failed canary ends in "rollback", not in "full".
    """
    stages: list[str] = ["gate+register", "shadow"]
    # TODO: route through canary; on failure end in rollback, on success reach full
    return stages  # placeholder: currently never reaches canary or full


def main() -> None:
    print(standardized_deploy("xgb_v7", canary_metric_ok=True))
    print(standardized_deploy("xgb_v8", canary_metric_ok=False))


if __name__ == "__main__":
    main()

def standardized_deploy(model_name: str, canary_metric_ok: bool) -> list[str]:
    """Return the ordered list of stages this model passed through."""
    stages: list[str] = ["gate+register", "shadow", "canary"]
    if not canary_metric_ok:
        stages.append("rollback")
        return stages
    stages.append("full")
    return stages


def main() -> None:
    print(standardized_deploy("xgb_v7", canary_metric_ok=True))  # ...canary, full
    print(standardized_deploy("xgb_v8", canary_metric_ok=False))  # ...canary, rollback


if __name__ == "__main__":
    main()

Summary

• A model that passes every offline gate can still fail live, because the offline set comes from the training pipeline and production traffic comes from the serving pipeline: train/serve skew is a feature-plumbing disagreement no offline test can see by construction.
• Shadow runs the candidate on the live serving stream and discards its output, so skew surfaces as a divergence rate at zero blast radius, at the cost of doubling inference compute for the window.
• Canary lets a bounded fraction of real decisions happen and watches their effect, catching the second-order failures shadow cannot; the fraction must be small enough to bound damage and large enough to be a trustworthy sample, or the gate runs on noise.
• Blast radius is exposure times error rate: a 5% canary caps damage at a twentieth of going straight to full, and rollback is the Lesson-3 alias re-point.
• Standardization is what keeps the speed durable: every model forced through the same single-entry path so no deploy is novel and the rollback branch stays warm; the first “skip the canary just this once” is what brings the slow, scary deploy back.

Check your understanding:

• Why can a shadow deployment never observe a second-order effect of the candidate’s decisions, no matter how long it runs?
• A team runs a 1% canary on a service taking a few hundred requests an hour, sees a green dashboard, and ships a regression. What was wrong with the canary, and what is the fix?
• Without looking back: what two distinct kinds of damage does one “urgent” model skipping the canary cause, and why is the rollback slow specifically in that case?