Make Failure Cheap: Release Strategy and One-Step Rollback

I promoted a build that had passed dev and pointed prod at it on the custom domain. It carried a bug that only showed under real traffic: predictions came back, looked fine, and were skewed. Getting back to the previous version meant finding the old commit, rebuilding the image, and redeploying, which took long enough that bad scores reached customers the whole time I worked, alone, with the founder asking in Slack why the demo looked wrong. The deploy itself was fine. My recovery was the disaster. A rollback you have to assemble under pressure is not a rollback.

In the last lesson you put the service on a real domain and watched the first hour fail normally while DNS propagated and the certificate provisioned. That made the service reachable. This lesson makes a bad release survivable. Two decisions own that outcome: the release strategy, which is how a new version replaces the old one, and rollback, which is how you undo it. Both are decided before the bad release ships, not during the incident. The thread that runs through all three sections is that the swap is a separate operation from the code change, and that operation can take you down even when the new image is perfect.

Recreate vs blue/green: the swap decides your worst case

A deploy ultimately changes one thing: which container the front-end router forwards traffic to. The router is the entry point from the first lesson in this module, the component that receives every public request and decides which running container answers it. A competent mid-level engineer reasons about a deploy as “build the new image, the platform swaps it in, done,” and treats the swap as instantaneous bookkeeping. That model is wrong in a way that does not show at three users and becomes an outage at a hundred requests a minute. The swap has an order — when the old container stops relative to when the new one starts and when the router cuts over — and that order sets your worst-case downtime before the new code runs a single request.

Watch the wrong model break. Here is a recreate-style deploy of an image that happens to be broken, written as the commands and the platform’s own output so the timeline is visible.

# Recreate: stop the old container, then start the new one.
$ deploy --strategy=recreate --image registry/scoring@sha256:bad...
[12:00:01] stopping current container (blue)        # old is GONE now
[12:00:02] router has no healthy target -> 503       # ZERO-HEALTHY WINDOW opens
[12:00:04] starting new container (green)
[12:00:04] green: loading model artifact...      # the slow part: model load
[12:00:19] green: readiness probe FAILED (model path wrong)
[12:00:34] green: readiness probe FAILED
# ...the probe never passes. The window never closes.
# Every request from 12:00:02 onward is a 503.

The window between “old terminated” and “new passes its probe” is a window with zero healthy instances, and every request in it fails with a 503 or a hung connection. Two things make that window dangerous. Its length is the new container’s full startup time, and for an ML service the slow part is loading the model artifact from the image into memory, which is seconds, not milliseconds. And if the new image is broken, the probe never passes, so the window never closes: the outage is open-ended and self-inflicted. The swap, not the bug, is what took the service down.

Blue/green inverts the order. The principle is that the old version must keep serving every request until the new version has proven it can serve, so the cutover is a routing change with no boot time inside it and the escape hatch is immediate. Watch the exact same broken image deploy under blue/green, and watch where the zero-healthy window from the recreate timeline fails to open at all.

# Blue/green: bring green up on SEPARATE capacity while blue keeps serving.
$ deploy --strategy=blue-green --image registry/scoring@sha256:bad...
[12:00:01] blue still serving 100% of traffic          # nobody is dropped
[12:00:02] starting green on new capacity
[12:00:02] green: loading model artifact...
[12:00:17] green: readiness probe FAILED (model path wrong)
[12:00:32] green: readiness probe FAILED
[12:00:47] abort: green never became ready, router NOT flipped
[12:00:47] blue served 100% throughout. Zero 503s.

Because blue was never touched, the same broken image causes zero customer impact: the router only flips after green passes a real readiness check, and a broken green never passes, so the flip never happens. When green is healthy, the cutover is near-instant because it is a routing change with no container boot inside it, and the fallback is immediate because blue is still warm. Canary is the same routing primitive applied gradually: route a small percentage to green, read per-version error and latency metrics, then ramp. That shrinks the blast radius of a bad release to the percentage you exposed, at the cost of traffic-splitting machinery and per-version metrics you have to build and watch.

The scrolly runs the same broken-image deploy twice, once per strategy, so the zero-healthy gap and where it opens is visible against the timeline.

The non-obvious cost is that blue/green is not free and not always right. It pays for two full environments during the overlap, and below the traffic level where the swap window drops real requests, that second environment buys nothing. Recreate plus a fast rollback is the honest default at low traffic: a few seconds of swap downtime off-hours costs nothing, and you recover by repointing at the last good version. The staff judgment is not “which strategy is best.” It is “at what traffic level does the cheap strategy’s downtime window cost more than the spare environment,” and the signal is dropped requests, not the calendar. The same recreate that was the right, cheap call at a handful of users became unacceptable once the window dropped paying requests.

"""Try It: compute the zero-healthy window of a recreate deploy.

Predict whether requests are dropped and for how long, then check against the
computed window.
"""


def zero_healthy_seconds(events: list[tuple[float, str]]) -> float:
    # Return the seconds with no healthy container:
    # from "blue terminated" until "green readiness PASSED".
    down_start = 0.0  # placeholder
    up_again = 0.0  # placeholder
    return up_again - down_start


if __name__ == "__main__":
    events: list[tuple[float, str]] = [
        (0.0, "blue serving"),
        (1.0, "blue terminated"),
        (1.0, "green booting, loading model"),
        (16.0, "green readiness PASSED"),
        (16.0, "router flips to green"),
    ]
    print("zero-healthy window:", zero_healthy_seconds(events), "s")

"""Solution: the zero-healthy window is the full startup time of the new container."""


def zero_healthy_seconds(events: list[tuple[float, str]]) -> float:
    down_start = next(t for t, e in events if e == "blue terminated")
    up_again = next(t for t, e in events if e == "green readiness PASSED")
    return up_again - down_start


if __name__ == "__main__":
    events: list[tuple[float, str]] = [
        (0.0, "blue serving"),
        (1.0, "blue terminated"),
        (1.0, "green booting, loading model"),
        (16.0, "green readiness PASSED"),
        (16.0, "router flips to green"),
    ]
    window = zero_healthy_seconds(events)
    print("zero-healthy window:", window, "s")
    print("at 100 req/min, dropped requests:", round(window / 60 * 100))

A health check that passes before the model loads is a lie

The cutover in the last section is only as safe as the signal that decides green is ready. The illusion a mid-level engineer carries is that the platform’s “deploy succeeded” means the new version can serve, because the platform drives that status from the health probe and the probe went green. The correct mental model is that “ready” must mean “can serve a real request,” not “is accepting connections,” and for an ML service those are two different events separated by the slow, fallible part: loading the model artifact from the image into memory. A probe that checks the cheap thing to certify the expensive thing is the gap every silent bad cutover lives in.

Here is the anti-pattern. The probe touches no model, no feature code, no registry artifact; it returns 200 the moment the web server is accepting connections.

from fastapi import FastAPI

app = FastAPI()
model = None  # loaded lazily, on first /predict

@app.get("/health")
def health() -> dict[str, str]:
    return {"status": "ok"}   # 200 the instant uvicorn binds the socket

@app.post("/predict")
def predict(record: dict) -> dict[str, float]:
    global model
    if model is None:
        model = load_model_artifact()  # the expensive, fallible work
    return {"probability": float(model.predict_one(record))}

The shallow /health returns 200 as soon as the ASGI app is accepting connections. It never touches the model. The expensive, fallible work, loading the model and reading config and opening downstream connections, is exactly what determines whether /predict works, and a probe that checks the socket cannot see any of it. If model-load is lazy, as it is above, the probe passes on a container whose model has never been touched. A liveness probe answers “should this be restarted”; a distinct readiness probe answers “is this ready to receive traffic,” and the reason for the split is precisely that an application can be alive but not ready while it connects to databases, loads plugins, or downloads serving files.

The honest readiness check loads the model and scores a canned record, returning ready only after a real prediction succeeds. The contrast below is the whole lesson in one comparison: a shallow check certifies a broken container, the deep one refuses to.

"""Contrast a shallow health check with a deep readiness check that scores a record."""


def shallow_health() -> dict[str, str]:
    # Touches nothing. 200 the instant the socket is open.
    return {"status": "ok"}


def deep_readiness(model: object | None, canned: dict[str, float]) -> tuple[int, str]:
    # Ready ONLY after a real prediction succeeds on a canned record.
    if model is None:
        return 503, "model not loaded"
    try:
        model.predict([list(canned.values())])  # exercise the real path
    except Exception as exc:  # noqa: BLE001 - probe must catch everything
        return 503, "predict failed: " + str(exc)
    return 200, "ready"


class FittedModel:
    def predict(self, rows: list[list[float]]) -> list[float]:
        return [0.0 for _ in rows]


if __name__ == "__main__":
    canned = {"income": 50000.0, "dti": 0.3}
    print("model not loaded yet:")
    print("  shallow:", shallow_health())  # says OK
    print("  deep:   ", deep_readiness(None, canned))  # says 503, honest

    print("model loaded:")
    print("  deep:   ", deep_readiness(FittedModel(), canned))  # now 200

The shallow check says OK while the model is None; the deep check says 503 until a real prediction succeeds, then 200. Wire the deep check as the gate, and the cutover from the last section becomes trustworthy: blue/green’s atomic flip and canary’s ramp both refuse to send traffic to a green that has not yet produced a correct score. The named failure mode is the silent bad cutover: a blue/green deploy whose green passed a shallow /health while its model path was wrong went green, the router flipped atomically, and one hundred percent of /predict started returning 500 the instant cutover completed. Blue/green did its job perfectly; it cut traffic to a container the gate had wrongly certified. The fix was never the deploy strategy. It was making the readiness probe load the model and score a canned record so “ready” meant “can serve.”

"""Try It: make readiness return ready only after a real prediction succeeds.

Modify the probe so it returns ready only after the model can produce a score on
a canned record, and 503 otherwise.
"""

CANNED: list[float] = [50000.0, 0.3]


class Service:
    def __init__(self) -> None:
        self.model: object | None = None  # set by load(), may stay None on failure

    def readiness(self) -> tuple[int, str]:
        # BROKEN: reports ready as long as the object exists.
        # Make it exercise a real prediction on the canned record below.
        return 200, "ok"


if __name__ == "__main__":
    svc = Service()
    print("before load:", svc.readiness())  # should be 503

"""Solution: readiness is two preconditions -- model exists and prediction succeeds."""

CANNED: list[float] = [50000.0, 0.3]


class FittedModel:
    def predict(self, rows: list[list[float]]) -> list[float]:
        return [0.0 for _ in rows]


class Service:
    def __init__(self) -> None:
        self.model: object | None = None

    def load(self) -> None:
        self.model = FittedModel()

    def readiness(self) -> tuple[int, str]:
        if self.model is None:
            return 503, "model not loaded"
        try:
            self.model.predict([CANNED])
        except Exception as exc:  # noqa: BLE001
            return 503, "predict failed: " + str(exc)
        return 200, "ready"


if __name__ == "__main__":
    svc = Service()
    print("before load:", svc.readiness())  # 503, honest
    svc.load()
    print("after load: ", svc.readiness())  # 200, can actually serve

Rollback is built at promotion time, not assembled under load

Rollback speed is decided at promotion time, not at incident time, and the deciding factor is whether the previous version still exists as an addressable artifact. The mid-level reflex is to treat rollback as a thing you do during the incident: notice the bug, find the last good commit, rebuild, redeploy. That is not a rollback. It is an incident response that happens to end in the old version, slowly. By incident time it is too late to create the precondition; you either have the old artifact or you are rebuilding the past while customers get bad scores. The correct mental model is that rollback is the same routing primitive as the blue/green cutover, run in reverse, pointed at a digest you already shipped and kept.

Here is the slow path, the one I lived in the opening story. Recovery means rebuilding from source because the previous artifact was never kept as an addressable thing.

# The slow "rollback" — actually a rebuild from source.
$ git log --oneline                      # find the last good commit... which one?
$ git checkout a1b2c3d
$ docker build -t scoring:fix .          # FULL rebuild: resolve deps, compile, layer
 => RUN uv sync --frozen                 # minutes, every time
 => COPY . .
$ docker push registry/scoring:fix
$ deploy --image registry/scoring:fix    # only NOW does recovery start
# bad scores reached customers for the entire build-and-push window

Every step there is build work happening after the incident started: find the commit, resolve and download dependencies, layer the image, push it. The reason it is slow is mechanical. The previous artifact does not exist anymore, so “go back” means “construct the past,” and construction is minutes. This is what shipping from a mutable tag like latest or straight from source forces: the prior build was overwritten or never stored, so there is nothing to point back at.

The fast path exists only because the last lesson on promotion shipped immutable image digests. An image digest is the @sha256:... content hash of a built image; unlike a tag such as latest, it never moves and never gets overwritten, so every version you ever shipped is still in the registry under its own digest. The platform also keeps a revision history, a record mapping each past deploy to the digest it ran. Rollback reuses that recorded template and renumbers it as the newest revision; the orchestrator tracks the previous version for you, so you do not. Watch the same recovery as one command against that history — note the platform loads the prior digest rather than building anything, which is why the timeline is under a minute.

# The fast path — repoint at a digest that already exists.
$ deploy history
 REV  DIGEST                  STATUS
 7    @sha256:bad...          live (skewed scores)
 6    @sha256:good...         superseded            # last known good, still in registry
 5    @sha256:older...        superseded
$ deploy rollback --to-revision 6     # ONE command, no build
[12:01:02] repointing live -> @sha256:good...
[12:01:02] revision 6 image already present, loading...   # only loads, never builds
[12:01:04] readiness PASSED
[12:01:04] router flipped to revision 6
# recovery is under a minute: the artifact was already there

The rollback is one command because the artifact already exists; recovery only has to load the image, not build it. If the prior revision’s container is still warm, in a platform that keeps the previous revision or a blue/green setup where blue never came down, rollback is a pure routing flip, sub-second, no boot. The same routing primitive runs the live pointer backward over the revision timeline, and traffic follows it. The scrolly walks the pointer move and contrasts the bottom track where no digest was kept, so the pointer has nowhere to go and recovery detours through a full rebuild.

The one judgment rollback still requires is not a reflex. Rolling back to the last good version can reintroduce a known vulnerability or bug the new version fixed, and the older or more visible that vulnerability, the more likely a weaponized exploit for it is already circulating. Rollback is the right first move during most incidents, because you mitigate and then diagnose, but it is not unconditionally safe: if the release you are undoing was a security patch, rolling back undoes the patch and you are racing attackers. The staff move is to keep rollback one-step and know what the previous digest contains, so the choice is informed rather than blind. The cost of the second environment that makes some of this instant is bounded in the next lesson.

"""Try It: implement a rollback that distinguishes a kept digest from a rebuild.

Return the digest to repoint at and the elapsed cost: a present digest only loads,
a missing one must rebuild from source.
"""

HISTORY: dict[int, str] = {
    7: "sha256:bad",
    6: "sha256:good",
    5: "sha256:older",
}
PRESENT: set[str] = {"sha256:bad", "sha256:good", "sha256:older"}


def rollback(
    history: dict[int, str], present: set[str], to_rev: int
) -> tuple[str, float]:
    # Return (digest, seconds). If the digest is already present, it only loads (~2s).
    # If it is NOT present, recovery must rebuild from source (~180s).
    digest = ""  # placeholder
    seconds = 0.0  # placeholder
    return digest, seconds


if __name__ == "__main__":
    print(rollback(HISTORY, PRESENT, 6))

"""Solution: a kept digest costs a load; an overwritten one forces a rebuild."""

HISTORY: dict[int, str] = {
    7: "sha256:bad",
    6: "sha256:good",
    5: "sha256:older",
}
PRESENT: set[str] = {"sha256:bad", "sha256:good", "sha256:older"}
LOAD_COST = 2.0
REBUILD_COST = 180.0


def rollback(
    history: dict[int, str], present: set[str], to_rev: int
) -> tuple[str, float]:
    digest = history[to_rev]
    seconds = LOAD_COST if digest in present else REBUILD_COST
    return digest, seconds


if __name__ == "__main__":
    print("kept digest:", rollback(HISTORY, PRESENT, 6))

    # Now simulate shipping from a mutable tag: the prior artifact was overwritten.
    overwritten = {"sha256:bad"}  # only the latest survived
    print("overwritten: ", rollback(HISTORY, overwritten, 6))

Summary

• A deploy changes which container the router forwards to, and the order of teardown, startup, and cutover sets your worst-case downtime before the new code runs. Recreate opens a zero-healthy window for the new container’s full startup time; a broken image makes that window open-ended.
• Blue/green keeps the old version serving until the new one passes a real readiness check, so a broken green takes zero traffic. The cost is two environments during the overlap, justified only once the swap window drops paying requests.
• A readiness probe must mean “can serve a real request,” not “the socket is open.” A shallow /health certifies a container whose model never loaded, and blue/green or canary will faithfully cut traffic to it.
• Rollback is the cutover primitive run in reverse against a digest that already exists. It is one command and a load, not a rebuild, only because promotion kept every shipped digest and the revision history points at them.
• Rollback is not unconditionally safe: undoing a security patch reintroduces a known, possibly weaponized vulnerability. Keep rollback one-step and know what the previous digest contains.

Check your understanding:

• During a recreate deploy of a broken image, why does the outage stay open-ended instead of lasting the new container’s startup time?
• A blue/green deploy flipped to a green that 500s on every request. The deploy strategy worked correctly. What was the actual defect, and which probe fixes it?
• Without looking back: what must already be true at promotion time for rollback to be one command, and why is “rebuild the previous commit” not a rollback?