Promote, Do Not Just Push

I shipped a typo straight to the only environment that existed. The change was a one-line edit to a config default, and the deploy loop felt finished after the last lesson: edit the service, push the image, refresh the host-assigned URL, watch the new build serve. I ran that loop without thinking, the platform reported “deploy succeeded,” and I closed the laptop. Forty minutes later the support inbox had three reports of the scorer returning errors. The typo had pointed the model loader at a path that did not exist, the /health probe went green the instant the web process accepted connections, and traffic shifted to a build that returned a 500 on every real request. The deploy and the outage were the same event, because there was nowhere that bad build could have failed first where no customer would see it.

That is the problem this lesson fixes. The service is live, which means the single live environment is doing two incompatible jobs at once: it is where a build is found to work or not work, and it is where customers are served. This lesson splits those jobs apart. The fix puts a second, identical environment in front of production, promotes one known-good image between them by its content digest, and injects the per-environment config and secrets the identical image needs to behave correctly in each. The goal is narrow and specific: the next change is tested somewhere before it reaches a customer.

One environment means every deploy is a production deploy

The mental model that feels right with one environment is that “deploy succeeded” means “the service works.” The platform polls the container, the probe returns 200, the platform reports success and shifts traffic. A competent engineer reads that sequence as a verification step: the platform checked the build before exposing it. That reading is the source of the outage.

The opening incident already showed the shape of the lie. The /health probe in the last lesson, a green probe sitting on top of a /predict that 500s, is exactly what shipped the typo to customers. The vocabulary for that gap is liveness (the process is running and accepting connections) versus readiness (the process can actually serve a real request). What “deploy succeeded” reports is a health probe, the HTTP endpoint the platform polls, and that probe checks liveness. It answers 200 the moment the web server accepts connections, before the model artifact is loaded and before any real route runs.

The named failure mode is a green-but-broken deploy. The symptom is “deploy succeeded” and a flood of 500s arriving together. The root cause is that the readiness signal the platform trusts checks liveness, not readiness, and the boundary it violates is the assumption that the probe and the route exercise the same code path. A deadlocked or still-initializing process is alive but not ready, and a process health check still believes the application is healthy because the process is still running. The full liveness-versus-readiness demo lived in the first lesson; the readiness fix, a probe that fails until the model is actually loaded, comes in a later lesson. What this lesson adds is the consequence of that gap under a single environment, and the consequence is severe.

With one environment, the machine that runs the new build is also the machine serving customers. The moment of discovery and the moment of exposure are the same instant. There is no safe place to fail.

Split the act of running a new build from the act of serving it to customers

A deployment target that also serves customers cannot also be the place a build is discovered to work or fail. Discovery and exposure become one event, so there is no safe failure. The fix is a second, identical deployment target that no stranger has the URL for: same image, same platform, a different URL. A bad build fails there, in front of nobody, and only a build that answered the real route correctly is allowed forward. This is the cheapest version of the staging-parity discipline: keep development and production as similar as possible, because every place production diverges from the environment the build was tested in introduces risk and hides issues until production.

A second environment does not make the probe honest. What it buys is a place for a human to catch the lie before customers do. The decision below is which discipline that second environment must enforce to be worth its cost.

The gate that makes a dev environment real is a smoke test against the live route, not the probe. A smoke test is run after the build is deployed but before it is released to traffic, and it makes a request the way a customer would. Here is that gate as a function: it scores one canned record against the dev URL and refuses to return a promotable verdict unless the response is a well-formed probability.

"""Gate a dev build on the real /predict route, not the health probe."""

import json


def smoke_test(dev_response_status: int, dev_response_body: str) -> bool:
    """Gate a dev build on the real route, not the health probe.

    Returns True only if /predict answered with a parseable probability.
    A green /health is necessary but not sufficient to promote.
    """
    if dev_response_status != 200:
        return False
    try:
        body = json.loads(dev_response_body)
    except json.JSONDecodeError:
        return False
    prob = body.get("probability")
    if not isinstance(prob, (int, float)):
        return False
    return 0.0 <= prob <= 1.0


if __name__ == "__main__":
    # The green-but-broken container: /health was 200, but /predict 500s.
    print("health-only green, predict 500:", smoke_test(500, ""))
    # A model that loaded but returns a malformed payload (wrong key) still fails.
    print("malformed payload:", smoke_test(200, '{"verdict": 0.7}'))
    # The only response that earns promotion: a real probability on the real route.
    print("real probability:", smoke_test(200, '{"probability": 0.31}'))

The function rejects the two failures the probe cannot see, a 500 on /predict and a structurally wrong payload, and passes only a well-formed probability. The 0.0 <= prob <= 1.0 check is doing more than type validation: a model that loaded but is scoring nonsense (a default-default fallback returning 1.0 for everything, the skew failure from the next concept) still passes the type check, so the range bound is the cheapest signal that the right model answered. The verdict this returns is the input to the promotion step, and only True is allowed to move forward.

"""Try It: which probe-and-route states would a /health-only gate wrongly serve?

Given the four states below, decide which ones the platform's /health-only gate
would call "ready to serve."
"""


def health_only_gate(health_status: int, predict_status: int) -> str:
    # The platform shifts traffic on a green /health alone.
    # It never calls /predict. Return "SERVES" or "blocked".
    return "blocked"  # placeholder -- fix me


if __name__ == "__main__":
    cases = [
        ("model loaded, route works", 200, 200),
        ("wrong MODEL_PATH, route 500s", 200, 500),
        ("still initializing, not accepting yet", 503, 503),
        ("model loaded but route raises", 200, 500),
    ]
    for name, h, p in cases:
        print(name, "->", health_only_gate(h, p))

"""Solution: count the broken builds a /health-only gate lets through."""


def health_only_gate(health_status: int, predict_status: int) -> str:
    # The platform shifts traffic on a green /health alone -- predict_status
    # is never consulted, which is exactly the blind spot.
    return "SERVES" if health_status == 200 else "blocked"


if __name__ == "__main__":
    cases = [
        ("model loaded, route works", 200, 200),
        ("wrong MODEL_PATH, route 500s", 200, 500),
        ("still initializing, not accepting yet", 503, 503),
        ("model loaded but route raises", 200, 500),
    ]
    for name, h, p in cases:
        verdict = health_only_gate(h, p)
        broken = " <- green-but-broken" if verdict == "SERVES" and p != 200 else ""
        print(name, "->", verdict + broken)

Promote the artifact, do not rebuild it

Two environments now exist. The mental model that feels right for moving a build between them is that the source is the source of truth: the dev deploy builds the image from a commit, validates it, and the prod deploy builds again from the same commit. Same source in, same image out, so prod is a faithful copy of the build that passed dev. This is the assumption every “build per environment” pipeline rests on, and it is wrong in a way that takes hours to diagnose.

A Docker image is not a deterministic function of its source. The breaking condition is that the build pulls inputs the source does not pin. FROM python:3.12 resolves to whatever digest the 3.12 tag points at right now, and that tag is republished with security patches. A floating tag like latest is re-pulled as a different image whenever a newer one has been published since the last run. Build-time inputs (a re-resolved apt index, timestamps, anything fetched over the network) get captured into layers. M5’s lock pins the Python packages but not the OS layer beneath them. So two builds of one commit, minutes apart, can produce two different images. Watch the build-per-environment pipeline produce exactly that: two docker build runs of one checked-out commit, hours apart, landing on different digests because the base tag was republished between them.

# Pipeline A: build per environment (the trap)
$ git checkout 9f2a1c             # the commit that passed dev
$ docker build -t scorer:dev .    # built Tuesday 09:00 — python:3.12 = digest X
# ... dev smoke test passes ...
$ docker build -t scorer:prod .   # built Tuesday 11:30 — python:3.12 = digest Y (retagged 10:15)
# scorer:dev and scorer:prod are NOT the same bytes.
# Dev was green. Prod 500s on a transitive OS-layer change nobody made on purpose.

The two builds differ even though the diff between them is empty, because the build, not the code, drifted. The named failure mode is build-per-environment drift: the symptom is a build that passed dev failing in prod with no code change between them, and the engineer burns an afternoon diffing source that is byte-identical because the divergence is in a re-resolved base layer, not the repository. The boundary it violates is the assumption that source determines the artifact. Re-deriving the deployable for each environment reintroduces every non-determinism the build contains, the exact drift M5’s locked image and M6’s inheriting image were built to kill, now reappearing at deploy time instead of build time.

The unit that moves between environments is the built artifact, not the source

The thing validated in one environment must be the identical thing that runs in the next, or the validation describes a different artifact than the one customers receive. The correct model is that a deployable artifact can be named two ways. A mutable tag (scorer:latest, scorer:prod) is a human-movable pointer that can be re-pointed at a different artifact at any time. A content digest (sha256:50cf…8566) is a name derived from the bytes themselves, a hashed sum of the image contents, so it is fixed by the image and identifies exactly one immutable artifact forever. Promotion goes by the digest because it is the only name that cannot silently come to mean a different artifact.

The trap is promoting by the mutable tag to both environments, because the tag can be re-pointed between the dev deploy and the prod deploy, so “same name” can mean “different artifact” with no warning. The before-and-after here is build-twice versus build-once-promote-once. The scrolly below traces one image from build through both environments under each scheme.

The digest is the only name whose meaning the bytes fix. Below, the promotion records which digest passed dev and refuses to promote anything else, the discipline the scrolly’s tag scheme lacked.

"""Decide what prod actually runs under each naming scheme: digest vs mutable tag."""


def resolve_promotion(
    passed_dev_digest: str, prod_target: str, tag_now_points_to: str
) -> str:
    """Decide what prod actually runs under each naming scheme.

    passed_dev_digest: the digest the smoke test validated on dev.
    prod_target: what the pipeline tells prod to run -- a digest or a tag.
    tag_now_points_to: the digest a mutable tag resolves to at prod-deploy time.
    """
    if prod_target.startswith("sha256:"):
        # Promote by digest: prod runs exactly the validated bytes.
        return prod_target
    # Promote by mutable tag: prod runs whatever the tag points at NOW.
    return tag_now_points_to


if __name__ == "__main__":
    validated = "sha256:50cf8566"
    # A new build republished `scorer:latest` between the dev and prod deploys.
    tag_resolves_to = "sha256:99ab0001"

    by_digest = resolve_promotion(validated, "sha256:50cf8566", tag_resolves_to)
    by_tag = resolve_promotion(validated, "scorer:latest", tag_resolves_to)
    print("validated on dev:    ", validated)
    print(
        "prod by digest runs: ",
        by_digest,
        "(matches dev)" if by_digest == validated else "",
    )
    print(
        "prod by tag runs:    ",
        by_tag,
        "(MISMATCH -- untested)" if by_tag != validated else "",
    )

Promoting by digest makes prod run the validated bytes; promoting by tag lets a republish between deploys substitute an untested artifact, and nothing in the pipeline signals it. The mismatch line is the build-per-environment-drift failure wearing a deployment costume: the source was the same, the name was the same, the bytes were not. This is also the precondition the next lessons consume. One-step rollback is only possible because every shipped digest stays addressable, which promoting by an immutable content-derived name guarantees.

"""Try It: promote only the digest that passed dev.

Complete promote so it refuses to ship anything other than the validated digest
to prod, and accepts the validated one.
"""


def promote(validated_digest: str, candidate_digest: str) -> str:
    # Only the digest that passed dev is allowed into prod.
    # Return "PROMOTED <digest>" or "REJECTED -- not the validated artifact".
    return "PROMOTED " + candidate_digest  # placeholder -- too permissive


if __name__ == "__main__":
    print(promote("sha256:50cf8566", "sha256:50cf8566"))
    print(promote("sha256:50cf8566", "sha256:99ab0001"))

"""Solution: promote only the byte-for-byte validated digest."""


def promote(validated_digest: str, candidate_digest: str) -> str:
    # The digest is the identity. Promote only if the candidate IS the
    # artifact the smoke test validated -- byte for byte.
    if candidate_digest == validated_digest:
        return "PROMOTED " + candidate_digest
    return "REJECTED -- not the validated artifact"


if __name__ == "__main__":
    print(promote("sha256:50cf8566", "sha256:50cf8566"))
    print(promote("sha256:50cf8566", "sha256:99ab0001"))

Config and secrets: same image, a different environment

Promotion by digest gives a guarantee: the digest that passed dev is the digest serving prod. The mental model that feels right for per-environment values keeps the values with the code that reads them: the dev model path, the dev database URI, the dev API key all belong in the image, baked in, so the build is self-contained. Each environment gets its own image with its own values, built from the same source. That model quietly destroys the promotion guarantee.

The guarantee only holds if the image contains no environment-specific value. The moment a value that must differ between environments lives inside a layer, the dev image and the prod image must differ, so one digest can no longer be promoted, and the pipeline is back to building twice and hoping the two match. Two failure modes bracket this. Bake a secret into the image and it is in the registry and in the image’s command history permanently: the image config records the history of the commands run to build the image, so a secret passed at build time is recoverable from that history, and anyone who can read the image can read any file in any layer. Forget to set a required env var in one environment and the second failure shows up.

import os

# The "self-contained" image bakes a fallback so it "always works":
MODEL_PATH = os.environ.get("MODEL_PATH", "/opt/bundled/default_model.pkl")
#                                          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
# Prod forgot to set MODEL_PATH. The container boots. /health is 200.
# /predict loads the bundled default model and scores every Lending Club
# record against the WRONG model — plausibly, never crashing.

The container is healthy and serving, and the predictions are wrong. This is the named failure mode: a silent default-fallback skew. The symptom is no error at all (/health green, /predict returning probabilities), and the root cause is a missing required env var papered over by a “safe” default; the boundary it violates is that a required value was treated as optional. It is the train/serve skew class from the data-wrangling module (M2) wearing a deployment costume: the model serving traffic is not the model anyone validated, and the discovery arrives as a customer complaint, not a probe failure.

A promotable image must be environment-agnostic; the environment supplies what differs at run time

If one immutable artifact runs in every environment, then everything that must differ between environments cannot live inside the artifact. It has to be supplied from outside at the moment the container starts. The correct model separates the immutable code from the per-environment values it consumes: the artifact carries the behaviour, and the values that vary by environment are read from outside the artifact only at run time, never embedded in it, in the registry, or in git. The same artifact started in dev reads dev values and loads the dev model; the identical bytes started in prod read prod values and load the prod model. Store configuration in environment variables, not in files checked into the codebase, so the same image deploys to dev and prod without a per-environment rebuild.

The diagram below is the structural separation: one immutable image, two run-time environments, each supplying its own config and secrets. The image depends on nothing environment-specific inside itself.

[scorer  sha256:50cf...]  as img
[dev config + secrets]  as devcfg
[prod config + secrets] as prodcfg
[dev container]  as devrun
[prod container] as prodrun

img --> devrun  : same bytes
img --> prodrun : same bytes
devcfg --> devrun  : injected at run time
prodcfg --> prodrun : injected at run time

The image arrow into both containers is the same digest; the only thing that differs is the config injected at start. The discipline that makes a missing value loud instead of silent is to read every required config value at boot and exit non-zero if any is absent. A program can fail fast at startup when a required value is missing, so a misconfigured environment dies obviously on the next deploy instead of scoring quietly-wrong in front of customers. Below is that boot check, contrasted with the fallback that caused the skew.

"""Read required config at boot; exit non-zero if any required value is missing."""


def load_required_config(env: dict[str, str]) -> dict[str, str]:
    """Read required values at boot; exit non-zero if any is missing.

    A misconfigured environment dies obviously on deploy instead of
    serving a silent default. (env passed in so this is testable.)
    """
    required = ["MODEL_PATH", "MLFLOW_URI"]
    missing = [name for name in required if not env.get(name)]
    if missing:
        # In a real boot this is sys.exit(1) -- the container refuses to start.
        raise SystemExit("FATAL: missing required config: " + ", ".join(missing))
    return {name: env[name] for name in required}


if __name__ == "__main__":
    # Prod env, correctly populated:
    good = {
        "MODEL_PATH": "models:/loan_default/Production",
        "MLFLOW_URI": "https://mlflow.prod",
    }
    print("loaded:", load_required_config(good))

    # Prod env that forgot MODEL_PATH -- fail-fast catches it at boot:
    try:
        load_required_config({"MLFLOW_URI": "https://mlflow.prod"})
    except SystemExit as e:
        print(e)

The good environment loads; the misconfigured one refuses to start with a precise message naming the missing variable, instead of booting and scoring against a bundled default. The signal that a value was classified correctly: deleting any required var makes the container refuse to start. The strictness of this check is itself a tunable, namely how aggressively the boot fails on absent or unexpected values, and the right setting moves with the blast radius.

The curve runs from lenient (every absence defaults, which hides a missing required value) through the default (required fails fast, optional defaults) to strict (nothing defaults, every deploy must set everything). The signal of drifting too lenient is a silent-wrong incident discovered from a complaint; the signal of drifting too strict is deploys failing on optional knobs nobody set, which fatigues a team into copy-pasting config and reintroducing the drift the second environment was meant to remove.

The other failure this concept brackets is the baked secret, and it deserves its own treatment because the cost is different in kind. The decision is where a secret lives.

A baked secret is not merely a leak; it is a leak that cannot be revoked without rebuilding, because the secret lives in the image history and layer blobs and the only way to change it is a new image. Run-time injection turns rotation into a config change. This is the same hazard M5 taught for image layers, itself the M1 “a committed secret survives deletion” git hazard one layer up: deleting the line that set the secret does not remove the layer that captured it.

"""Solution: fail fast on required config, default only the genuinely optional."""


def boot_config(env: dict[str, str]) -> dict[str, str]:
    required = ["MODEL_PATH", "MLFLOW_URI"]
    optional = {"LOG_LEVEL": "INFO"}
    missing = [name for name in required if not env.get(name)]
    if missing:
        raise SystemExit("FATAL: missing required config: " + ", ".join(missing))
    config = dict(optional)
    config.update({name: env[name] for name in required})
    return config


if __name__ == "__main__":
    print(
        boot_config(
            {
                "MODEL_PATH": "models:/loan_default/Production",
                "MLFLOW_URI": "https://mlflow.prod",
            }
        )
    )
    try:
        boot_config({"MLFLOW_URI": "https://mlflow.prod"})  # MODEL_PATH missing
    except SystemExit as e:
        print(e)

Summary

• A single live environment makes “deploy succeeded” and “break production” the same command. “Deploy succeeded” reports a health probe (liveness, the process accepts connections), not readiness (the real route works), so a green /health can mask a /predict that 500s. The fix is a second identical environment plus a smoke test against the real route.
• Promote the built artifact, not the source. A Docker image is not a deterministic function of source, because a floating base tag is republished and build inputs are re-resolved, so building per environment can produce two different images from one commit (build-per-environment drift). Move one content digest between environments; it names exactly one immutable artifact.
• A mutable tag can be re-pointed between the dev deploy and the prod deploy, so “same name” can mean “different bytes.” Promote by the digest that passed dev, recorded explicitly, never by “the latest build.”
• A promotable image must be environment-agnostic: every value that differs between environments is injected at run time, never baked in. A baked secret persists in image history permanently and cannot be rotated without a rebuild.
• Read required config at boot and fail fast on any missing value. The strictness is a curve: too lenient hides a missing required value as a silent default-fallback skew; too strict fails deploys on optional knobs and fatigues teams into copy-pasted config.

Check your understanding:

• The platform reports “deploy succeeded” and customers immediately get 500s on /predict. Which signal did the platform actually check, and what would the dev environment have to do to catch this before promotion?
• Without looking back: why can two docker build runs of the same commit produce different images, and which name do you promote by to make that impossible to ship?
• A required MODEL_PATH is unset in prod and the container is happily returning probabilities. What is the name of this failure, and what one-line discipline at boot would have turned it into an obvious crash on deploy instead?