Make It Safe to Run Anywhere

The reproducible image from the last lesson ran the same everywhere, including a database URL and an API key I had hardcoded into score.py to get it working. When I pushed the image to a shared registry, the secret went with it, readable by anyone who pulled it. I have watched a credential live inside a Docker layer long after the source was “cleaned,” because deleting it in a later layer does not remove it from the earlier one. A reproducible image that ships its secrets is reproducibly insecure.

The previous lesson made the build deterministic: the uv.lock copied into the image, uv sync --frozen installing the recorded graph, the dependency layer ordered before COPY . . so a code edit no longer reinstalls everything. That fixed how the image is built. This lesson fixes what the built image carries when it moves between machines. Three things ride along that a working image gets wrong by default: the configuration that must differ between dev and prod, the secrets that must never be readable, and the files that quietly enter the image because nobody told the build to leave them out. The plan is one hardening pass per problem: config out of the image and into the environment, secrets out of every layer, and a single decision about what the build context is even allowed to contain.

Configuration belongs in the environment, not the image

A reasonable-looking move, once the image builds and runs, is to write the values it needs directly into the code or the Dockerfile: the database URL, the scoring threshold, the API endpoint. The image now runs end to end with no external setup, which feels like the finished state. It is the trap. An image with a value baked in is one image that works in exactly one place, and the entire reason to build an image was to run the same artifact everywhere.

A configuration value here is anything the program reads to decide where to run or how to behave (a database URL, an endpoint, a threshold) as opposed to the program logic itself. Watch the baked-in version look correct and then fail the moment the image leaves the machine it was built for.

# score.py — config baked into the code
DB_URL = "postgres://staging-db.internal:5432/scores"   # frozen at write time
THRESHOLD = 0.5                                           # frozen at write time

def main() -> None:
    write_scores(DB_URL)   # always staging, in every environment the image runs in

This image scores correctly. It also writes to the staging database in production, because the URL is a constant compiled into the artifact, and promoting the image to prod carries the staging endpoint with it. The failure is not visible at build time or at run time: the container starts, scores a record, exits zero. It is visible only in where the writes landed, days later.

The correct model is config in the environment, code in the image. The image is the same artifact everywhere; only the values injected into the running container differ. A value written at build time, by ENV in the Dockerfile or a literal in the source, is frozen into a layer, so changing it for a new environment means rebuilding and reshipping a different image, and now there are N images where there should have been one. An environment variable is a name-value pair the operating system hands to a process when it starts; Docker sets one on the container with docker run -e NAME=value, and the Python process reads it through os.environ. The injection happens at docker run time, against the container, not at docker build time against the image. That difference is the whole point: one built image reads dev config in dev and prod config in prod.

The Dockerfile makes a vocabulary distinction here, and confusing it is how config sneaks back into the image. ARG declares a build-time variable supplied to docker build; ENV sets a run-time environment variable that is written into the image and present in every container started from it. Setting a value with ENV API_ENDPOINT=... does not inject it at run time; it bakes it into the image exactly like a code constant, with the same one-image-per-environment problem. ENV is correct for values that genuinely belong to the image and never vary by environment (the PYTHONUNBUFFERED flag, a fixed install path); it is wrong for anything that differs between dev and prod. The rule that keeps them straight: if the value changes when the environment changes, it is injected with -e at run time, never set with ENV at build time.

Here is the same scorer reading both values from the environment, with defaults so it still runs with nothing injected. Watch the same code path produce a different scoring decision purely from an injected value, with no rebuild and no code change.

import os


def load_config() -> dict[str, str | float]:
    # read from the environment; fall back to a safe local default
    return {
        "db_url": os.environ.get("DB_URL", "postgres://localhost:5432/dev"),
        "threshold": float(os.environ.get("SCORE_THRESHOLD", "0.5")),
    }


def decision(probability: float, threshold: float) -> str:
    return "approve" if probability < threshold else "decline"


# default run: nothing injected
cfg = load_config()
print("default threshold:", cfg["threshold"], "->", decision(0.55, cfg["threshold"]))

# simulate `docker run -e SCORE_THRESHOLD=0.6`: the orchestrator sets the env, then the
# process reads it -- same code, different injected value
os.environ["SCORE_THRESHOLD"] = "0.6"
cfg = load_config()
print("injected threshold:", cfg["threshold"], "->", decision(0.55, cfg["threshold"]))

The probability 0.55 did not move and the function did not change, yet the decision flipped from decline to approve because the injected threshold crossed it. That is the property the baked-in version threw away: behaviour that the deploying environment controls, on one immutable image. The container invocation that drives it is docker run -e SCORE_THRESHOLD=0.6 -e DB_URL=postgres://prod-db:5432/scores loan-scorer. The image is identical to the one running in dev; the -e flags are the only difference.

The named failure here is config drift through promotion. A “dev” image with a hardcoded staging database URL gets promoted to prod unchanged, the standard promotion flow with the same artifact moving up the environments, and it writes production scores to the staging table because the endpoint travelled inside the image. The symptom is data landing in the wrong place with no error anywhere; the root cause is that the config crossed the image boundary it should never have been on. The non-obvious cost of the “working” way is that it stays invisible until the artifact is promoted, which is exactly when it is hardest to catch: the build passed, the tests passed, the container ran. The boundary it violated is the one between the artifact and its environment: anything that varies by environment has to be kept outside the artifact, or promotion silently carries the wrong value.

import os

# Refactor these two constants into environment reads with defaults.
THRESHOLD = 0.5
DB_URL = "postgres://staging:5432/scores"


def load_config() -> dict[str, str | float]:
    # TODO: read THRESHOLD from SCORE_THRESHOLD and DB_URL from DB_URL, with defaults
    return {"threshold": 0.5, "db_url": "postgres://staging:5432/scores"}


cfg = load_config()
print("threshold:", cfg["threshold"])
print("db_url:", cfg["db_url"])

os.environ["SCORE_THRESHOLD"] = "0.7"
os.environ["DB_URL"] = "postgres://prod:5432/scores"
cfg = load_config()
print("threshold:", cfg["threshold"])
print("db_url:", cfg["db_url"])

import os


def load_config() -> dict[str, str | float]:
    return {
        "threshold": float(os.environ.get("SCORE_THRESHOLD", "0.5")),
        "db_url": os.environ.get("DB_URL", "postgres://staging:5432/scores"),
    }


# no injection: defaults
cfg = load_config()
print("threshold:", cfg["threshold"])
print("db_url:", cfg["db_url"])

# prod-style injection (what `docker run -e ...` would set)
os.environ["SCORE_THRESHOLD"] = "0.7"
os.environ["DB_URL"] = "postgres://prod:5432/scores"
cfg = load_config()
print("threshold:", cfg["threshold"])
print("db_url:", cfg["db_url"])

One typed settings object, not scattered reads

The os.environ.get(name, default) pattern above is correct, and it does not scale. It works at two values; a real service reads a dozen, and the scattered form rots in four specific ways. Every call site re-states its own default, so two reads of the same variable can disagree about what “unset” means. Coercion is hand-rolled at every numeric read — the float() in the block above — and a read that forgets the cast compares a string against a number. An empty string sails straight through as a present value, because get only distinguishes set from unset. And the worst one fails open: a required value given a “safe” fallback boots anyway, /health goes green, and the service quietly runs against the default instead of refusing to start — the exact silent skew the cloud-deployment module later dissects. On top of all four, there is no single place a reviewer can see the service’s whole configuration; it is smeared across every file that happens to read the environment.

The standard that fixes all four at once is one typed settings object: a Settings class built on pydantic-settings’ BaseSettings, conventionally in a settings.py beside the app. Every config value becomes a typed field. A required value declares no default, so a missing variable raises a ValidationError at construction — at startup, naming the field — instead of booting on a fallback. An optional value declares its default once, in the one place a default belongs. Coercion comes from the type: the environment always hands over strings, and the field’s annotation turns "0.6" into the float the comparison needs, with no hand-written cast at any call site. The class reads the process environment (and a .env file in development) at construction, so the entire runtime surface of the service is one reviewable object.

"""The typed settings object: one class, validated at construction, fail-fast.

The scattered os.environ version works and then rots: every call site re-states
its own default, coercion is hand-rolled per read (float(...)), an empty string
sails through as a value, and a missing REQUIRED value silently falls back
instead of refusing to start. pydantic-settings collapses all of it into one
typed Settings class -- the curriculum's standard shape for runtime config,
conventionally a settings.py next to the app.
"""

import os

from pydantic import ValidationError
from pydantic_settings import BaseSettings


class Settings(BaseSettings):
    db_url: str  # required: no default, so a missing value refuses to start
    score_threshold: float = 0.5  # optional: typed default, coerced from the env string


# The orchestrator injects config at run time (docker run -e ...); simulate it:
os.environ["DB_URL"] = "postgres://prod-db:5432/scores"
os.environ["SCORE_THRESHOLD"] = "0.6"  # env vars always arrive as strings

cfg = Settings()
print("db_url:         ", cfg.db_url)
print(
    "score_threshold:", cfg.score_threshold, f"({type(cfg.score_threshold).__name__})"
)

# The misconfigured environment: the required DB_URL was never injected.
del os.environ["DB_URL"]
try:
    Settings()
except ValidationError as e:
    first = e.errors()[0]
    print("refused to start:", first["loc"][0], "--", first["msg"])

The injected string became a float because the field’s type did the coercion, and the missing DB_URL refused to start with the field named — the fail-fast that turns a misconfigured deploy into an error at boot instead of a wrong-model skew discovered from a customer complaint. Two boundaries are worth keeping straight from here on. This settings object is runtime configuration: the per-environment values injected at docker run. It is not build configuration — pyproject.toml and the lock file, which declare how the artifact is built and are committed to the repo. A value belongs to exactly one of those layers, and a secret belongs in neither file: it is injected at run time and read by this class, never written into the image or the repo. The serving module later puts this same pydantic machinery at the request boundary, validating what callers send; the cloud-deployment module hand-writes this exact fail-fast check once to make the mechanism visible, then points back to this class as the tool that does it for you.

Why a secret in a layer is still there after you remove it

Once config is injected instead of baked, the next instinct is to treat secrets the way you would a temporary file: COPY the credentials in, use them during the build, then RUN rm them before the image is finished. The final ls shows the file gone, docker run confirms it is absent, and the image looks clean. It is not. The mental model that fails here is that the final filesystem is what ships. It is not: the stack of layers is what ships, and rm does not erase a layer.

A Docker image is not a single flattened filesystem. It is an ordered stack of immutable, content-addressed layers, each one the filesystem diff produced by a single Dockerfile instruction. What the running container sees is a union view the overlay storage driver (the one most Linux hosts use) computes on the fly by merging the layers from the top down, where a file in an upper layer shadows the same path in a lower one. The layers themselves never merge into one copy; docker history, docker save, and a registry pull all expose every layer individually.

The trap is what RUN rm secret.txt actually does to that stack. Watch the secret get added in one layer and “removed” in the next.

FROM python:3.12-slim
COPY config/credentials.txt /tmp/credentials.txt   # layer N: writes the secret bytes
RUN setup-using-secret /tmp/credentials.txt && \
    rm /tmp/credentials.txt                         # layer N+1: "removes" it

The merged view after this build shows no /tmp/credentials.txt. The bytes are still in layer N. The overlay filesystem cannot delete a file out of a lower read-only layer, because a layer is immutable once written, so instead it records the deletion in the upper layer as a whiteout: a special marker entry, implemented as a character device with device number 0/0 or a zero-size file carrying the trusted.overlay.whiteout extended attribute, that tells the union view “treat this path as absent.” A whole removed directory is masked with an opaque-directory flag (trusted.overlay.opaque=y) instead. The merged view obeys the marker and the file looks gone; layer N still holds the complete, intact secret, addressable directly by checking out that layer or reading it out of docker history. The same copy-on-write model is why a modified lower-layer file leaves its original bytes untouched: overlay copies the file up before editing the copy.

ENV API_KEY=... is worse than the file case. A file lives in a layer’s filesystem, which at least requires extracting a layer to read. An ENV value is not in any layer’s filesystem at all: it is stored in the image’s config metadata in plain text, alongside the build history, and docker inspect prints it directly with no extraction. The build history records the command that set it, so even the step that wrote the value is visible. There is no rm for image config; the value ships and can be read by anyone with the image.

Here is the mechanism modelled as plain Python: layers as an append-only list, a deletion that records a whiteout marker instead of removing the entry. Watch the merged view report the file as gone while the history still holds it.

def merged_view(layers: list[dict[str, str]]) -> dict[str, str]:
    # build the union view top-down: upper layers shadow lower ones; a whiteout hides a path
    view: dict[str, str] = {}
    whiteouts: set[str] = set()
    for layer in reversed(layers):  # top layer first
        for path, contents in layer.items():
            if contents == "<whiteout>":
                whiteouts.add(path)
            elif path not in view and path not in whiteouts:
                view[path] = contents
    return view


# layers are strictly additive; nothing is ever removed from one once written
layers: list[dict[str, str]] = [
    {"app.py": "print('score')"},  # layer 0
    {"credentials.txt": "API_KEY=sk-live-9f3"},  # layer 1: secret written
    {"credentials.txt": "<whiteout>"},  # layer 2: `rm` records a whiteout
]

print("merged view (what the container sees):", merged_view(layers))
print("layer 1 still holds:", layers[1]["credentials.txt"])

The merged view reports no credentials.txt, so a casual docker run ... ls confirms it gone, yet layers[1] still contains the live key, exactly what docker history plus a layer dump would recover. The whiteout in layer 2 changed only what the union view presents, not what the stack contains. This is why a leaked secret cannot be cleaned by a later commit: every machine that pulled the image already copied the immutable lower layer that holds it.

The two safe paths never write the secret to a layer at all. The first is run-time injection: pass the secret with docker run -e API_KEY=..., or mount it from a secret store the container reads at start, so the value lives only in the running container and is gone when it stops. Be precise about what -e buys, because “safe” here means safer than a layer, not invisible: an injected variable is still printed by docker inspect on the running container, inherited by every process the container spawns, and one careless debug statement away from a log line — which is exactly why the file-mount form (a secret mounted at a path only the service reads) is the stricter choice on any host other engineers can inspect. The second is BuildKit’s RUN --mount=type=secret, for the case where a secret is genuinely needed during the build, say to authenticate a private package download. The secret is exposed to that one RUN instruction through a tmpfs mount and is never committed to any layer, so it is present for the build step and absent from the final image. Build arguments and ENV are explicitly the wrong tools for this, because they persist in the image.

The named failure here is the un-removable leaked credential. A registry image is found to contain an API key that was “removed” several commits earlier; the cleanup added a whiteout layer, but the key sits in the original layer and comes out of docker history and a layer dump. The symptom is a live credential in a shipped artifact; the root cause is the append-only layer model meeting a RUN rm that only shadows. The non-obvious cost, the part that catches teams, is that the remediation is not “delete it and rebuild.” Because every pull copied the immutable lower layer, the credential is already in every cached copy of that image everywhere. It must be rotated, invalidated at the source, not just removed, and the replacement image must never have written it in the first place.

# These layers model: COPY creds in, RUN rm creds out.
layers: list[dict[str, str]] = [
    {"app.py": "code"},
    {"creds.txt": "DB_PASSWORD=hunter2"},  # COPY
    {"creds.txt": "<whiteout>"},  # RUN rm
]


def secret_still_in_image(layers: list[dict[str, str]]) -> bool:
    # TODO: return True if any layer's stored bytes still contain the live secret,
    # regardless of whiteouts in later layers
    return False


print("secret recoverable from history:", secret_still_in_image(layers))
print("safe fix: replace this string with how to get the secret in without a layer")

layers: list[dict[str, str]] = [
    {"app.py": "code"},
    {"creds.txt": "DB_PASSWORD=hunter2"},  # COPY: bytes written into this layer
    {"creds.txt": "<whiteout>"},  # RUN rm: records a whiteout, bytes untouched
]


def secret_still_in_image(layers: list[dict[str, str]]) -> bool:
    # walk every stored layer; a real secret value is recoverable even if a later
    # layer whiteouts the path in the merged view
    for layer in layers:
        for value in layer.values():
            if value != "<whiteout>" and "PASSWORD" in value:
                return True
    return False


print("secret recoverable from history:", secret_still_in_image(layers))
print(
    "safe fix: inject at run time with `docker run -e DB_PASSWORD=...`, "
    "or use BuildKit `RUN --mount=type=secret` for build-time use"
)

Keep it out of the build context: .dockerignore and what COPY . . actually sends

The two leaks above came from instructions you can see: a COPY secret.txt, an ENV API_KEY. The sharper version happens with no such line at all. The natural reading of COPY . . is “copy my project directory into the image,” and the natural assumption is that it copies from the disk in front of you. Neither is quite right, and the gap is where a .env ends up in a layer without anyone writing a single line that mentions it.

COPY does not read your disk. It reads the build context, a snapshot of the build directory that the Docker client packages and transfers to the daemon before any instruction runs, because the daemon, not the client, executes the build and can only copy files it has received. The first line of docker build output, “transferring context,” is that upload. Two consequences follow, and both are failures. The first is cost: the entire context is sent every build, so whatever sits in the directory inflates both the upload and the build the daemon holds, including .git history, the local virtualenv, and a multi-gigabyte data sample. The second compounds the first into a leak: everything in that uploaded context is eligible for COPY . ., which sweeps the whole snapshot into a layer.

Watch what COPY . . actually pulls in when the context is left at its default of “the entire directory.”

# project/ contains: score.py, model.pkl, .git/, .venv/, data/sample.parquet, .env
FROM python:3.12-slim
WORKDIR /app
COPY . .                      # copies the WHOLE context: .git, .venv, data, AND .env
ENTRYPOINT ["python", "score.py"]

The image now carries the git history, a redundant copy of the virtualenv, a large data file the scorer never reads, and the dangerous one, the .env holding a database credential, baked into a permanent layer. This is the exact leak from the previous section, except there was no COPY secret.txt and no RUN rm to even notice: the secret rode COPY . . into a layer purely because nothing told the build to exclude it.

A .dockerignore file lists path patterns the client excludes from the context before sending it, so ignored files are never transferred and can never be COPYd. The exclusion happens at packaging time, upstream of the build: an ignored .env is not in the snapshot the daemon receives, so no instruction, deliberate or accidental, can reach it. This is the direct Docker analog of the .gitignore from the Python module: the same judgment about what belongs in a tracked boundary, applied to a different boundary. Where .gitignore decides what enters git history, .dockerignore decides what enters the build context.

A cache interaction makes the ignore file matter even when nothing leaks. The previous lesson established that COPY’s cache key is a checksum of the files being copied. A churning file left in the context, whether .git’s changing refs, a regenerated model.pkl, or a touched log file, changes that checksum on builds that touched no source, busting the COPY . . cache layer and re-triggering everything chained after it. So an unignored, frequently-changing file silently turns cached rebuilds into full ones, the same way an unpinned dependency turned reproducible builds into drifting ones. Excluding it restores the cache.

Here is the classification made concrete: given an inventory of files tagged source, regenerable, large, or secret, the same rule that drives .gitignore decides what the context may contain. Watch every non-source category get excluded, with the reason attached.

inventory: dict[str, str] = {
    "score.py": "source",
    "model.pkl": "source",  # the model artifact the image needs at run time
    ".git/": "regenerable",  # rebuildable from the remote; never needed in the image
    ".venv/": "regenerable",  # reinstalled from the lock; redundant in the image
    "__pycache__/": "regenerable",
    "data/sample.parquet": "large",  # belongs mounted or in object storage, not baked in
    ".env": "secret",  # the accidental-leak path
}


def dockerignore_lines(inventory: dict[str, str]) -> list[str]:
    # source enters the context; everything regenerable, large, or secret is excluded
    excluded = []
    for path, tag in inventory.items():
        if tag != "source":
            excluded.append(path + "    # excluded: " + tag)
    return excluded


for line in dockerignore_lines(inventory):
    print(line)

Every line that is not source ends up in the ignore file, each with the reason it is out: regenerable files are rebuilt from the lock or the remote, the large data file belongs outside the image entirely, and the .env exclusion is the single line that closes the accidental-secret path. The two source files, the scorer and the model artifact, are all the context needs to carry.

The decision is not three tools; it is one rule applied at three strictness levels. The principle underneath every level is the same: let source into the context; keep regenerable, large, and secret files out. What changes across the levels is the default, meaning what happens to a file nobody has classified yet.

The allowlist level is worth the maintenance cost on any project that handles credentials, because it inverts the failure direction. With the exclude-list approach, the cost of forgetting a line is a leak: a new .secret file ships because nobody added it to the ignore list. With the allowlist, the cost of forgetting a line is a build error: a needed file does not get copied, and the build fails loudly. A build that fails is recoverable; a credential in a pushed image is not.

The named failure here is the silent context leak. A build that “works” is shipping the local .venv and a multi-gigabyte data sample into the context on every run, and a .env in the same directory rides COPY . . straight into a layer. The symptom is a build that uploads far more than it should and an image carrying data and a credential nobody put there on purpose; the root cause is the absence of a single .dockerignore line. It is the leak from the previous section reached by a completely different path: not a careless COPY secret.txt, but the default context including a file no one classified. A smaller context is also the cheapest size cut available, ahead of the base-image and multi-stage work in the next lesson: excluding the venv and the data sample shrinks the upload and the image before a single dependency is touched.

inventory: dict[str, str] = {
    "predict.py": "source",
    "pipeline.joblib": "source",
    ".git/": "regenerable",
    ".venv/": "regenerable",
    "data/train.csv": "large",
    ".env": "secret",
    "notes.log": "regenerable",
}


def build_dockerignore(inventory: dict[str, str]) -> list[str]:
    # TODO: return one "path  # reason" line per excluded (non-source) file
    return []


def leak_preventing_entry(inventory: dict[str, str]) -> str:
    # TODO: return the path whose exclusion stops the COPY . . secret leak
    return ""


for line in build_dockerignore(inventory):
    print(line)
print("prevents the leak:", leak_preventing_entry(inventory))

inventory: dict[str, str] = {
    "predict.py": "source",
    "pipeline.joblib": "source",
    ".git/": "regenerable",
    ".venv/": "regenerable",
    "data/train.csv": "large",
    ".env": "secret",
    "notes.log": "regenerable",
}


def build_dockerignore(inventory: dict[str, str]) -> list[str]:
    return [path + "    # " + tag for path, tag in inventory.items() if tag != "source"]


def leak_preventing_entry(inventory: dict[str, str]) -> str:
    for path, tag in inventory.items():
        if tag == "secret":
            return path
    return ""


for line in build_dockerignore(inventory):
    print(line)
print("prevents the leak:", leak_preventing_entry(inventory))

Summary

• Config in the environment, code in the image. A value baked at build time (a source constant or ENV) freezes into a layer and forces one image per environment; an environment variable injected with docker run -e and read via os.environ lets one image read dev config in dev and prod config in prod. ARG is build-time, ENV is run-time-but-baked, and neither is the tool for a value that varies by environment.
• A secret in a layer is not removed by RUN rm. Layers are append-only, immutable diffs; the overlay driver records a deletion as a whiteout marker in an upper layer while the bytes stay intact in the lower one, so docker history reads past the whiteout. ENV secrets are worse, stored in image config metadata in plain text. Inject at run time or use BuildKit --mount=type=secret, and rotate anything that already shipped.
• COPY . . copies the build context, not your disk. The client packages and uploads the whole directory before any instruction runs; without a .dockerignore that snapshot includes .git, the venv, large data, and any .env, and COPY . . can sweep a credential into a permanent layer with no instruction naming it. The ignore file excludes by the same rule as .gitignore: source in, regenerable/large/secret out.

Check your understanding:

• Without looking back: why does promoting a “dev” image with a hardcoded staging URL to prod write to the wrong database, and where should that URL have been set instead?
• A teammate says they deleted a leaked API key in a later layer and rebuilt, so the image is clean now. Why are they wrong, and what is the actual remediation?
• Name two distinct failures a missing .dockerignore causes, one about build cost and cache and one about secrets, and state what COPY . . actually reads from.