Make the Build Reproducible

I once owned the CI pipeline for a fleet of model services, and the Dockerfile that built them had a line near the top that read RUN pip install scikit-learn pandas numpy. It had worked for over a year. Then a Tuesday came where the same Dockerfile, the same git SHA, the same command produced an image whose model scored differently from the one shipped the week before, and the build that used to finish in seconds now took the better part of four minutes on every push. Nobody had touched the dependency line. The package index had moved underneath it, and the layer order meant every code edit was paying the full reinstall cost. It took the better part of a week to see that both problems came from the same two mistakes: nothing was pinned, and the install ran after the code was copied in.

The previous lesson built a working image: a Dockerfile that ran score.py, loaded the model from the classical-ML module, and scored a Lending Club record with one command. That image runs. This lesson confronts the gap between it runs and it builds the same, because the Lesson 1 image is reproducible only by accident. The pip install line in it is not a version, it is a request the index answers differently as time passes, and the layer order means a trivial code change throws away the most expensive work in the build. Both are reproducibility failures, and both have a precise mechanism. The work here is to freeze the dependency graph with the same uv lock file that pinned the project back in the Python module, copy that lock into the image, and order the Dockerfile so the install layer survives a code edit. The dominant mode of this lesson is debugging: a system that builds is broken in two ways that only surface later, and you find each one by forming a hypothesis about what changed and testing it.

Why an unpinned install drifts over time

Same git SHA, same Dockerfile, same build command, and a model that scores differently than it did last week: that is the broken system the Lesson 1 image hands you after a year of building fine. The hypothesis a competent engineer reaches for first is “something in our code changed” — a different git SHA, a stray edit, a bad merge. So you check the SHA. It is identical. You diff the Dockerfile against last week’s. Identical. The build command is byte-for-byte the same. Under the wrong mental model this is impossible: same inputs, same command, same output. The hypothesis has to be wrong, and the thing it gets wrong is what counts as an input to the build.

A bare package name is a constraint, not a version

A line like pip install scikit-learn does not name a version. It names a constraint — “the newest release that satisfies this” — and the resolver answers that constraint at build time, against the index as it exists the moment the build runs. The resolver works out the dependencies of the requested packages, the dependencies of those dependencies, and so on, picking the most recent version at each step until every node in the graph has one version that satisfies every constraint at once. The version that walk lands on is a function of the index’s state, not of the text in the Dockerfile.

Here is the line as it appears in a Dockerfile that builds today and a different image tomorrow. The failure is invisible in the source because the source did not change:

FROM python:3.12-slim
RUN pip install scikit-learn pandas numpy
COPY . .
ENTRYPOINT ["python", "score.py"]

The mechanism that lets one publish to the index reshape the install is worth making concrete. A new upstream release does two things at once: it becomes the new “newest satisfying” version for your bare constraint, and it can declare a different set of transitive dependencies than the previous release did. When that happens the resolver’s walk lands on a different graph — not one bumped version, but a different version of everything underneath that the new release pulls in. The block below simulates the resolver landing on a different version of the same constraint as the index gains a release, and prints that the constraint string was identical in both builds. Watch the resolved version move while the Dockerfile line stays put:

def resolve(constraint: str, available: list[str]) -> str:
    """Resolve a bare-name constraint to the newest available release.

    A bare name means 'any version', so the resolver returns the maximum.
    """
    versions = [v.split("==")[1] for v in available if v.startswith(constraint + "==")]
    newest = max(versions, key=lambda s: tuple(int(p) for p in s.split(".")))
    return constraint + "==" + newest


index_monday = ["pkg==1.4.0", "pkg==1.4.1"]
index_tuesday = ["pkg==1.4.0", "pkg==1.4.1", "pkg==1.5.0"]

print("constraint written in Dockerfile:", "pkg")
print("Monday build resolves to:        ", resolve("pkg", index_monday))
print("Tuesday build resolves to:       ", resolve("pkg", index_tuesday))
print("Dockerfile line changed?         ", False)

The resolved version moved from 1.4.1 to 1.5.0 with no edit to the line that requested it. This is the root cause of the Tuesday incident: a bare name is a standing order for “whatever is newest,” so the build is a snapshot of the index, not of your repository. The named failure mode is resolver drift — an image whose model scores differently from last week’s with an unchanged git SHA, because a numeric library underneath scikit-learn published a release that changed a default or a floating-point reduction path, and the unpinned constraint let it in. The symptom is a metric that moves with no diff; the root cause is that the install line never named a version; the boundary it violates is “the image is built from the repository,” because the index is an input the repository does not control.

The non-obvious cost is that the lint rule everyone reaches for does not actually fix this. hadolint’s DL3013 flags the bare form and tells you to pin versions in pip — so engineers pin the direct dependency to scikit-learn==1.5.0 and consider it solved. It is not solved. Pinning the direct dependency freezes one node in the graph and leaves every transitive dependency below it still resolving to “newest,” so the numeric library two levels down still drifts on Tuesday. Pinning is the right instinct applied at the wrong layer: it freezes the visible dependency and lets the invisible ones keep moving, which is exactly the part of the graph that produced the scoring change.

The same drift hides one layer below the install line, in the FROM itself, and it is the one most teams miss after they have locked their Python dependencies. FROM python:3.12-slim looks pinned — it names a version — but 3.12-slim is a mutable tag: the maintainers rebuild and republish it regularly, patching the OS, bumping the bundled OpenSSL, moving the system Python within the 3.12 line. The same Dockerfile, the same lock file, the same git SHA can pull a different base image next month, and a changed system library underneath your wheels can move a numeric result exactly the way an unpinned pip install does. This is resolver drift again, now in the foundation the whole image stands on, and the lock file does nothing about it because the lock file pins Python packages, not the image they install into. The fix is to pin the base image to its content digest rather than its tag: FROM python:3.12-slim@sha256:<hash>, where the digest is the immutable fingerprint of one exact image build. The standard guidance is to prefer a specific version tag over a mutable one like latest, and better still to pin by digest hash, so the exact same image is deployed across every environment regardless of when the build runs. The cost is a manual bump: a digest does not float forward to pick up security patches, so pinning by digest trades automatic updates for reproducibility, and you take the patch deliberately by updating the digest when you choose to, the same trade the lock file makes for Python packages. A tag answers “roughly this version, whenever you ask”; a digest answers “this exact image, forever.”

def resolve_newest(name: str, index: list[str]) -> str:
    versions = [v.split("==")[1] for v in index if v.startswith(name + "==")]
    # placeholder so the starter runs; replace with the real selection
    return name + "==" + (versions[0] if versions else "0")


build_a = ["model-lib==2.0.0", "model-lib==2.1.0"]
build_b = ["model-lib==2.0.0", "model-lib==2.1.0", "model-lib==2.2.0"]

print("A:", resolve_newest("model-lib", build_a))
print("B:", resolve_newest("model-lib", build_b))

def resolve_newest(name: str, index: list[str]) -> str:
    versions = [v.split("==")[1] for v in index if v.startswith(name + "==")]
    newest = max(versions, key=lambda s: tuple(int(p) for p in s.split(".")))
    return name + "==" + newest


build_a = ["model-lib==2.0.0", "model-lib==2.1.0"]
build_b = ["model-lib==2.0.0", "model-lib==2.1.0", "model-lib==2.2.0"]

print("A resolves to:", resolve_newest("model-lib", build_a))
print("B resolves to:", resolve_newest("model-lib", build_b))
print(
    "Same request, different answer:",
    resolve_newest("model-lib", build_a) != resolve_newest("model-lib", build_b),
)

The lock file is the frozen graph

Pinning every transitive version by hand in requirements.txt, one == per line, collapses past a handful of packages — a transitive graph cannot be enumerated by hand, and the moment one direct dependency bumps, every line below it has to be re-derived. The fix for resolver drift is not that. It is to stop sending a constraint to the index and start replaying a decision you already made: resolution is a step to run once, record, and never repeat inside a build. The thing that records the whole graph for you is a lock file.

What the lock captures and how it differs from a pin list

A lock file records the exact resolved version of every direct and transitive dependency, so reinstalling from it reproduces the same environment on another machine. This is the distinction the hand-pinned requirements.txt misses: pinning records the versions a human chose to name, while the lock records the versions the resolver actually landed on for the entire graph — including the transitive package two levels down that produced the Tuesday drift. The uv.lock you generated in the Python module is a cross-platform lockfile that captures the exact resolved versions that would be installed across all possible Python markers; it is checked into version control precisely so the install is consistent and reproducible across machines.

The block below models the difference between a hand-pin list and a full lock. A pin list names the top-level packages and stays silent on what they pull in; the lock enumerates the closure. Watch which transitive packages exist in the lock that the pin list never mentioned:

# A human-authored pin list: only the packages someone chose to name.
pin_list = {"scikit-learn": "1.5.0", "pandas": "2.2.0"}

# The full resolved closure the resolver actually landed on (what a lock records).
lock = {
    "scikit-learn": "1.5.0",
    "pandas": "2.2.0",
    "numpy": "2.0.1",  # transitive: pulled in by both, never pinned by hand
    "scipy": "1.14.0",  # transitive: pulled in by scikit-learn
    "joblib": "1.4.2",  # transitive: pulled in by scikit-learn
    "threadpoolctl": "3.5.0",
    "python-dateutil": "2.9.0",
}

unpinned = sorted(set(lock) - set(pin_list))
print("packages named in the pin list:", len(pin_list))
print("packages recorded in the lock: ", len(lock))
print("transitive deps the pin list left free to drift:", unpinned)

The pin list froze two nodes and left five transitive packages free to resolve to “newest” on the next build — numpy among them, the exact kind of numeric library whose release can shift a model’s scores. The lock froze all seven. The non-obvious cost of the lock is that this completeness is also its only failure mode: a lock is a snapshot of a resolution, so it can go stale relative to your declared dependencies. The named failure mode here is lock skew — someone edits pyproject.toml to add or bump a dependency but does not regenerate the lock, so the lock and the declared dependencies disagree. An install that silently trusts the stale lock ships an environment that no longer matches what the project says it needs, and the disagreement surfaces later as an import error or a missing feature in a container that “built fine.”

Install from the lock, do not re-resolve

A lock alone is not enough; the install command has to honor it instead of re-resolving against the index and defeating the entire point. uv sync --frozen installs from the lock file without re-resolving: it uses the versions in the lockfile as the source of truth instead of checking whether the lockfile is up to date. That is the command for the build — the lock is treated as the answer, the index is never consulted for versions. The block below contrasts what each install strategy reads as its source of truth, and what each one does when the lock and the live index disagree:

def install(
    strategy: str, lock: dict[str, str], index_newest: dict[str, str]
) -> dict[str, str]:
    """Return the versions that actually get installed under each strategy."""
    if strategy == "unpinned":
        # bare names: take whatever the index says is newest right now
        return dict(index_newest)
    if strategy == "frozen":
        # --frozen: lock is the source of truth, index is never consulted
        return dict(lock)
    raise ValueError(strategy)


lock = {"numpy": "2.0.1", "scipy": "1.14.0"}
index_newest = {"numpy": "2.1.0", "scipy": "1.14.1"}  # index moved since the lock

print("unpinned install gets:", install("unpinned", lock, index_newest))
print("frozen install gets:  ", install("frozen", lock, index_newest))
print("frozen ignores the moved index:", install("frozen", lock, index_newest) == lock)

The frozen install returns the locked versions even though the index has newer ones available, which is the whole guarantee: the build no longer depends on when it runs. There is a stricter sibling worth knowing for CI. uv sync --locked requires that the lockfile is up to date and exits with an error if it is missing or needs updating, where --frozen uses the lock as written without that staleness check. The trade-off is direct: --frozen is faster and never touches the network to verify, so it builds the same even from a stale lock — which means it will happily reproduce lock skew. --locked catches lock skew by failing the build, at the cost of a resolution check. The signal for which to use: run --locked in CI where a stale lock should fail the pipeline loudly, and --frozen in the production image build where the lock has already been verified upstream and the goal is a fast, network-free, deterministic install.

def build_image(
    strategy: str, lock: dict[str, str], index: dict[str, str]
) -> dict[str, str]:
    if strategy == "frozen":
        return {}  # placeholder: should return the locked versions
    if strategy == "unpinned":
        return {}  # placeholder: should return the index's newest
    raise ValueError(strategy)


lock = {"numpy": "2.0.1", "scipy": "1.14.0"}
index = {"numpy": "2.1.0", "scipy": "1.14.1"}

print("frozen:  ", build_image("frozen", lock, index))
print("unpinned:", build_image("unpinned", lock, index))

def build_image(
    strategy: str, lock: dict[str, str], index: dict[str, str]
) -> dict[str, str]:
    if strategy == "frozen":
        return dict(lock)  # lock is the source of truth; index ignored
    if strategy == "unpinned":
        return dict(index)  # take whatever is newest at build time
    raise ValueError(strategy)


lock = {"numpy": "2.0.1", "scipy": "1.14.0"}
index = {"numpy": "2.1.0", "scipy": "1.14.1"}

frozen = build_image("frozen", lock, index)
unpinned = build_image("unpinned", lock, index)
print("frozen ships numpy:  ", frozen["numpy"])
print("unpinned ships numpy:", unpinned["numpy"])
print("same numpy across builds run a week apart?", frozen["numpy"] == "2.0.1")

Layer order decides what a code edit costs

What does the lock not fix? The second half of the Tuesday incident: a build that used to take seconds now takes minutes on every push, even on pushes that change one line of score.py. Freezing what gets installed does nothing for how often the install gets paid for. The hypothesis a mid-level engineer holds here is that Docker caching is automatic — “Docker caches layers, so unchanged work is reused, so a one-line edit should be nearly free.” That model is half right, and the half it gets wrong is the half that costs four minutes a build.

The cache key and chain invalidation

Each Dockerfile instruction that writes or deletes files creates a layer, and Docker reuses a cached layer only if the instruction and the files it depends on have not changed since the last build. The cache key differs by instruction type, and this is the part the “caching is automatic” model misses. A plain RUN keys only on the command string — the files it touches are not examined. A COPY or ADD keys on a checksum computed from the copied files’ metadata, and the modification time (mtime) is explicitly not part of that checksum, so editing a copied file’s contents busts the COPY layer but merely touching it does not. The decisive rule sits on top of these: once any layer is invalidated, every layer after it is rebuilt regardless of whether its own inputs changed, because each layer is a diff applied on top of a specific parent — change the parent and the child has nowhere valid to attach.

The scrolly below walks a single code edit through a Dockerfile with COPY . . placed before the install, so you can watch one changed source file invalidate the copy layer and then cascade down through the expensive install:

The principle the reorder applies is to order instructions from most stable to most volatile: the most stable yet expensive instructions — heavy dependency installs, model downloads — belong at the start of the Dockerfile, and volatile fast operations like copying application code belong at the bottom. The reason is the chain rule above. A code edit invalidates its own copy layer and everything below it; if the expensive install sits below the code copy, the edit pays for it every time, and if the install sits above the code copy, the edit cannot reach it. The block below models the cache outcome of both orderings against the same one-line code edit, summing the cost of every layer that has to rerun:

# Cost in arbitrary "seconds" per layer when it must rebuild.
COST = {"FROM": 0, "COPY_LOCK": 1, "COPY_ALL": 1, "RUN_INSTALL": 240, "ENTRYPOINT": 0}


def rebuild_cost(order: list[str], changed: str) -> int:
    """Sum the cost of every layer at or after the first invalidated one.

    A code edit invalidates the first COPY layer that includes the file,
    then chain-invalidates everything below it.
    """
    invalidated = False
    total = 0
    for layer in order:
        if layer == changed:
            invalidated = True
        if invalidated:
            total += COST[layer]
    return total


code_first = ["FROM", "COPY_ALL", "RUN_INSTALL", "ENTRYPOINT"]
lock_first = ["FROM", "COPY_LOCK", "RUN_INSTALL", "COPY_ALL", "ENTRYPOINT"]

print("edit score.py, COPY . . first :", rebuild_cost(code_first, "COPY_ALL"), "s")
print("edit score.py, lock-first     :", rebuild_cost(lock_first, "COPY_ALL"), "s")

The code-first order pays the full install on a one-line edit; the lock-first order pays only the cheap copy, because the install layer sits above the changed file and its cache entry stays valid. In the model above, the same one-line score.py edit costs 241 units under the code-first order and 1 under the lock-first order: the install is 240 of those units, and the only thing that decides whether the edit pays it is which side of the code copy the install sits on. The named failure mode of the wrong order is cache-busting on every push: a build whose wall-clock time is roughly the full uncached time even on trivial diffs, because the slowest layer sits below the most frequently changed files. The cache is present and full; the order makes it worthless, because a change low in the Dockerfile invalidates every expensive layer beneath it. Ordering stable-and-expensive above volatile-and-cheap is the documented Docker guidance for exactly this reason.

The non-obvious cost of the lock-first order is the discipline it demands. Splitting COPY . . into a lock copy followed by an install followed by a full copy means the lock must be copied and installed before the code exists in the image, so any build step that needs both the code and the dependencies cannot live in the install layer. More subtly, the order is only correct as long as the lock genuinely changes less often than the code — which is the empirical claim that makes the whole optimization pay off, since a developer updates and changes program files far more often than dependencies. If your workflow churns the lock on every commit, the lock-copy layer invalidates as often as the code copy and the reorder buys nothing; the optimization is a bet on the relative volatility of two files, and it only wins when that bet holds.

Here is the reproducible Dockerfile the lesson builds toward — lock copied and frozen-installed first, code copied last, so a score.py edit never touches the install:

FROM python:3.12-slim
WORKDIR /app

# Stable + expensive: copy ONLY the lock metadata, then install from it.
# This layer is cached across every code edit because neither file below changes.
COPY pyproject.toml uv.lock ./
RUN uv sync --frozen --no-dev

# Volatile + cheap: the code comes last, so editing it invalidates only this layer.
COPY . .
ENTRYPOINT ["uv", "run", "python", "score.py"]

This is the order the whole lesson converges on: the expensive install layer sits above every file a developer edits, so a score.py change can only ever invalidate the cheap copy beneath it. The two fixes compose here — the lock freezes what installs, and the ordering freezes how often that install is paid for.

COST = {"FROM": 0, "COPY_LOCK": 1, "COPY_ALL": 1, "RUN_INSTALL": 240, "ENTRYPOINT": 0}


def rebuild_cost(order: list[str], changed: str) -> int:
    invalidated = False
    total = 0
    for layer in order:
        if layer == changed:
            invalidated = True
        if invalidated:
            total += COST[layer]
    return total


# Fix this order so a code edit does NOT rebuild RUN_INSTALL.
order = ["FROM", "COPY_ALL", "RUN_INSTALL", "ENTRYPOINT"]

print("cost of editing score.py:", rebuild_cost(order, "COPY_ALL"), "s")

COST = {"FROM": 0, "COPY_LOCK": 1, "COPY_ALL": 1, "RUN_INSTALL": 240, "ENTRYPOINT": 0}


def rebuild_cost(order: list[str], changed: str) -> int:
    invalidated = False
    total = 0
    for layer in order:
        if layer == changed:
            invalidated = True
        if invalidated:
            total += COST[layer]
    return total


order = ["FROM", "COPY_LOCK", "RUN_INSTALL", "COPY_ALL", "ENTRYPOINT"]

print("cost of editing score.py:", rebuild_cost(order, "COPY_ALL"), "s")
print("install layer rebuilt?   ", "RUN_INSTALL" in order[order.index("COPY_ALL") :])

Summary

• A bare pip install <pkg> is a constraint, not a version: the resolver answers it against the index at build time, so the same line ships different versions as the index moves — the root cause of a model whose scores drift with no code change.
• Pinning the top-level package freezes one node and leaves the transitive graph free to drift; the lock file records the exact resolved version of every direct and transitive dependency, which is the only thing that freezes the whole graph.
• uv sync --frozen installs from the lock as the source of truth and never re-resolves; --locked adds a staleness check that fails the build on lock skew — use --locked in CI, --frozen in the production image build.
• Docker reuses a cached layer only if its instruction and inputs are unchanged, and once any layer is invalidated every layer below it rebuilds — so COPY . . before the install makes a one-line code edit pay the full reinstall.
• Order from most-stable-and-expensive to most-volatile-and-cheap: copy the lock and install first, copy the code last, so editing score.py invalidates only the cheap final layer.

Check your understanding:

• The git SHA is identical to last week’s but the containerized model scores differently. What is the input you have not controlled, and what freezes it?
• A teammate moves COPY . . above RUN uv sync to “group the copies together.” Without running it, what happens to build time on the next one-line code edit, and why?
• When would you choose uv sync --locked over --frozen, and what specific failure does the --locked check catch that --frozen ships silently?