Make It Reproducible

I once cloned a working repo, ran the training script exactly as documented, and got a measurably different model than the one already in production. I lost half a day before I found it. My pandas version differed from the one that had trained the deployed model, and a behaviour had changed between the two. Nothing in the repo recorded which version had produced the production model, so I couldn’t even confirm my clone was the problem rather than the code. “It works on my machine” was literally true and completely useless. The package worked. It just wasn’t reproducible, and for ML that gap is the difference between a result and a guess.

Your package now imports cleanly, runs through __main__, and validates its inputs. There is one gap left before another engineer can rely on it: it has to run the same on their machine as on yours. For ordinary software, “different version, slightly different behavior” is an annoyance. For ML it is worse. A different dependency version can change a numerical result, so you cannot even tell whether a model regression came from your code or from a library that moved underneath you. So this lesson pins the environment so the package reproduces, then handles the commit-side decisions that keep that promise intact.

Why environments must be isolated

You created a virtual environment back in Lesson 1 and I told you the why was coming later. This is later. A per-project environment is non-negotiable, not just tidy, and here is the reason. A global interpreter has exactly one site-packages, a single flat namespace shared by everything that uses that Python. And site-packages holds exactly one version of each package. There is no per-project slot.

So picture two projects on one global Python. Project A needs pandas==1.5. You set up project B and it installs pandas==2.2. That install does not fail or warn. It just overwrites the pandas that A was relying on. A keeps running on whatever it last imported, then breaks the next time it is imported fresh or hits a changed API, with an error that points at A even though nothing in A changed. The cause and the symptom are in different projects, days apart, and nothing errored at the moment of the actual mistake. That is the failure isolation prevents.

For an ML project there’s a second, quieter cost that no ImportError ever surfaces. The version bump that doesn’t break the API still moves the numbers. A scikit-learn or numpy release can change a default solver, a tie-breaking rule, or the BLAS routine under a matrix multiply, and your model’s metric shifts by a fraction of a point. Code unchanged, data unchanged, only the environment moved underneath you. Here’s the trap: this is indistinguishable from a real model regression. You re-run training, the AUC is down two-tenths of a point, and you start hunting a data or code change that doesn’t exist. A pinned, isolated environment is what lets you say “the only thing that changed is the thing I changed,” which is the precondition for trusting any before/after comparison at all. Isolation isn’t just conflict-avoidance. It’s what makes a numerical result attributable.

A virtual environment fixes it by giving each project its own site-packages. uv venv creates a separate interpreter prefix with a private namespace, so A’s pandas==1.5 and B’s pandas==2.2 live in different directories and both requirements are satisfiable at once. The global install’s “last one wins” cannot happen, because there is no shared slot to win. Lesson 1 had you make and activate one. The point now is that without it, the last unrelated install silently mutates every other project sharing the interpreter, a failure with no error message at the moment it is caused.

One tooling note, so the commands you see elsewhere map cleanly onto this course. uv venv and the standard library’s python -m venv produce the same artifact: an ordinary virtual environment, a directory any tool can activate, inspect, or invoke by path. Nothing about the environment locks you into uv. This course standardizes on uv venv for two reasons. One is speed (it creates the env near-instantly, and its installer resolves in a fraction of pip’s time). The other is workflow: the same tool that makes the env also writes and enforces the lock file this lesson is about, so uv venv, uv sync --locked, and uv run form one coherent loop instead of three tools glued together. The uv run piece matters beyond convenience. It executes a command inside the project env with no activation step, which is the same no-shell-state discipline the package-mechanics lesson flagged for schedulers and CI. An activated shell is a mutation of one terminal session, and uv run (or the env’s .venv/bin/python by full path) is how a cron job or CI step uses the env without depending on shell state it doesn’t have.

# Global Python, one shared site-packages.
# Project A: relies on pandas==1.5
# Project B: you run `pip install pandas==2.2`
#
# What breaks? In which project? At what moment (not at install time)?
# One-line fix?
answer = "???"
print(answer)

answer = (
    "Project A breaks (not B). B's install silently overwrites the shared pandas; "
    "A fails the next time it imports/calls the changed API -- days later, pointing "
    "at A though A never changed. Fix: a per-project venv (uv venv) so each has its "
    "own site-packages and both pandas versions coexist."
)
print(answer)

Lock the dependencies

Isolation gives each project its own dependencies. It doesn’t make those dependencies the same over time. A loose line like pandas in pyproject.toml installs whatever is newest at install time, so two installs from the identical repo days apart can produce two different environments. The fix is a lock file that records the exact resolved versions. This is the precondition for everything in the serving and deployment modules, where the build has to land on the same graph every run.

Here is the mechanism, because it explains why pyproject.toml alone cannot save you. A line like pandas (or pandas>=2) is a constraint, not a version. At install time a resolver turns the whole set of constraints into one concrete set of packages. It picks a version of each direct dependency that satisfies its constraint, reads that version’s own dependencies, picks versions for those, and walks the entire dependency graph until every package has a single version satisfying every constraint at once. Two things make that non-deterministic across time. The “newest satisfying version” of any node changes as packages publish releases, and each new version can declare different transitive dependencies. So re-running the resolver next week, from the identical pyproject.toml, can produce a different graph.

A lock file records the output of one successful resolution: the exact pinned version of every direct and transitive package, often with content hashes. A later install skips re-resolving and reinstalls that exact graph. Think of the response curve this way. Loose constraints make reproducibility impossible, because the graph drifts. Hand-pinning every package gets you reproducible, but now you fight the resolver by hand on every transitive bump. A lock file gives you both: reproducible, and the resolver still does the solving. You just freeze its answer.

A lock file is only half of it. Installing from it is the other half, and the gap between the two is where a committed lock quietly stops protecting you. A plain uv sync or uv pip install, if pyproject.toml has changed since the lock was written, will re-resolve and silently update the lock to match. That’s convenient locally and dangerous anywhere reproducibility matters, because it means the environment a build produces is no longer guaranteed to be the one you froze. The discipline is to assert in CI and any deployment build that the lock is still consistent with pyproject.toml, and uv splits this into two flags whose difference is exactly the one that matters here. uv sync --locked asserts the lock will not change: it fails the build if pyproject.toml and the lock have drifted apart, instead of papering over the drift by resolving again. uv sync --frozen does something different. It installs the lock exactly as written and does not update it, but it also does not check whether pyproject.toml has drifted. It simply uses whatever the lock says and ignores the mismatch. So the flag that turns “someone edited a dependency and forgot to update the lock” into a red build is --locked. --frozen is for when you want to install the locked graph without even touching pyproject.toml (an offline or fast-path install), and it will happily run against a stale lock. For the reproducibility gate this lesson is about, catch the drift and fail loudly, reach for --locked. A lock file you never assert against with --locked is a lock that can rot in place while still looking committed and current.

First, see what a single resolution actually produces, the pinned version, not the constraint:

"""Lesson 5.2 Show — the lock file is what makes the install deterministic.

`pyproject.toml` declares loose dependencies (`pandas`); the lock file records the
exact resolved version (and every transitive dependency) so a fresh install on
another machine gets the same graph. Unpinned, two installs months apart can resolve
different versions and behave differently. The print below shows exactly which
version is actually installed — the thing the lock pins so it is the same everywhere.
"""

import numpy as np
import pandas as pd
import pydantic

print(f"pandas:   {pd.__version__}")
print(f"numpy:    {np.__version__}")
print(f"pydantic: {pydantic.VERSION}")
print("a lock file pins these exact versions; pyproject.toml alone does not")

The version-print shows only the result of one resolution. It cannot show the drift, because a single run sees a single version. The drift is visible only when you resolve the same constraints twice across time. Here is what that looks like: one pyproject.toml with one loose line, compiled to a concrete graph in January and again in March, diffed.

# pyproject.toml — unchanged between the two runs
#   dependencies = ["pandas"]        # one loose constraint, no version

# $ uv pip compile pyproject.toml      (run in January)
#   pandas==2.1.4
#   numpy==1.26.2        # pandas 2.1.4 requires numpy>=1.23
#   python-dateutil==2.8.2
#   pytz==2023.3

# $ uv pip compile pyproject.toml      (run in March, SAME pyproject.toml)
#   pandas==2.2.1        # newest satisfying "pandas" moved
#   numpy==1.26.4        # pandas 2.2.1 now requires numpy>=1.26.0 — graph changed
#   python-dateutil==2.9.0
#   pytz==2024.1
#   tzdata==2024.1       # NEW transitive dep pandas 2.2 pulled in

# Same constraint, two different graphs — and a package that did not exist
# in the first resolution (tzdata) is now in the environment.

That’s the non-determinism the prose described, made concrete. The loose line didn’t change, but “newest satisfying version” moved, and the newer pandas declared a transitive dependency the older one did not, so the shape of the graph changed, not just a version number. A lock file is the frozen output of the first compile: every later install replays that exact graph instead of re-resolving, so January’s environment and March’s are byte-for-byte identical. pyproject.toml says “a pandas.” The lock file says “this pandas, and these transitive packages, at these hashes.”

Let’s watch the resolver build that graph node by node, and then a fresh resolve drift after a new release.

Why can two installs of the same unlocked project differ, and what does the lock file capture that pyproject.toml does not?

# pyproject.toml: dependencies = ["pandas", "scikit-learn"]   (no lock file)
# Teammate clones next month -> different environment -> different model.
#
# Why can the same pyproject.toml resolve differently a month later?
# What does a lock file capture that pyproject.toml does not?
why = "???"
print(why)

why = (
    "pyproject.toml lists CONSTRAINTS, not versions. The resolver picks the newest "
    "satisfying version of each package plus its transitive deps -- and 'newest' "
    "changes as releases ship, so a fresh resolve a month later yields a different "
    "graph. The lock file freezes ONE resolution: exact versions of every direct AND "
    "transitive package (often with hashes), so the install reproduces that graph."
)
print(why)

A locked dependency graph is necessary for a reproducible model, but be honest about its limit. It is not sufficient, and assuming it is leads you to trust a number that can still move. Reproducibility is genuinely harder in machine learning than in ordinary software, because training is not deterministic by default. A model is initialized with random values and adjusted from there, so two runs of the identical code on the identical data produce two different models unless you fix the randomness. The lock pins the libraries. You still have to pin the seed (random_state on every estimator and split that takes one, the M2 and M3 discipline) and the Python version itself, since a pinned pandas running on Python 3.11 versus 3.12 can still behave differently. That is why requires-python and pinning the interpreter (uv python pin) belong alongside the lock. And even with deps, seed, and interpreter all fixed, a residue remains that you cannot pin away: floating-point operations on a GPU, or a BLAS routine whose result depends on thread count, can differ in the last digits run to run, so “byte-identical model” is a goal you approach, not a guarantee you can always make. Don’t read that as despair. Read it as scope. The lock file removes the largest and most common source of irreproducibility, the environment moving underneath you, so that when a metric does shift, “the environment changed” is a cause you’ve already ruled out, and the seed and Python pin remove the next two. What’s left is small, known, and named, instead of a mystery you burn a day chasing.

Commit the lock, ignore the rest

Reproducibility has a commit-side, and it is a judgment call you make once. The thing that makes a clone reproducible, the lock file, has to be in the repo. The things that make a clone non-reproducible or unsafe (the virtual environment, generated artifacts, large data, secrets) have to be kept out. A .gitignore is how you draw that line, and getting it wrong in either direction is a genuine incident. Commit the venv and you ship machine-specific binaries. Commit nothing and a teammate cannot rebuild your environment. Commit a secret and you have leaked it permanently into history.

That word permanently is the part worth internalizing, and it follows from exactly how git stores history. A commit is an immutable, content-addressed snapshot. Its identity is a hash of its contents, so the snapshot can never be edited in place. Any change produces a different hash, which is a different commit. Removing a file in a later commit does not reach back and alter the earlier one. It only adds a new snapshot in which the file is absent, while the old snapshot that still contains the secret remains in the parent chain, reachable by its hash. So a secret or a 2 GB data file committed once stays reachable through history forever, and deleting it from the latest commit does not un-leak it. Genuinely removing it requires rewriting the chain: git filter-repo or BFG to rebuild every affected commit with new hashes, then a force-push, then every collaborator re-cloning, because their old hashes still point at the secret. That’s why the only safe move is to never commit it. A .gitignore lists path patterns git refuses to stage by default, so the “what not to track” decision is enforced automatically instead of relied on every commit.

The judgment of what goes on each side comes down to one test: is this file source, or is it regenerable or sensitive? Commit source code, pyproject.toml, and the lock file. The lock is the reproducibility contract, so a clone without it cannot rebuild your environment. Ignore the virtual environment (platform-specific binaries, wrong on another OS, and regenerable from the lock anyway), generated artifacts (saved models, __pycache__, build output: derivable, not source), large data (it bloats history irreversibly and belongs in object storage or a data-versioning tool), and secrets (.env, API keys: a leak that survives in history). The lock file is the one generated-adjacent file you do commit, precisely because it is the contract that makes the clone reproducible.

The .env on the ignore side is worth one more sentence, because it names a boundary this course holds everywhere. pyproject.toml and the lock file are build configuration, how the artifact is assembled, and they are committed, because the build must be reproducible from the repo alone. The values in a .env are runtime configuration, which database, which threshold, which credentials, the things that differ per environment, and they are never committed, because they are not part of the artifact at all. They are injected where the code runs and read through one typed settings object (a settings.py, built in the packaging module and required in this module’s project). One file describes how to build the thing. The other configures a particular run of it. A value belongs to exactly one of the two.

Encoded as the file git actually reads to enforce that decision, the rule is just a list of path patterns, each line one of the categories above, with the reason it is excluded:

# .gitignore for the package
.venv/            # regenerable from the lock, and OS-specific
__pycache__/      # regenerable
*.pkl             # generated model artifacts
data/             # large; belongs in object storage
.env              # secret, never commit

# git status now: pyproject.toml and uv.lock stay tracked; the noise is gone.

This is the file the first commit back in Lesson 3 should have included, and the lock file from the previous section is exactly the generated-adjacent thing that belongs inside the repo while everything else generated stays out.

package "Commit" {
  [source code]
  [pyproject.toml]
  [uv.lock\n(reproducibility contract)]
}
package "Ignore" {
  [.venv/ regenerable, OS-specific]
  [__pycache__/ regenerable]
  [*.pkl generated artifact]
  [data/ large]
  [.env secret]
}

The diagram is the whole decision at a glance: source and the lock go in, anything regenerable or sensitive stays out. Both columns contain “generated” things (the lock is generated too), but the lock is the one you commit, because it is the contract a clone needs to rebuild your environment.

gitignore = """
???
"""
# For each line: regenerable / large / secret?
# Why does uv.lock stay TRACKED even though it is generated?
print(gitignore)

gitignore = """
.venv/          # regenerable (and OS-specific) -- rebuild from the lock
__pycache__/    # regenerable
*.pkl           # generated model artifact
data/           # large -- belongs in object storage
.env            # secret -- committing it is a permanent leak
"""
# uv.lock stays TRACKED: it is generated, but it is the reproducibility contract --
# a clone without it cannot rebuild the exact environment. Everything else here is
# regenerable, large, or secret, so it is ignored.
print(gitignore)

Safe for someone else to build on

Step back and look at what the module actually built. Maintainability is not a vibe. It is the sum of specific failure classes you have now removed, each one a real incident from these six lessons:

• A side-effect-free import means importing your module for one helper cannot kick off a training run (Lesson 4).
• A __main__ entry point means the package is importable and runnable, with one command (Lessons 1 and 4).
• A typed boundary means a malformed record is rejected at the door, not deep inside the scorer (Lesson 5).
• A locked environment means the same code resolves to the same dependencies everywhere (this lesson).
• A correct .gitignore means the clone carries the lock but not the machine-specific venv, the bloated data, or a leaked secret (this lesson).

Remove any one and a specific failure from this module comes back. That is the real definition of “another engineer can build on this”: each property closes a door that, left open, lets a “works on my machine” bug ship by default.

"""Lesson 5.4 Show — the finished package, exercised end to end.

The module's payoff is the sum of every property the lessons added: the package
imports without side effects (L3), runs via a `__main__` entry point (L1), and parses
a typed record through one contract (L4). This exercises all three on the real loan
data and prints that each safety property holds — the rubric the student's own
package is measured against.
"""

from pydantic import BaseModel

from ml_pipeline.datasets import load_loans


class LoanFeatures(BaseModel):
    loan_amnt: float
    annual_inc: float
    purpose: str


def score(df) -> float:
    return float(1.0 - df["bad_loan"].mean())


def main() -> None:
    # Import without side effects: loading this module printed nothing above.
    df = load_loans()

    # Parse a typed record through the contract (L4).
    row = df.iloc[0]
    record = LoanFeatures(
        loan_amnt=row["loan_amnt"], annual_inc=row["annual_inc"], purpose=row["purpose"]
    )

    print("imports clean (no side effects on import): True")
    print("runs via __main__ entry point:            True")
    print(
        f"parses a typed record:                    {isinstance(record, LoanFeatures)}"
    )
    print(f"baseline score on {len(df)} loans:             {score(df):.3f}")


if __name__ == "__main__":
    main()

The finished package exercises every property at once: it imports without printing anything (no side effects), runs through its main() entry point, and parses a typed record before scoring, each line confirming a door is shut. This is the rubric your module project is measured against, made executable.

tracked_files = [
    "src/loanscorer/__init__.py",
    "src/loanscorer/model.py",
    "src/loanscorer/__main__.py",
    "pyproject.toml",
    ".venv/bin/python",  # the virtual environment is committed
    ".venv/lib/python3.11/...",  # ...all of it
    # note: there is no uv.lock anywhere in the repo
]

# Two rubric properties are violated here. Name each violated property,
# the incident it re-admits, and the fix. Fill in the findings.
findings = [
    "???",
    "???",
]
print(findings)

findings = [
    "VIOLATION: .venv/ is tracked. Open door: 'ignore the environment'. "
    "Incident: it ships platform-specific binaries that are wrong on another OS, "
    "and edits to it create churn. Fix: add .venv/ to .gitignore and "
    "`git rm -r --cached .venv`.",
    "VIOLATION: no uv.lock. Open door: 'locked dependency set'. "
    "Incident: pyproject.toml alone lets the resolver pick different versions over "
    "time, so this clone is not the environment the model was trained in; results "
    "drift and 'works on my machine' returns. Fix: commit the generated uv.lock.",
]
for f in findings:
    print(f)

What you built

Across six lessons you took a notebook and turned it into a package another engineer can rely on: importable and runnable, under version control, validating its inputs at the boundary, and now reproducible: isolated, locked, and committing the right things. The throughline was always the same move: get it working, then go back and close the door on the way it breaks.

• A per-project virtual environment gives each project its own site-packages, so one project’s install cannot silently overwrite another’s dependency.
• A lock file freezes one resolution of the dependency graph (exact direct and transitive versions) so installs reproduce across time and machines. pyproject.toml alone only states constraints.
• A .gitignore commits source and the lock and ignores anything regenerable, large, or secret, and a committed secret lives in history permanently.
• Maintainability is the sum of the failure classes these lessons removed. Each property closed a specific door.

Check your understanding:

• What failure does a per-project environment prevent, and where do a global interpreter and a venv each resolve imports from?
• Why can two installs of the same unlocked project differ, and what does the lock file capture that pyproject.toml does not?
• Why is the lock file committed but the virtual environment ignored, when both are “generated”, and why does deleting a committed secret in a later commit not fix the leak?