Package Mechanics

Before you can turn anything into a package, you have to know what a package actually is. And here’s the thing: almost everything that later breaks when ML code leaves one machine is a property of this structure, decided right here. This lesson builds the target shape on its own. We’ll cover what a module and an import really are, why the layout has a src/ folder, how a package gets one obvious way to run, and how an editable install makes import mypkg resolve from anywhere. The next lesson takes a real notebook and converts it into exactly this shape, so the mechanics here are the blueprint that conversion follows.

The fastest way to understand modules, packages, and imports is to build the smallest real one and run it. So that’s what we’ll do: small working pieces, each one a mechanism you’ll reach for the moment you start moving code that has to run somewhere other than where it was written.

What a package is

Three words carry this whole lesson, so let me ground them before we build anything.

A module is one .py file. That’s not an analogy, it is the file. When you import data, Python runs data.py exactly once and hands you back a single object whose attributes are the top-level names that file bound: its functions, its classes, its constants. A package is the same idea one level up: a directory Python is willing to treat as one importable unit, with the modules inside it hanging off as attributes, like mypkg.data and mypkg.model. The file that tells Python “treat this directory as a package” is __init__.py. Its body runs once, the first time the package is imported, and it’s the package’s own top-level code. And an import is just the act of loading one of these so its names become usable somewhere else.

One thing to settle early, because it decides what you put where. __init__.py runs on every entry into the package. That means import mypkg, and even import mypkg.data, which imports mypkg first. So whatever lives in it is paid on every import. Keep it near-empty, or use it only to re-export your public API (from mypkg.model import train_and_score, so callers write from mypkg import train_and_score instead of reaching into submodules). The moment you put real work there, meaning load a config file, read an environment variable, import the heavy training stack, you’ve made importing the package at all do that work, including in a test that only wanted one pure function from one submodule. A slow model.py costs you only when you import model. A slow __init__.py sits on the path of every import into the package, so it taxes the whole project and surfaces as “our imports got slow” with no obvious culprit. Definitions and cheap re-exports go in __init__.py; real work goes behind functions that run only when called. And when a package genuinely needs runtime configuration, like a data path, a threshold, a connection string, the standard home for it is not __init__.py and not os.environ reads scattered through the modules. It’s one typed settings object (a settings.py built on pydantic-settings, which the packaging module teaches in depth and this module’s project has you write), constructed by the entry point that runs, never as a side effect of the import.

Here’s the part that separates a package from a folder with files in it. Every time one module imports from another, that’s an edge in your dependency graph, and those edges are load-bearing. Split the code so features.py imports from data.py and nothing imports backward, and you can import features in a test without dragging in the training stack. Let the edges tangle, so model imports features imports model, and importing any single file pulls in everything, including the heavy training code you were trying to avoid touching. Which code goes in which file isn’t cosmetic. It’s the difference between a package you can test a piece of and one you cannot.

This is the layout almost every real ML package converges on:

myproject/
  pyproject.toml        # declares the package + its dependencies
  src/
    mypkg/
      __init__.py       # marks the folder as a package
      data.py           # loads the dataset
      features.py       # the feature transform
      model.py          # train / score
      __main__.py       # the entry point: what `python -m mypkg` runs
  tests/                # tests that import mypkg

The tree shows where the files sit; the arrows that matter are the import edges between them. This is the dependency graph that decides what you can test in isolation. The solid edges run one direction only. data is imported by features, features by model, and nothing imports backward, so importing features in a test pulls in data and stops there. The dashed red edge is the failure: let model import features while features imports model, and the cycle means touching any one file drags in the entire training stack, the exact heavy code you were trying to keep out of a feature test.

[pyproject.toml\ndeclares package + deps] as pyproj
[__init__.py\nmarks the package] as init
[__main__.py\nentry point] as main
[data.py\nload + validate] as data
[features.py\nfeature transform] as feat
[model.py\ntrain / score] as model
main --> model : runs
model --> feat : imports
feat --> data : imports
model ..> data : imports
feat -[#e05555]..> model : CYCLE drags in\nthe training stack

The forward chain (__main__ → model → features → data) is the shape you want; the red back-edge from features to model is the tangle to avoid. Keeping the arrows pointing one way is what lets you import a piece without importing the whole.

When a cycle does bite, the error it throws sends you to the wrong place, so it’s worth recognizing on sight. Let model.py and features.py import each other at the top level, and the failure is ImportError: cannot import name 'X' from partially initialized module 'model', raised while features is running, naming a missing attribute, with no mention of the word “cycle.” The instinct is to go hunting for a typo in features. The actual cause is that the two modules import each other, so when one is still half-executed the other tries to use a name it hasn’t bound yet. Partially initialized in the message is the tell. The fix isn’t to shuffle lines until it happens to work. You break the cycle, usually by moving one import off the top level into the function that needs it (so it runs after both modules finish loading), or by lifting the shared name into a third module both depend on. This is the practical reason the one-directional graph above is worth keeping: it’s what keeps you out of a traceback that blames the wrong file.

One detail here trips up almost everyone the first time, and it’s worth slowing down on: the src/ folder. Putting mypkg/ under src/ instead of at the repo root means it is not importable merely because the terminal is in the project root. It only becomes importable after you install it (the last section of this lesson). That looks like a pointless hoop. It’s actually a guardrail, and the reason is a precedence rule worth stating exactly. Python resolves import mypkg by walking the search path in order, and in the workflows that matter here (running tests, or launching with python -m from the repo root) entry zero is the current working directory. (Run a script by its path instead, python path/to/file.py, and entry zero is that script’s directory rather than the cwd; the test and -m launches are the ones where it’s the cwd.) With a flat layout, mypkg/ sits at the repo root, so the moment your terminal is in that root, entry zero shadows everything else and the import resolves to your working copy whether or not the package is installed correctly. With a src/ layout, mypkg/ is not at the root, so entry zero no longer matches, and the import resolves only through the install. That single difference is the guardrail. The flat layout lets a broken package pass its tests, because the tests import the working copy that the cwd happens to expose. The src/ layout forces the tests to exercise the shipped artifact, because that’s the only copy the import can reach.

Enough structure. Here’s the smallest real version of it: a function defined in one module, imported and called from another, printing a baseline score on the loan data.

"""The smallest real package layout that runs.

A package is not a folder of scripts; it is modules with clear responsibilities that
import each other. Here `load_and_score` lives in one place and is imported and called
from `main()` — one name defined, one name that uses it, across a module boundary. That
single crossing is the whole idea; everything else in a real package is more of it.

The "score" is the majority-class baseline on the Lending Club label (most loans are
repaid), the number every later module's real model has to beat on the right metric.

The data here is a small in-file sample sized to the real Lending Club rate (183 bad
loans in 1,000 → baseline 0.817), so this example runs anywhere — in the browser Run
button and in CI — without a CSV on disk or the `de_refs` package. The real dataset
(163,987 rows) gives the same 0.817 baseline; later lessons load it for real once the
`ml_pipeline` package exists.
"""

import pandas as pd


def load_loans() -> pd.DataFrame:
    """Return the loan data as a DataFrame — one module's whole job.

    A self-contained sample that reproduces the real majority-class rate, so the
    baseline this prints matches the full dataset without needing a file to load.
    """
    # 183 defaults in 1,000 loans — the real Lending Club default rate.
    bad_loan = [1] * 183 + [0] * 817
    return pd.DataFrame({"bad_loan": bad_loan})


def load_and_score(df: pd.DataFrame) -> float:
    """Return the majority-class baseline accuracy on the bad_loan label."""
    return float(1.0 - df["bad_loan"].mean())


def main() -> None:
    df = load_loans()
    baseline = load_and_score(df)
    print(f"loans loaded: {len(df)}")
    print(f"majority-class baseline accuracy: {baseline:.3f}")


if __name__ == "__main__":
    main()

That load_and_score lives in one place and is called across a module boundary from main(), a package in miniature. The number it prints is the majority-class baseline: the accuracy you get by always guessing the most common outcome. On the loan data, where most loans are repaid, that’s already around 82% accuracy while catching zero defaulters. Hold onto that number. It’s the floor every model you build later has to clear, and a model that scores below it has learned something actively wrong, which is the kind of result that looks fine on an accuracy report and is worthless in production.

Why is the majority-class baseline worth printing as a real number rather than assuming any trained model beats it? What would a model scoring below it tell you?

# flat_script.py — everything in one file
import pandas as pd


def load_data():
    return pd.DataFrame({"amount": [1000, 2000, 3000], "bad": [0, 1, 0]})


df = load_data()
baseline = 1 - df["bad"].mean()
print(f"baseline accuracy: {baseline:.3f}")

# Which line DEFINES the data? Which lines USE it to produce output?
# Split them in comments: data.py vs __main__.py

import pandas as pd


# data.py — loading is one module's whole job
def load_data() -> pd.DataFrame:
    return pd.DataFrame({"amount": [1000, 2000, 3000], "bad": [0, 1, 0]})


# __main__.py — what runs when the package is the program
def main() -> None:
    df = load_data()  # the import edge: the entry point USES the data module
    baseline = 1 - df["bad"].mean()
    print(f"baseline accuracy: {baseline:.3f}")


main()

One command to run it

A package needs one obvious way to run it, and “obvious” means a single command, not “open the file and run the bottom half.” That command is python -m yourpackage, and the code it runs lives in __main__.py.

The reason to prefer -m over python path/to/file.py isn’t style. It’s that the two launch your code into different worlds. python -m mypkg imports mypkg as a package first, then runs its __main__.py as part of that installed package, so every import inside resolves the same way it will in a test or in production. Run the file directly by its path and it’s an orphan: its sibling imports resolve against whatever directory you happened to be standing in, which is why the same file runs on your laptop and dies under a scheduler that launched it from somewhere else. We’ll take that failure apart properly in Lesson 4. Here the discipline is enough: one command, and the run-code in one canonical place.

This is also the moment to settle how one module in the package imports another, because there are two forms and the default matters. The first is an absolute import, naming the sibling by its full package path: inside model.py you write from mypkg.features import build_features. It reads the same to a person as to the interpreter, it doesn’t care where the file sits, and it survives the file being moved. That’s why it’s the form the reference package uses (from ml_pipeline.data import load) and the one to reach for by default. The second is a relative import, using a leading dot for “the module next to me in the same package”: from .features import build_features. Relative imports shine in deeply nested packages, where spelling the full path every time gets noisy, but the standard advice (and the books are explicit here) is to use them sparingly, only when the absolute path is genuinely cumbersome, because over-using the dots damages clarity. For a flat ML package like this one, absolute imports are the right default.

Relative imports also carry a sharp, recognizable failure worth knowing now: they break the instant you run a file as a script. A relative import only has meaning when the file is running as part of a package, which python -m mypkg guarantees and python src/mypkg/model.py does not. Run that same file by its path and Python has no idea what package it belongs to, so the dot resolves against nothing and you get ImportError: attempted relative import with no known parent package. The mistake almost everyone makes is to “fix” the error by rewriting the import until the script runs, which papers over the real issue. The fix is to stop running the file by its path and run the package with -m. When you see “attempted relative import with no known parent package,” don’t touch the import; change how you launched it.

That place is the if __name__ == "__main__": guard, and the cleanest way to understand it is to watch what __name__ actually holds in each case:

"""Lesson 1.2 Show — what `__main__` means and why the guard matters.

`__name__` is `"__main__"` only when the file is run directly; on import it is the
module's name. The `if __name__ == "__main__":` guard is what lets `python -m pkg`
run the scorer end to end while `import pkg` stays silent — the same module is both
importable and runnable, with no work firing just because someone imported it.
"""


def run_scorer() -> str:
    return "scorer ran"


print(f"__name__ at import/run time is: {__name__!r}")

if __name__ == "__main__":
    # Only this branch fires under `python 02_main_entry_point.py` or `python -m`.
    print(run_scorer())

Run alone, that file always prints '__main__', which is exactly the catch: a file run by itself can never show you its import-time name, so the asymmetry the guard exists for stays hidden. Put a second module next to it that imports it, and both values appear at once:

# caller.py — sits next to mypkg/, imports the entry point
import mypkg.__main__   # importing runs entry_point's top-level print

# What you see, in order:
#   __name__ at import/run time is: 'mypkg.__main__'   <- the IMPORT case
# ...and the guarded run_scorer() line never fires, because the guard was False.

# Now run the entry point DIRECTLY instead:
#   $ python -m mypkg
#   __name__ at import/run time is: '__main__'         <- the RUN case
#   scorer ran                                          <- guard True, work fires

The same top-level print reports two different strings depending on how the module was reached: the dotted package name when imported, the literal '__main__' when run. That’s the whole mechanism in one contrast. When you run the file directly, Python sets that module’s __name__ to the string "__main__", so the guarded block fires. When something imports the file instead, __name__ is the module’s real dotted name ("mypkg.__main__"), so the guard is false and the block stays quiet: you get the file’s functions without its job running. That gap is the entire point of the guard. A definition (def/class) only binds a name and is free to import, but a call at the top level runs the moment the module is imported. Keep your definitions at the top level and your run-calls inside the guard, and a file becomes safe to import for its parts and runnable as a program, the two things a loose script can’t be at once.

def run() -> str:
    return "scored 100 loans"


# Runs on import too — that is the bug.
print(run())

def run() -> str:
    return "scored 100 loans"


if __name__ == "__main__":
    # Fires only on a direct run / `python -m`, never when imported for run().
    print(run())

A place to install into

Before you install your package, you need somewhere to put it that belongs to this project alone: a virtual environment. It’s a private Python for the project, so installing your package and its dependencies here can’t disturb anything else on the machine.

Here’s the mechanism, because it makes the whole thing click. A Python interpreter resolves imports and installs from a single directory called site-packages. A global interpreter has exactly one, shared by everything that uses it. A virtual environment is a lightweight copy of that arrangement: a directory with its own pyvenv.cfg, its own bin/python (a shim back to a base interpreter), and, the part that matters, its own empty site-packages. You make one with uv venv. Activating it prepends the env’s bin/ to your PATH, so python now points at the env’s copy and anything uv pip install puts down lands in the env’s site-packages, where imports resolve first.

uv venv                          # create the project's own environment
source .venv/bin/activate        # activate — the env's python goes first on PATH
which python                     # .../myproject/.venv/bin/python  (inside the project)
uv pip install pandas            # lands in THIS env's site-packages, nowhere else

Nothing about that directory is precious beyond those few files, which is exactly the point: delete it and recreate it and you have a clean slate, so a per-project environment is disposable in a way the global interpreter never is. We create it before installing our own package in the next section so that the editable install writes its bookkeeping into this box, not into the system Python every other project on the machine depends on.

Activation is worth understanding mechanically, because it’s the source of a failure that looks like the environment “not working.” source .venv/bin/activate does one main thing: it prepends the env’s bin/ to the PATH of the shell you ran it in, so python and pip now resolve to the env’s copies. That scope, the shell you ran it in, is the whole footgun. Activation isn’t a property of the project on disk. It’s a mutation of one shell session’s environment variables. A new terminal tab is not activated. A cron job, a systemd unit, an IDE’s run button, or a CI step doesn’t inherit your interactive shell’s activation, so it falls back to the system python and your carefully-installed package is simply not there. That’s the same “works in my terminal, dies under the scheduler” shape that recurs through this whole curriculum, here in its environment form. Two habits defuse it. First, know that activation is a convenience, not a requirement: the env’s interpreter works by its full path whether or not you sourced anything, which is why uv run <command> and .venv/bin/python script.py execute inside the env with no activation at all. That’s exactly what you want a scheduler or a CI job to invoke, because it can’t depend on a shell state that those contexts don’t have. Second, when a job fails with ModuleNotFoundError for a package you know you installed, suspect the interpreter before the install: run which python (or have the job print sys.executable) and check whether it points inside .venv or at the system Python. Nine times out of ten the package is fine and the wrong interpreter is running it.

The reason the box has to be per-project, not a convenience, is the single site-packages slot. Each package name occupies exactly one slot in a given site-packages: install pandas there and it holds one pandas, not one per project. So if the loan scorer and some older project both install into the global interpreter, the loan scorer’s pip install pandas overwrites the version the older project was pinned to. Nothing fails at install time. The older project breaks days later on a changed pandas API, and its traceback points at its own code, which hasn’t changed, rather than at the install that moved the floor under it. A per-project site-packages gives each project its own slot, so that overwrite can’t happen. The full anatomy of that collision is Lesson 6’s job. Here you need the habit and the reason for it: make one, activate it, install into it, because the alternative is a silent overwrite that surfaces as someone else’s bug.

Without scrolling up: what does activating an environment change about where import and uv pip install look, and why make the environment before installing your own package?

Install it so imports just work

Now the payoff. Install your package once in editable mode against a pyproject.toml, and import mypkg resolves from anywhere, no path juggling, ever.

Let’s see the problem first. Stand in a different folder before installing, and Python has no idea where your package lives:

$ cd analysis/
$ python -c "import mypkg"
ModuleNotFoundError: No module named 'mypkg'

pyproject.toml is the declarative description of the package (its name, version, and dependencies) that the build backend reads to know what it’s installing:

[project]
name = "mypkg"
version = "0.1.0"
dependencies = ["pandas", "scikit-learn"]

[build-system]
requires = ["setuptools>=68"]
build-backend = "setuptools.build_meta"

That version = "0.1.0" is not decoration, and the three numbers are not arbitrary. This is semantic versioning (semver), a convention every published Python package follows, and the format is MAJOR.MINOR.PATCH. Each position tells someone reading your package a specific thing about compatibility: MAJOR (the first number) changes when you break the existing interface — code that imported and called your package the old way will no longer work. MINOR (the middle) changes when you add something new without breaking what was there — safe to upgrade into. PATCH (the last) changes for a bug fix that alters nothing about how the package is used. The number is a promise to whoever depends on you: read 2.1.4 and you know it is the same interface as 2.0.0 with features and fixes layered on, while 3.0.0 means something they relied on has changed and they need to check their code. One more thing the number communicates: a leading zero. A 0.x version like 0.1.0 is the signal for pre-1.0, unstable — the interface may change at any time, and no compatibility is promised yet. Publishing 1.0.0 is the moment you commit to the rules above. That is why a fresh package starts at 0.1.0: it is honestly saying “still finding its shape.” (You do not bump this number yet — that is the release ritual you meet in the git lesson; here the point is only to read what the number you just wrote means.)

uv pip install -e .

Worth settling here, because it confuses people coming from older Python projects: your dependencies belong in pyproject.toml’s dependencies, not in a loose requirements.txt. The two aren’t interchangeable. pyproject.toml declares what your package needs to function. It travels with the package, so anyone who installs it gets those dependencies pulled in automatically. A requirements.txt is a flat list a person runs by hand with pip install -r; it installs packages but tells the build backend nothing about your package’s own requirements, so an install of your package wouldn’t know it needs pandas. The failure when you split them is quiet: the loan scorer installs and imports fine on your machine because you happened to pip install -r requirements.txt once, then breaks for the next person (or in CI) who installed the package without running that separate file, and hits ModuleNotFoundError: No module named 'pandas' for a dependency the package never declared it needed. Put the real dependencies in pyproject.toml so they ship with the package. That’s the whole reason the file exists.

The -e is the load-bearing part, and it’s worth knowing what it does mechanically. A regular install copies your code into site-packages, so edits do nothing until you reinstall, a trap that has cost people hours of “why is my change not taking effect.” An editable install instead writes a small pointer into the env’s site-packages, historically a .pth file holding your src/ path, now often a finder shim under the modern editable-install standard, that resolves import mypkg straight back to your working tree. Two consequences fall out of that pointer: the name resolves from any directory, because resolution no longer depends on the current folder, and your edits are live, because the installed package is your source rather than a copy of it.

“Live,” though, has a precise edge that is the editable install’s own footgun: it’s live for your code, but not for the metadata pyproject.toml declares. Edit the body of a function in model.py, and the next import sees it. Add a brand-new module under your package, and the next import sees that too, because the default editable install (the kind uv pip install -e . produces) writes a .pth pointing at your whole src/ directory, so anything you add inside it is on the path automatically. What is not live is anything generated from pyproject.toml at install time: rename or add a [project.scripts] console-script entry point, change the package’s dependencies, or bump its metadata, and the editable install doesn’t pick that up until you rerun pip install -e .. The console-script wrappers are written into the env’s bin/ once, at install. Rename mycli to newcli in pyproject.toml and the old mycli command keeps working while the new name simply doesn’t exist until you reinstall. The trap is that “editable means I never reinstall” is almost true. It’s true for all your Python code, false for the project metadata. So the one time it bites, your renamed CLI command still runs the old name (or a newly added dependency is missing), and it reads as a mystery rather than “the declared shape changed and the install hasn’t caught up.” Your modules and their code are live; the entry points and dependencies declared in pyproject.toml are not, until you reinstall. (One caveat for completeness: setuptools’ strict editable mode, opt-in via --config-settings editable_mode=strict, pins an explicit module list instead of the src/ directory, and there a new module does need a reinstall, but the default mode this lesson uses picks new modules up live.)

There’s a second edge that catches people far more often, and it’s the one to internalize: “the next import sees your edit” is true, but “the running process sees your edit” is not. A short-lived python -m mypkg re-imports everything on each run, so an editable install feels instant there. A long-lived process, like a Jupyter kernel, a running FastAPI server, or a worker the scheduler keeps warm, imported your module once and keeps using that already-loaded copy no matter how many times you save the file. The editable install kept the file on disk current; it does nothing about the module already loaded in memory. So the editing loop that feels broken, the “I changed the function, the notebook still runs the old behavior” one, is almost always a stale kernel, and the fix is to restart the process (or use %autoreload in a notebook), not to reinstall the package or doubt that -e is working. This is the single most common “my changes are not taking effect” in day-to-day ML work, and it’s a process you forgot to restart, not a broken install.

Now stand in a different directory, the one that raised ModuleNotFoundError before the install, and run the same import. This time it resolves, because the pointer doesn’t depend on where you’re standing:

$ cd analysis/
$ python -c "import mypkg; print('imported from', mypkg.__name__)"
imported from mypkg

That’s the mirror of the failure at the top of the section: same import, different folder, but now it finds the package through the install instead of the launch directory. This is the step that converts “works when I run it from this one folder” into “importable like any real library,” and it’s the reason Lesson 4’s imports resolve at all. From here on, the package is something you install once and import by name.

pyproject = """
[project]
name = "???"
version = "???"
dependencies = ["???"]

[build-system]
requires = ["setuptools>=68"]
build-backend = "setuptools.build_meta"
"""
# Comment: the editable-install command + what it drops into the env
print(pyproject)

pyproject = """
[project]
name = "loanscorer"
version = "0.1.0"
dependencies = ["pandas", "scikit-learn"]

[build-system]
requires = ["setuptools>=68"]
build-backend = "setuptools.build_meta"
"""
# uv pip install -e .
# Drops a POINTER (a .pth / finder shim), not a copy, into the env's
# site-packages -- pointing back at your src/ tree. So `import loanscorer`
# resolves from any folder and your edits take effect without reinstalling.
print(pyproject)

What you built

You turned a script into a package: named modules with clear jobs and a clean dependency graph, one command to run it, a private environment, and an editable install that makes it importable from anywhere. It works, and the lessons that follow start exactly here, because each one takes this working package and breaks it on purpose to show you where “works on my machine” comes apart.

• A module is one .py file; a package is a directory of modules marked by __init__.py; an import runs a module once and binds its names where you ask for them.
• The src/ layout makes the package importable only after install, which forces your tests to exercise what you actually ship.
• python -m yourpkg runs the package through __main__.py with run-code behind if __name__ == "__main__":, so importing the package never triggers its job; only a direct run does.
• A virtual environment is a private site-packages; an editable install (uv pip install -e .) writes a pointer into it so imports resolve from any directory and your edits stay live.

Check your understanding:

• What file marks a folder as a package, and what runs the first time that package is imported?
• Why does python -m yourpkg resolve imports differently than running the file by its path?
• After uv pip install -e ., why does import yourpkg work from any directory, and what did -e write into the environment to make that true?