Make Model Loading Survive Real Traffic

The working endpoint from the last lesson handled my test requests instantly. Then it went behind real traffic — only a small number of users, a handful of requests per second at peak — and the p99 latency was measured in seconds, not milliseconds. I had quietly written joblib.load() inside the request handler, so the model was being re-read from disk on every single call. It looked perfect at one request at a time, and it cliffed the moment concurrent traffic evicted the file from the operating system cache. Working is not the same as survives-load. This lesson closes that gap.

In the last lesson you wrote a /predict handler that loaded the Lending Club model at module level and returned a structured {"probability", "label"} response. The placement of that load — at module scope, not inside the handler — was stated as a rule without its consequences. This lesson is the hardening pass that earns the rule. It covers two failures that single-request testing cannot reveal: where the model load lives relative to the request path, and which environment the container reconstructs around it. Both are quiet at one request and catastrophic under concurrency or after a routine redeploy. The plan is to take the working endpoint, break it the two ways production will break it, then fix each one.

Why loading the model per request cliffs under load

The plausible mental model is that loading the model inside the handler is wasteful but bounded: each request pays the load cost once, and a fast disk makes that cost small enough to ignore. That model survives exactly as long as the test harness sends one request at a time. The cost is real, but the cost is not the whole story, and the part the model misses is where the entire failure lives.

Here is the anti-pattern, the version that looks correct and passes every single-request test thrown at it.

import joblib
from fastapi import FastAPI

app = FastAPI()

@app.post("/predict")
async def predict(record: dict) -> dict:
    model = joblib.load("loan_default_model.joblib")  # re-read every call
    proba = float(model.predict_proba([list(record.values())])[0][1])
    return {"probability": proba, "label": proba >= 0.5}

A handler invocation is a fresh function call with a fresh local scope, so model is constructed from nothing on every call and garbage-collected after the response is sent. Loading a serialized model means reading the file off disk and reconstructing the Python object graph — deserialization — and for a model carrying large weight arrays that reconstruction runs from milliseconds to seconds depending on model size. At one request that cost is invisible, because the operating system keeps the just-read file in its page cache and the next read comes from memory. The cliff is hidden by the cache, not absent.

The cost compounds the moment requests overlap, and it compounds two different ways at once. The first way is the disk: under concurrent traffic the page cache gets evicted by everything else competing for memory, so the next joblib.load misses the cache and hits cold disk, and the same code on the same model runs an order of magnitude slower. The second way is the scheduler, and it is the half the wasteful-but-bounded model never sees.

Load expensive resources once and share them; never put one-time setup on the per-request path

The principle is that setup whose cost does not depend on the request belongs at startup, paid once, and shared across every request. Loading the model is exactly that kind of setup: the work is identical for every caller and the result is reusable. Putting it on the per-request path multiplies a fixed cost by the request count, and concurrency turns that multiplication into a latency cliff that one-at-a-time testing structurally cannot produce.

The deeper half of the failure is how the server schedules concurrent work. The term to ground first is the event loop — the single thread inside each uvicorn worker process that runs one task at a time and only switches between tasks at an await point. There is one such loop per worker, running on one CPU core, executing handlers cooperatively. Cooperative means a task keeps the thread until it voluntarily yields, and it yields only at await.

That single-threaded design is why the handler’s declaration matters. A handler declared async def runs directly on the event loop. Any synchronous, blocking, or CPU-bound call inside it — a joblib.load, a model.predict, a disk read with no await — holds the only thread for its full duration. While it holds the thread, the loop cannot advance any other task, so every other in-flight request is frozen until the blocking call returns. This is the named failure mode: one slow synchronous call inside an async def does not slow one request, it freezes all of them. A 200 ms load inside an async def, under fifty concurrent connections, does not cost one caller 200 ms — it serializes all fifty behind that load, and the fiftieth waits ten seconds. The root cause is the violated assumption that an async def handler performs only non-blocking work between its await points; FastAPI trusts that contract and schedules accordingly.

The scrolly below traces one slow synchronous call holding the loop while the queue of waiting requests grows behind it, then the fix releasing them.

The fix is to move the load off the request path, and a measurement makes the hidden cost visible. The model loads once at module scope, the handler references the already-loaded object, and the block times a per-request reload against the cached reuse so the gap the cache was hiding becomes a number.

import pickle
import time

# Stand in for a trained model: an object whose deserialization is non-trivial.
# A real joblib.load reconstructs large numpy arrays; this simulates that cost
# with a payload that takes measurable time to unpickle.
_payload: bytes = pickle.dumps([list(range(1000)) for _ in range(2000)])


def load_model() -> list[list[int]]:
    return pickle.loads(_payload)


# Per-request placement: pay deserialization on every call.
def handler_per_request(record: dict[str, float]) -> float:
    model = load_model()  # noqa: F841 -- reconstructed from scratch every call
    return float(sum(record.values()) % 7) / 7.0


# Startup placement: load ONCE, the handler closes over the shared object.
MODEL: list[list[int]] = load_model()


def handler_startup(record: dict[str, float]) -> float:
    model = MODEL  # noqa: F841 -- reference the one already-loaded object
    return float(sum(record.values()) % 7) / 7.0


record: dict[str, float] = {"loan_amnt": 12000.0, "annual_inc": 48000.0}
n: int = 200

start = time.perf_counter()
for _ in range(n):
    handler_per_request(record)
per_request_ms = (time.perf_counter() - start) * 1000

start = time.perf_counter()
for _ in range(n):
    handler_startup(record)
startup_ms = (time.perf_counter() - start) * 1000

print(f"per-request load: {per_request_ms:7.1f} ms for {n} calls")
print(f"startup load:     {startup_ms:7.1f} ms for {n} calls")
print(f"per-request is {per_request_ms / max(startup_ms, 0.001):.0f}x slower")

The gap is exactly the deserialization the page cache was hiding at one request, now multiplied by every call. The startup version pays the reconstruction a single time when the module is imported and then does nothing but pointer reuse. Module-level model loading rides on a mechanic from the Python module: top-level code executes once, when the module is first imported, because the import system records the module in its cache and a repeat import returns the cached module rather than re-running its body. That is what makes MODEL = load_model() at module scope a one-time cost and the same line inside a handler a per-call cost.

The non-obvious cost of the fix is what it does and does not buy. Hoisting the load to startup removes the per-request deserialization, but it does not make prediction parallel. A handler declared plain def is run by FastAPI in a worker thread pool, which keeps the event loop free, so a blocking call inside a def handler does not freeze the loop the way it does inside an async def. The thread pool buys responsiveness — the loop keeps accepting and dispatching requests — but it does not buy parallel compute, because CPython’s global interpreter lock means each thread must acquire the lock before it can execute Python bytecode, so the actual predictions still run one at a time. The thread pool prevents the freeze; it does not give you more cores. The last lesson of this module returns to that ceiling.

import json


def get_threshold(record: dict[str, float]) -> float:
    # Re-reads and re-parses on every call -- the per-request anti-pattern.
    config: dict[str, float] = json.loads('{"threshold": 0.5}')
    return config["threshold"]


# TODO: hoist the config parse to module scope so it runs once.
# Then predict: under 100 concurrent calls, what happens to latency
# in the per-request version vs the hoisted version, and why?

THRESHOLD: float = 0.5  # placeholder so this starter runs


def get_threshold_fixed(record: dict[str, float]) -> float:
    return THRESHOLD


print(get_threshold({"loan_amnt": 12000.0}))
print(get_threshold_fixed({"loan_amnt": 12000.0}))

import json

# Parsed ONCE at import time; every call reads the already-built dict.
_CONFIG: dict[str, float] = json.loads('{"threshold": 0.5}')
THRESHOLD: float = _CONFIG["threshold"]


def get_threshold(record: dict[str, float]) -> float:
    return THRESHOLD


# Prediction: at one request the per-request parse is invisible because the
# file stays warm in the page cache. Under 100 concurrent calls the cache is
# pressured and evicted, so the per-request version starts hitting cold disk
# and its latency cliffs; the hoisted version is unaffected -- it parsed once
# at startup and every call is a pointer read.
print(get_threshold({"loan_amnt": 12000.0}))

Containerize the service on the locked image, do not re-derive it

The working endpoint loads the model correctly now. The next thing production breaks is the environment the container reconstructs around that model. The plausible model here is that a serving image is a clean, modern base plus the dependencies: start FROM python:3.12-slim, pip install numpy and scikit-learn and FastAPI, copy the code, done. Each piece is current, the build runs, and the image is small. That model is wrong in a way that does not surface at build time and does not surface in testing — it surfaces weeks later, on a redeploy nothing was connected to.

Here is the before state, the from-scratch serving Dockerfile that looks clean and correct.

FROM python:3.12-slim

# Re-resolves the entire dependency graph at build time, every build.
RUN pip install numpy scikit-learn joblib fastapi uvicorn

COPY app.py model.joblib /app/
WORKDIR /app
CMD ["uvicorn", "app:app", "--host", "0.0.0.0", "--port", "8000"]

The breaking condition is a build that happens later. A model’s numeric output depends on the exact versions of the libraries that produced it. The scikit-learn documentation states it directly: there are no supported ways to load a model trained with a different version of scikit-learn, and operations on such data can give different and unexpected results, or even crash the Python process — loading under an inconsistent version raises an InconsistentVersionWarning. An unpinned pip install re-resolves the dependency graph at build time, so the day a newer numpy or scikit-learn is published, the next build silently picks it up. No code changed. Only the build date did.

A serving container inherits the locked training environment; it does not re-derive it

The principle is train/serve environment parity by construction. The model was fit inside one specific resolved set of library versions, and serving it correctly means running it under that same set — not a set that happens to be current, and not a set that re-resolves every build. The packaging module produced a locked, reproducible image that pins those versions. The serving image’s job is to add only the web server and the route code on top of that locked image, not to rebuild the environment.

The term to ground is base image: the image named in the Dockerfile’s FROM line, which the new image inherits in full. Pinning the base image to a specific locked tag inherits its already-resolved dependency graph rather than resolving a new one. That is the mechanism by which inheriting the training image fixes the serving versions — the serving process imports arrays of the same library versions the model was trained against, because they are the literal same layers.

The named failure mode is the one the from-scratch image sets up. My serving image rebuilt its own environment FROM python:3.12-slim and reinstalled numpy and scikit-learn unpinned. Weeks later a routine redeploy pulled a newer numpy, and a fraction of predictions flipped across the decision threshold versus the offline model — same model file, same code, only the build date had moved. Because the model had not been touched in a month, the morning went to looking everywhere except the Dockerfile. The root cause was that the serving environment had silently diverged from the training environment, and the build was the only thing that had changed. The boundary it violated is train/serve parity, and the from-scratch image violates it on every build by re-resolving instead of inheriting.

Here is the after state. The fix is a one-line FROM change that inherits the locked image and deletes the entire reinstall block, because the dependencies already live in the inherited layers.

# Inherit the LOCKED training image — same resolved dependency graph
# the model was fit against. No re-resolution, no drift.
FROM registry.example.com/lending-club-train:locked-2026-05

# Add only what serving needs on top of the training environment.
RUN pip install --no-deps fastapi uvicorn

COPY app.py model.joblib /app/
WORKDIR /app
CMD ["uvicorn", "app:app", "--host", "0.0.0.0", "--port", "8000"]

The brevity is the proof. The inheriting image needs almost no install lines, because numpy and scikit-learn were already locked upstream and arrive in the inherited layers — the few lines on top are the visible evidence that the environment was resolved once, in the training image, and reused. The non-obvious trade-off is that this couples the serving image to the training image’s size and contents: you inherit whatever the training image carries, including build tooling unused at serve time, and slimming that is a separate decision the packaging module owns. Parity first; size second.

The parity check can be machine-verifiable instead of trusted. The reason a version mismatch is dangerous is that the same code produces a different number, so the test that proves parity is that the loaded model’s library version matches the version it was trained under.

import warnings


class InconsistentVersionWarning(UserWarning):
    """Stands in for scikit-learn's real warning of the same name."""


def load_under(trained_version: str, serving_version: str) -> str:
    # Mirrors what scikit-learn does: loading under a different version is
    # unsupported and warns, because numeric output can differ or crash.
    if trained_version != serving_version:
        warnings.warn(
            f"trained on {trained_version}, loading under {serving_version}",
            InconsistentVersionWarning,
        )
    return serving_version


trained_on: str = "1.4.2"

# From-scratch image: a redeploy re-resolved to a newer version.
with warnings.catch_warnings(record=True) as caught:
    warnings.simplefilter("always")
    load_under(trained_on, serving_version="1.5.0")
    print("from-scratch image warnings:", len(caught))

# Inherited locked image: the version is the literal same layer.
with warnings.catch_warnings(record=True) as caught:
    warnings.simplefilter("always")
    load_under(trained_on, serving_version="1.4.2")
    print("inherited image warnings:  ", len(caught))

The from-scratch path raises the version-mismatch warning the moment the resolved version drifts; the inherited path raises nothing, because it never re-resolved. A real serving image promotes that warning to a startup failure — refuse to start under a mismatched version rather than serve quietly wrong numbers — which is the difference between an incident caught at deploy and one caught on a Saturday from a scoring complaint.

The static structure underneath this is one inheritance relationship, and seeing what sits in which layer is what makes “inherit, do not re-derive” concrete.

[python:3.12-slim] as base
[locked training image] as train
[serving image] as serve

base <-- train : FROM + pinned numpy/sklearn
train <-- serve : FROM (inherits resolved graph)
note right of train : numpy, scikit-learn\nresolved ONCE here
note right of serve : adds only fastapi + uvicorn\n+ route code

The training image resolves numpy and scikit-learn exactly once, against the model. The serving image inherits that resolution wholesale and adds only the serving layer, so there is no second place for the versions to drift. A from-scratch serving image would replace the middle node with its own fresh resolution, which is precisely the divergence the diagram shows you avoiding.

# The Dockerfile is given as a list of lines so you can edit it in Python.
# Rewrite it: change the FROM to inherit the locked training image, and
# remove the line that re-resolves numpy/scikit-learn.

dockerfile: list[str] = [
    "FROM python:3.12-slim",
    "RUN pip install numpy scikit-learn fastapi uvicorn",
    "COPY app.py model.joblib /app/",
    'CMD ["uvicorn", "app:app", "--host", "0.0.0.0", "--port", "8000"]',
]

locked_image: str = "registry.example.com/lending-club-train:locked-2026-05"

# TODO: build `fixed` so the FROM inherits `locked_image` and the numpy/
# scikit-learn reinstall is gone (fastapi + uvicorn may stay, with --no-deps).
fixed: list[str] = dockerfile  # placeholder

for line in fixed:
    print(line)

dockerfile: list[str] = [
    "FROM python:3.12-slim",
    "RUN pip install numpy scikit-learn fastapi uvicorn",
    "COPY app.py model.joblib /app/",
    'CMD ["uvicorn", "app:app", "--host", "0.0.0.0", "--port", "8000"]',
]

locked_image: str = "registry.example.com/lending-club-train:locked-2026-05"

fixed: list[str] = [
    f"FROM {locked_image}",
    "RUN pip install --no-deps fastapi uvicorn",  # numpy/sklearn already inherited
    "COPY app.py model.joblib /app/",
    'CMD ["uvicorn", "app:app", "--host", "0.0.0.0", "--port", "8000"]',
]

assert not any("scikit-learn" in line for line in fixed), "still re-resolving sklearn"
assert any(locked_image in line for line in fixed), "not inheriting the locked image"
for line in fixed:
    print(line)

Summary

• A model load whose cost does not depend on the request belongs at startup, paid once and shared, because putting it on the per-request path multiplies a fixed deserialization cost by every call.
• The per-request load looks fine in single-request testing because the operating system page cache keeps the file warm; concurrency evicts the cache, the next load hits cold disk, and latency cliffs.
• Each uvicorn worker runs one event loop on one thread; a blocking or CPU-bound call inside an async def handler holds that thread and freezes every other in-flight request, not just its own.
• A plain def handler runs in a thread pool that keeps the loop responsive, but the GIL still serializes the actual prediction, so the pool buys responsiveness, not parallel compute.
• A serving image inherits the locked training image and adds only the web server and route code; a from-scratch FROM python:3.x plus fresh pip install re-resolves the dependency graph and can serve the model under a different library version than it was trained on, changing predictions or crashing.

Check your understanding:

• Where must an expensive model load live so it is not paid per request, and why did the per-request version look fine in single-request testing?
• A joblib.load inside an async def handler stalls the whole service under concurrency. Which thread does it hold, and why does that freeze unrelated requests?
• Why does building the serving image FROM the locked training image protect prediction correctness, not just build time — and what does an unpinned from-scratch image re-resolve that breaks that parity?