Get a Working /predict Endpoint Running

I trained a default scorer that everyone agreed was an upgrade, committed the notebook, and called the work done. The model lived as a fitted object inside that one Python process, and the moment the process exited, nothing else on the system could reach it. The checkout service that needed a score, the nightly batch job, the dashboard the risk team wanted: none of them could import a notebook on my laptop. I had built a model with no door into it. The accuracy was real and completely inert, because a prediction nobody outside the process can ask for is not a prediction anyone can use.

The fastest way to understand what a model service is to build the smallest one that works and call it. This lesson explains what an API, a request, a response, a client, and a server are, shows the shape almost every serving app has (request, handler, model, response), and then builds that shape into a running /predict endpoint that returns a real prediction over curl. By the end the thing runs. The next four lessons make it survive contact with real callers: the malformed input it accepts, the leakage feature it lets through, the batch job that hammers it, and the tail latency that pages on-call. Everything in this module hardens the endpoint built here, so this lesson’s only job is to get a real one standing.

What an API is, and what a serving app looks like

A competent engineer’s first mental model is that “deploying the model” means saving the fitted object and making sure the file is somewhere the rest of the system can find it. Save the joblib artifact to shared storage, point the other services at the path, done. That model is wrong in a way that does not announce itself: a serialized file is not a running model. To get a prediction out of it, every consumer would have to load the file into its own process, reconstruct the Python object graph, and call predict itself. That means every consumer needs the same library versions, the same memory budget, and a copy of code it should not own. The file is reachable; the prediction is not.

A trained model is a Python object graph sitting in one process’s heap: arrays of learned weights reachable only through references that exist inside that interpreter. Nothing outside the process can touch it. Another service, even one running on the same machine, has no pointer into that memory. An API (application programming interface) closes that gap by putting a process that stays running in front of the model and giving it a network address, so other code asks for a prediction over the network instead of reconstructing the model itself. Here are the words the rest of the module depends on, each grounded once:

• A client is whatever calls the service: a checkout service, a nightly job, a dashboard. It does not hold the model; it asks for a prediction.
• A server is a process that stays running and answers callers. It holds the model in its heap and serves it for the life of the process.
• A request is what the client sends: an HTTP method (here, POST), a path (here, /predict), and a body of feature values as JSON.
• A response is what comes back: a status code that says whether it worked, and a body holding the prediction, as JSON.
• An API is the documented contract of which requests the server accepts and what it returns.

HTTP is the contract underneath all of this. The caller opens a TCP connection, sends a request as bytes (a method, a path, headers, and a JSON body), and the server’s framework parses those bytes back into Python objects, runs a function, and writes a response (a status line, headers, a JSON body) back over the same connection. The model never leaves the server’s heap; only its answer crosses the wire, serialized to text on one side and reconstructed on the other. That serialization sits on the critical path of every single request, which is why Lesson 4 spends a whole section measuring it. It is also why the shape of a serving app is fixed regardless of which model is inside it.

Every serving app in this module follows the same four-box shape, and seeing the destination before the first line of code is what gives each piece somewhere to land:

title Serving app: the model lives inside the server process

actor "client\n(checkout / job / dashboard)" as client

rectangle "server process (stays running)" {
  component "ASGI server\n(uvicorn)" as uvicorn
  component "FastAPI app" as app
  component "handler\n(route function)" as handler
  component "model\n(loaded once, in the heap)" as model
}

client --> uvicorn : HTTP POST /predict (JSON body)
uvicorn --> app : parsed request
app --> handler : match (POST, /predict), call it
handler --> model : feature values
model --> handler : probability / label
handler --> uvicorn : return dict
uvicorn --> client : 200 + JSON prediction

Read it left to right and then back. The request crosses the network as text and lands on the ASGI server (uvicorn), the long-lived process holding the socket. uvicorn hands it to the FastAPI app, which routes it to the handler, an ordinary Python function the framework calls once per request. The handler calls the model, which was loaded once at startup and is shared across every request, and the answer travels back out as a status code plus a JSON body. The handler is the one box an engineer writes. The framework owns everything around it, which is exactly why the same shape appears in every serving app no matter what the model does.

Before a web server is involved at all, the core of that shape is plain Python: load the model once, write a function that turns a record into a prediction, call it. The block below builds the smallest real version. It loads a model (a tiny logistic-regression default scorer standing in for the Lending Club model so this runs anywhere), defines a predict function that validates a record and returns a structured result, and calls it on one applicant. Watch the prediction come out as a {"probability", "label"} object rather than a bare number: that structure is the response shape the rest of the module builds on.

import numpy as np
from sklearn.linear_model import LogisticRegression

# Stand in for the Lending Club default scorer trained in M4.
# Two application-time features: annual income and requested loan amount.
_X = np.array(
    [[40000.0, 12000.0], [120000.0, 8000.0], [25000.0, 20000.0], [90000.0, 5000.0]]
)
_y = np.array([1, 0, 1, 0])  # 1 = defaulted
model = LogisticRegression().fit(_X, _y)

THRESHOLD = 0.5


def predict(record: dict[str, float]) -> dict[str, float | bool]:
    """Turn one validated record into a structured prediction."""
    features = np.array([[record["annual_inc"], record["loan_amnt"]]])
    probability = float(model.predict_proba(features)[0, 1])
    return {"probability": probability, "label": probability >= THRESHOLD}


result = predict({"annual_inc": 30000.0, "loan_amnt": 18000.0})
print(result)

That function did the entire job a serving app does, minus the network: it took a record, ran the shared model, and returned a status-bearing answer the caller can act on. Self-explanation prompt: the model is loaded outside predict, at module level, and the function only references it. What would change about the cost of each call if the LogisticRegression().fit(...) line lived inside predict instead? Hold that question; it is the entire subject of Lesson 2. For now the point is that the four-box shape is real before any HTTP exists, and the next section gives the network the door it is missing.

# A checkout service needs to score a loan application before approving it.
# Fill in each box for THIS scenario (replace the ??? strings):

client: str = "???"  # who makes the call?
request_body: str = "???"  # what feature values are in the POST body?
model_lives: str = "???"  # which process holds the fitted model?
response: str = "???"  # what does the caller get back?

mapping: dict[str, str] = {
    "client": client,
    "request_body": request_body,
    "model_lives": model_lives,
    "response": response,
}
for box, value in mapping.items():
    print(box + ": " + value)

client: str = "the checkout service deciding whether to approve the loan"
request_body: str = "the applicant's annual_inc and loan_amnt as JSON"
model_lives: str = "the long-running FastAPI server process, loaded once at startup"
response: str = "an HTTP status code plus a JSON {probability, label} prediction"

mapping: dict[str, str] = {
    "client": client,
    "request_body": request_body,
    "model_lives": model_lives,
    "response": response,
}
for box, value in mapping.items():
    print(box + ": " + value)

This shape runs as plain Python, but a function in a file is still unreachable by another process. The next section gives it the network address: it binds predict to a route so an HTTP request can reach it, and it puts the model load where it belongs so the cost is paid once.

Load the model once, then bind it to a route

The reasonable next assumption is that “exposing the function over HTTP” means writing the networking by hand: open a socket, read the request bytes, parse the HTTP frame, decode JSON, route on the path, encode the response. That is a real amount of fragile, security-sensitive code, and writing it per endpoint is how serving apps grow subtle bugs in header parsing and content negotiation that have nothing to do with the model. A web framework exists precisely so that plumbing stays unwritten: FastAPI owns the socket-to-Python translation and hands back a function call. The handler is the one piece an engineer writes; the framework writes everything around it. That is the WHY before the syntax. The framework is not convenience, it is the difference between owning the HTTP spec and owning one function.

A route is the binding of an HTTP method and path to a function. In FastAPI a route is created with a path operation decorator, @app.post("/predict"), placed above the function. The non-obvious part is what the decorator does not do: it does not wrap or rewrite the function. At import time the decorator runs once and registers the function. It adds an entry to the app’s routing table mapping the pair (POST, /predict) to the function object, then leaves the function exactly as it was, a plain callable still invokable directly. When a request arrives, uvicorn reads the bytes, hands an ASGI event to FastAPI, FastAPI matches method and path against that routing table, and calls the function. The decorator is a registration, not a transformation.

That the function stays an ordinary callable is the reveal in the code below, and it is worth watching for. The app loads the model at module level, binds predict_endpoint to POST /predict and a /health route to GET /health, then the example calls predict_endpoint directly, with no server running, and gets the same prediction. If the decorator had transformed the function into some HTTP-only object, that direct call would fail. It does not, because the route is plain Python the framework merely wraps. Here is the file deployed to production, shown complete:

# app.py — the serving app. Shown complete; runs under: uvicorn app:app
import joblib
from fastapi import FastAPI
from pydantic import BaseModel

# Module-level: runs ONCE when the module is first imported, as the app starts.
model = joblib.load("default_scorer.joblib")
MODEL_VERSION = "2024.06.1"
THRESHOLD = 0.5

app = FastAPI()


class LoanRecord(BaseModel):
    annual_inc: float
    loan_amnt: float


@app.post("/predict")
def predict_endpoint(record: LoanRecord) -> dict[str, float | bool]:
    features = [[record.annual_inc, record.loan_amnt]]
    probability = float(model.predict_proba(features)[0, 1])
    return {"probability": probability, "label": probability >= THRESHOLD}


@app.get("/health")
def health() -> dict[str, str]:
    return {"status": "ok"}

That file needs a real model artifact and a live server. The runnable version below proves the two claims that matter, that the decorator leaves the function callable and the model loads exactly once, without the disk or the socket. It uses a stand-in decorator that registers into a dict (the routing table) and returns the function untouched, plus a load counter so the load fires a single time no matter how many requests arrive.

import numpy as np
from sklearn.linear_model import LogisticRegression

_load_calls = {"count": 0}


def load_model() -> LogisticRegression:
    _load_calls["count"] += 1  # count every time the model is loaded
    X = np.array([[40000.0, 12000.0], [120000.0, 8000.0], [25000.0, 20000.0]])
    y = np.array([1, 0, 1])
    return LogisticRegression().fit(X, y)


# Module-level load: runs ONCE as the module is imported.
model = load_model()
THRESHOLD = 0.5

# A stand-in routing table: the decorator REGISTERS, it does not transform.
routes: dict[tuple[str, str], object] = {}


def post(path: str):
    def register(fn):
        routes[("POST", path)] = fn
        return fn  # the function comes back UNCHANGED

    return register


@post("/predict")
def predict_endpoint(record: dict[str, float]) -> dict[str, float | bool]:
    features = np.array([[record["annual_inc"], record["loan_amnt"]]])
    probability = float(model.predict_proba(features)[0, 1])
    return {"probability": probability, "label": probability >= THRESHOLD}


# Simulate three requests hitting the route via the table.
handler = routes[("POST", "/predict")]
for inc in (30000.0, 95000.0, 22000.0):
    print(handler({"annual_inc": inc, "loan_amnt": 10000.0}))

# And call the decorated function DIRECTLY -- same function, no server.
print("direct call:", predict_endpoint({"annual_inc": 30000.0, "loan_amnt": 10000.0}))
print("times the model was loaded:", _load_calls["count"])

After three routed requests and a direct call, the load counter prints 1: the model was loaded once, at module scope, and every invocation shared that one object. The direct call returns the same prediction as the routed calls, which is the proof that the decorator registered the function without changing it. Self-explanation prompt: load_model increments the counter every time it runs, yet the count is 1 after four calls to the handler. What property of module-level code makes that true, and what would the count be if the model = load_model() line moved inside predict_endpoint?

The answer is the run-once-on-import mechanic from the Python module: top-level code in a module executes a single time, when the module is first imported, so model = load_model() at module scope runs once as the app starts and binds one shared object every handler closes over. Loading inside the handler would reconstruct the model on every request, so the load counter would climb with each call. That placement is stated here as the rule; Lesson 2 returns to it as a failure and shows exactly how badly per-request loading cliffs under concurrent traffic.

One more thing decides how the handler behaves under load, and it is set by how the handler is declared. A handler declared plain def is run by FastAPI in an external thread pool, so a slow or blocking call inside it does not stall the server. A handler declared async def runs directly on the single event loop, cooperatively, and must never block: if it does, it freezes every other in-flight request, not just its own. For a synchronous, CPU-bound model call, plain def is the safe default, and Lesson 5 returns to why the GIL still bounds even the thread-pool version. For now: declare the handler def.

from typing import Callable

routes: dict[tuple[str, str], object] = {}

MODEL_VERSION = "2024.06.1"


def get(path: str) -> Callable:
    def register(fn: Callable) -> Callable:
        routes[("GET", path)] = fn
        return fn

    return register


@get("/")
def root() -> dict[str, str]:
    return {"service": "default-scorer"}


# Add a GET /version route here that returns {"model_version": MODEL_VERSION}.
def version() -> dict[str, str]:
    return {
        "model_version": "???"
    }  # placeholder -- wire it to the constant and register it


print(routes[("GET", "/")]())
# print(routes[("GET", "/version")]())  # uncomment once your route is registered

from typing import Callable

routes: dict[tuple[str, str], object] = {}

MODEL_VERSION = "2024.06.1"


def get(path: str) -> Callable:
    def register(fn: Callable) -> Callable:
        routes[("GET", path)] = fn
        return fn

    return register


@get("/")
def root() -> dict[str, str]:
    return {"service": "default-scorer"}


@get("/version")
def version() -> dict[str, str]:
    return {"model_version": MODEL_VERSION}


print(routes[("GET", "/")]())
print(routes[("GET", "/version")]())

The app now loads the model once and binds it to a route, but it is still inert in a file: nothing is listening on a socket, and no client has called it. The next section launches the server, sends a real request across the process boundary, and reads the response the way a correct caller must.

Run it, curl it, read the response

One last assumption needs dismantling: that a 200-or-not check is paranoia, that a well-formed request lets the caller read response.json()["probability"] and move on. That works in the demo and breaks the first time the input is wrong. An HTTP response carries two separate channels: a status code and a body. On a successful call the body is the prediction; on a failed call the body is an error object with no probability field at all. Reading the body before the status is the classic serving bug, and here is what it looks like when it bites:

# The caller reads the body first, trusting it is a prediction.
resp = httpx.post("http://localhost:8000/predict", json={"annual_inc": "forty thousand"})
score = resp.json()["probability"]   # KeyError: 'probability'
#                    ^^^^^^^^^^^^^
# resp.status_code is 422 — the body is {"detail": [...validation error...]},
# there is no "probability" key. The KeyError masks the real failure,
# which was that the request was rejected at the door.

That KeyError is the dangerous part: it raises in the caller, three layers from the real cause, and reads like the caller’s parsing code is broken when the truth is the server rejected the request. A status code exists so the response can say whether it worked separately from what the answer is. The convention is fixed: 2xx means success and the body is the answer; 4xx means the caller’s request was wrong and the body describes what; 5xx means the server failed and the body is an error, not a prediction. The first thing a correct caller reads is the status, and the status gates whether reading the body for a prediction even makes sense.

How does a request physically become a response? The scrolly below makes that two-channel read visible. A serving app is not finished when the function returns the right value in a notebook; it is finished when a separate client gets a prediction back across the process and network boundaries. Those boundaries are exactly what a beginner cannot see, so the steps walk one request physically crossing each one, in order, from the client serializing the record to the client reading the status before the body.

Launching the server is one command. uvicorn imports the app.py module, finds the app object, and holds the socket for the life of the process:

uvicorn app:app --reload

app:app means “the object named app in the module app”; --reload restarts the process when the file changes, for development only. With the server running, a client sends a real POST and reads the two channels in the correct order. The block below simulates the full round trip in one process so it runs here. A roundtrip function stands in for the network, returning a status code and a body exactly as the wire would. Watch the caller read the status first on both a good and a bad request, and watch the bad request’s body have no probability key at all.

import numpy as np
from sklearn.linear_model import LogisticRegression

_X = np.array([[40000.0, 12000.0], [120000.0, 8000.0], [25000.0, 20000.0]])
model = LogisticRegression().fit(_X, np.array([1, 0, 1]))
THRESHOLD = 0.5


def roundtrip(body: dict) -> tuple[int, dict]:
    """Stand in for the server: validate, predict, return (status_code, body)."""
    try:
        inc = float(body["annual_inc"])
        amt = float(body["loan_amnt"])
    except (KeyError, ValueError):
        # Validation fails at the boundary: 422, body is an ERROR, not a prediction.
        return 422, {"detail": "annual_inc and loan_amnt must be numbers"}
    probability = float(model.predict_proba(np.array([[inc, amt]]))[0, 1])
    return 200, {"probability": probability, "label": probability >= THRESHOLD}


def call_and_read(body: dict) -> None:
    status, payload = roundtrip(body)
    print("status:", status)  # CHANNEL 1: whether it worked -- read FIRST
    if status == 200:
        print("  prediction:", payload)  # CHANNEL 2: the answer
    else:
        print("  error (no prediction):", payload)  # body is an error object


call_and_read({"annual_inc": 30000.0, "loan_amnt": 18000.0})
call_and_read({"annual_inc": "forty thousand", "loan_amnt": 18000.0})

A good request returns 200 and a {"probability", "label"} body, the full request-to-response round trip the previous sections built, now crossing a client/server boundary. The bad request returns 422 and a body that is an error with no probability key. Because call_and_read checks the status first, the bad request is handled cleanly instead of raising a KeyError deep in the caller. That status-gates-body discipline is what Lesson 5 leans on when a dead model must return 503 and the caller has to branch on the status rather than assume the body is an answer.

def call(body: dict) -> tuple[int, dict]:
    # Stand-in server. Rejects a negative loan_amnt with 422.
    if body.get("loan_amnt", 0) < 0:
        return 422, {"detail": "loan_amnt must be non-negative"}
    # Toy scorer: bigger loan relative to income -> higher default probability.
    prob = min(0.95, body["loan_amnt"] / (body["annual_inc"] + 1.0))
    return 200, {"probability": round(prob, 3), "label": prob >= 0.5}


# BUG: this reads the body before checking the status -- crashes on a 422.
status, payload = call({"annual_inc": 40000.0, "loan_amnt": 5000.0})
score = payload["probability"]  # rewrite to check status FIRST, only read on 200
print("score:", score)

def call(body: dict) -> tuple[int, dict]:
    if body.get("loan_amnt", 0) < 0:
        return 422, {"detail": "loan_amnt must be non-negative"}
    prob = min(0.95, body["loan_amnt"] / (body["annual_inc"] + 1.0))
    return 200, {"probability": round(prob, 3), "label": prob >= 0.5}


def read(body: dict) -> None:
    status, payload = call(body)
    if status == 200:
        print("status 200 -> label:", payload["label"], "prob:", payload["probability"])
    else:
        print("status", status, "-> error:", payload["detail"])


read({"annual_inc": 40000.0, "loan_amnt": 5000.0})  # small loan
read({"annual_inc": 40000.0, "loan_amnt": 30000.0})  # large loan
read({"annual_inc": 40000.0, "loan_amnt": -1.0})  # rejected at the boundary

The endpoint now exists for real: it loads the model once, binds it to a route, runs under uvicorn, and answers a separate client with a status-bearing prediction. That is the entire happy path, and it is also the entire surface the next four lessons attack, because every part of the endpoint built here trusts the caller, trusts the load timing, and trusts that nothing fails. Lesson 2 starts the hardening pass with the part that breaks first under real traffic: where and when the model is loaded.

Summary

• A trained model is a Python object living in one process’s heap; an API is the boundary that puts a long-running server process in front of it so a client can ask for a prediction over HTTP. Only the answer crosses the wire; the model never leaves the server.
• Every serving app has the same four-box shape: request, handler, model, response. The handler is the one piece you write; the framework owns the socket, parsing, routing, and serialization around it.
• The model is loaded once at module level, not inside the handler, because top-level code runs a single time on import and binds one shared object every request closes over. Loading per request reconstructs the model every call: the failure Lesson 2 dissects.
• A @app.post("/predict") decorator registers the function in the routing table and leaves it a plain callable; it does not transform the function. Declare a CPU-bound handler plain def so the framework runs it in a thread pool instead of stalling the event loop.
• An HTTP response has two channels: a status code (2xx success, 4xx client error, 5xx server error) and a body. The caller reads the status first, because on a non-2xx the body is an error object, not a prediction — reading the body first turns a clean rejection into a KeyError.

Check your understanding:

• Without looking back: what is the four-step shape every request flows through, and which of those steps is the only code you write?
• A model artifact is saved to shared storage and every consumer loads the file itself. What does that buy, and what does it fail to provide that a running endpoint does?
• When does module-level code run, and why does that make model = joblib.load(...) at module scope the right placement instead of inside the handler?
• A caller does response.json()["probability"] and gets a KeyError. What is the most likely real cause, and what should the caller have checked first?