What This Module Covers

I once watched a model server pass every test in staging and fall over within minutes of real traffic: the first request was still loading the model into memory when the second arrived, and the second one hung waiting on the first. The endpoint had answered a single request perfectly in every demo. It had never been asked two at once. The container module already left you a service holding the port open and answering with a prediction, but it trusts whatever arrives and says nothing about how fast it responds. This module turns that running service into a real serving endpoint with a proper request contract, and then hardens it against the things real traffic does to it.

What this module is

A serving endpoint is a long-running process that loads the model once, accepts a request over HTTP, and returns a prediction. Getting one to answer a single request is almost the easy part. The hard part is everything that arrives with real traffic: a second request landing while the first is still loading the model, an input shaped wrong, a batch call where the predictions come back in the wrong order, and the tail of the latency distribution blowing out under load while the median still looks fine. Each of those is a failure the happy-path endpoint passes right up until it meets production.

This module uses FastAPI to serve the loan-default model from the earlier modules. The judgment it teaches is about the boundary between a model that scores a record in a notebook and a service other teams can call: loading, concurrency, input validation, alignment, and the failure modes that only appear under traffic.

What you will learn

The module moves through five lessons, each taking the endpoint one step closer to something that survives real use:

• Get a working /predict endpoint running. Stand up a FastAPI app that loads the model once and binds it to a route, then run it, send it a request, and read the response. You will learn what a REST request and response actually are, how a route decorator registers a handler, and why the status code is a separate channel from the body.
• Make model loading survive traffic. A model loaded inside the request path, or a blocking call inside an async handler, freezes every other request in flight. You will load the model once at startup, learn why a synchronous handler runs in a threadpool while a blocking async one holds the whole event loop, and containerize the service on the locked training image — so a routine redeploy cannot silently serve the model under library versions it was never trained on.
• Make the endpoint reject bad input. An undeclared or wrong-typed field flows past a trusting handler and surfaces as a misleading error deep in the model. You will validate the request body with pydantic so malformed input is rejected at the door with a clear 422, and use the request schema to make a leaky feature structurally impossible to send.
• Add a batch path and measure what it costs. Scoring many records in one call is faster per record, but only if the predictions come back aligned to the inputs that produced them. You will add a batch endpoint, keep the alignment exact, and measure the latency the batch path actually costs.
• Survive real traffic: tail latency, degradation, scale. The endpoint is fast until it is not. You will read p50 against p99 to see the tail the median hides, make the service fail fast instead of hanging when the model dies, and find where one box stops being enough.

The module ends with a project in the same repository, on the Phase 2 starter: a schema-validated /predict service for the Adult / Census Income model — loaded once at startup, every leakage field structurally unsendable, a batch path that stays aligned with its input, a fast typed 503 when the model dies, and a latency budget in p50/p99 terms you actually measured. The lessons serve the Lending Club and KKBox models; the project is where you prove the serving layer yourself, against a rubric that maps line by line to these lessons.

How it builds on what came before

This module runs the artifact the packaging module produced. The locked, slim image is what the service deploys, and the train/serve contract you built in the data and modeling modules is what the request schema now enforces at the boundary. The leakage you learned to audit in features returns here as a structural rule: a field the request schema does not declare cannot reach the model. Nothing here asks you to abandon what you built; it puts it behind a door other systems knock on.

What you will be able to do

By the end you will stand up a serving endpoint that loads the model once and answers requests, keeps loading and concurrency off the request path, rejects malformed and leaky input at the boundary, returns batch predictions aligned to their inputs, and degrades gracefully when load or failure arrives instead of hanging. The thread through all five lessons is the same: answering one request is the start, and surviving real traffic is the job.