Survive Real Traffic: Tail Latency, Degradation, Scale

I once ran a real-time scoring service that served millions of inferences a day. The median latency was flat and reassuring for months, so I watched it and felt fine. Then one afternoon a small fraction of requests started timing out. The p50 had not moved at all, but the p99 had quietly doubled, and even a fraction of a percent at that volume was still tens of thousands of requests every day, enough to trip the caller’s timeouts and page me. The endpoint was “working” by every average I had been reporting; it was the tail that took it down.

The previous lesson made the endpoint honest about its latency: a batch path that scores many records in one request, output kept index-aligned with input, and a timer wrapped around the full validate → predict → serialize path instead of around model.predict() alone. That gave a measured p50, a real end-to-end number rather than a guess. A measured p50 is necessary and not sufficient. Under real traffic the number that pages on-call is the one no single measurement on a quiet machine ever shows: the tail. This lesson is the last hardening pass on the module’s endpoint. It closes three gaps the average hides. Why does p99 diverge from p50 under load? That is the first gap, and the answer is structural rather than noise. That divergence sets up the second gap: what happens to in-flight requests when the model server dies, and the two cheap controls that convert a hang into a survivable failure. The third gap is where a Python model service stops scaling, and which resource must be measured first. By the end the student can write and defend the latency budget the project rubric demands, in p50/p99 terms they have actually measured.

p50 vs p99: the endpoint is fast until it is not

A reasonable engineer, asked “how fast is the endpoint,” reports the mean or the median. Both summarize the typical request, and for a service that handles each request in a few milliseconds, both read as a few milliseconds. The mental model underneath is that latency is a single number with a little jitter around it, so one representative value describes the service. That model is what gets a service paged.

Start with the numbers before the explanation. Here is a sample of request times shaped the way real traffic is shaped (most requests fast, a few far slower) and the three summaries an engineer might report. Watch which summaries move and which one does not.

import statistics


def percentile(values: list[float], p: float) -> float:
    """Nearest-rank percentile: the value at or below which p% of samples fall."""
    ordered = sorted(values)
    rank = max(
        0, min(len(ordered) - 1, int(round((p / 100.0) * len(ordered) + 0.5)) - 1)
    )
    return ordered[rank]


# 1000 requests: 950 land near 8 ms, 50 land in the tail (queueing + a GC pause)
fast = [8.0 + (i % 5) * 0.4 for i in range(950)]
slow = [160.0 + (i % 7) * 12.0 for i in range(50)]
samples = fast + slow

print("mean  :", round(statistics.fmean(samples), 1), "ms")
print("p50   :", round(percentile(samples, 50), 1), "ms")
print("p95   :", round(percentile(samples, 95), 1), "ms")
print("p99   :", round(percentile(samples, 99), 1), "ms")

The p50 reports 8.8 ms because fifty slow requests out of a thousand cannot move the middle of the distribution, while the mean is dragged to 18.1 ms: the tail pulls the average up but leaves the median where it was. The p99 reports 220 ms, more than an order of magnitude above the median, because the 99th percentile is one of those slow requests by definition. A caller with a 200 ms timeout does not experience the 8.8 ms median; it experiences the request it actually got, and one in a hundred of those is in the tail. Reporting only the mean or the p50 is not a rounding choice; it is reporting a number no unlucky caller ever sees.

The correct model is that latency is a distribution, and the tail is a separate quantity from the center that must be reported on its own. The percentile notation names it: p50 is the value half of requests come in under (the median), p99 is the value 99% come in under, p999 the value 99.9% come in under. Tail latency is the high-percentile region, the slowest few percent, and it is what callers fail on, because a caller fails when its request is slow, not when the average request is fast. The SRE practice that follows is to alert on percentiles, never on the mean: a service can have a mean that never changes while five percent of its requests run twenty times slower, and mean-based alerting shows a flat line straight through the incident.

Knowing the tail is a separate quantity raises the question of why it pulls away from the center as load rises. Three forces do it, and they compound rather than merely add.

The first force is queueing. When requests arrive faster than the server drains them, the overflow waits in line, and the wait does not grow linearly with load. The order-of-magnitude shape is the M/M/1 queue (a single server with random arrivals) whose expected wait scales like $\frac{1}{1 - \rho}$, where $\rho$ is utilization, the arrival rate divided by the service rate. The shape, not the exact multiplier, is the lesson: as $\rho$ climbs toward one the denominator collapses toward zero and the wait knees up. Read the multiplier straight off $\frac{1}{1-\rho}$: at 50% utilization the expected wait is 2× the bare service time, at 80% it is 5×, and at 90% it is 10×. The exact numbers depend on the arrival and service distributions, but the knee is unavoidable in any single-server queue, which is why utilization becomes a problem well before it reaches 100%. The median request still usually finds the server free and is untouched; the unlucky request that lands behind a burst waits for everything ahead of it. So utilization can rise with the median holding flat while the tail blows out: the divergence is structural, built into the queue, not a sign of a new bug.

That structural wait is the floor; garbage collection then pulls individual requests off it. The Python runtime periodically stops execution to reclaim memory, and a request that lands during a pause eats the full pause on top of its own work while its neighbors a millisecond earlier or later are untouched. That makes GC a source of latency outliers: it shifts individual requests into the tail, it does not shift the median. A tuned collector limits the pause but does not remove it, so the pause is a permanent feature of the tail, not a bug to be fixed away.

A request can also fall into the tail without queue or pause, through a cold cache, the third force, carried straight over from Lesson 2. A process is slower right after start than in steady state, and any request that misses (an evicted OS page, a freshly started worker, the model file evicted under the per-request load Lesson 2 warned about) pays the full cold cost the warm median never sees. The request that is both behind a burst and lands during a GC pause and misses a cold cache stacks all three delays, which is how a tail doubles from causes that each look minor alone. None of the three move the median; all of them inflate p99.

One amplifier makes the tail worse the moment the endpoint is not the leaf of the call graph. When a single caller fans out to several backends and waits for all of them, the overall latency is set by the slowest response, not the average, so a request that touches five backends is at the mercy of the worst tail among the five. A p99 that is acceptable for one service composes into a much worse p99 once five of them are chained, because the probability that at least one of five independent calls lands in its own tail is far higher than the probability that any single one does. This is why a tolerable single-service tail is not a tolerable system tail, and why the documented budget has to account for the callers a service fans out to.

The scrolly below shows the distribution deforming as load rises, one step at a time, the time evolution prose cannot draw in two sentences.

This ties directly to the rubric. The latency budget the project demands is defensible only because this section (what p99 is and why it diverges) and Lesson 4 (a measured end-to-end p50) together produce a measured p50/p99 pair, not a guessed one. A budget stated as “p99 under 200 ms at 70% utilization, behind at most three fan-out hops” is a number defensible in a design review; “it is fast” is not.

def percentile(values: list[float], p: float) -> float:
    ordered = sorted(values)
    rank = max(
        0, min(len(ordered) - 1, int(round((p / 100.0) * len(ordered) + 0.5)) - 1)
    )
    return ordered[rank]


# 200 requests: 196 fast, 4 extreme outliers from a stop-the-world pause
samples: list[float] = [12.0] * 196 + [340.0, 360.0, 410.0, 520.0]

p50: float = 0.0  # replace with the computed p50
p99: float = 0.0  # replace with the computed p99
print("p50:", p50, "ms")
print("p99:", p99, "ms")
# Which one crosses the 200 ms timeout? Write the answer in a comment.

def percentile(values: list[float], p: float) -> float:
    ordered = sorted(values)
    rank = max(
        0, min(len(ordered) - 1, int(round((p / 100.0) * len(ordered) + 0.5)) - 1)
    )
    return ordered[rank]


samples: list[float] = [12.0] * 196 + [340.0, 360.0, 410.0, 520.0]

p50 = percentile(samples, 50)
p99 = percentile(samples, 99)
print("p50:", p50, "ms")
print("p99:", p99, "ms")
# p50 = 12 ms, well under the 200 ms timeout.
# p99 = ~410 ms, past it. The caller times out on its tail, not its median,
# so the p99 is the number the timeout depends on.

Graceful degradation: when the model dies, fail fast, not open-ended

The intuitive worry about a model server is that it crashes. A crash is loud, the orchestrator notices, and a new instance starts: recoverable by design. The mental model that the crash is the dangerous failure is exactly backward. The dangerous failure is the server that does not crash: it stays up, keeps accepting connections, and holds each one open against a model that can no longer answer. A crash frees the request; a hang holds it.

An endpoint can fail in several distinct ways, and conflating them is how a small outage becomes a cascade. Map the failure modes one at a time, because each one needs a different control.

First, model load fails at startup. The server process starts and the socket binds, but joblib.load() throws on a corrupt artifact, a missing file, or an out-of-memory kill mid-load. The trap here is a single /health route that returns 200 the instant the process is alive, because the orchestrator then routes traffic to an instance whose model never loaded, and every request 500s. The fix is the liveness vs readiness split. Liveness asks “is the process alive?” and a failed liveness check kills and re-creates the container. Readiness asks “can this instance serve a request right now?” (model loaded, dependencies reachable) and only ready instances receive traffic from the load balancer. A liveness probe alone is insufficient: a process can be alive and deadlocked, passing liveness while serving nothing. Readiness must reflect the model’s real state, not the process’s, or it routes traffic into a black hole.

That black hole is benign compared to the second failure mode, the cascade, which takes down more than the failing service. When the model server dies mid-request, in-flight requests get no response. Without a client-side timeout, each one hangs until the OS eventually tears down the socket, which can take minutes. The killer detail is that each hung request holds one slot in the caller’s finite connection or thread pool, so a few seconds of a dead backend drains the caller’s pool, and now the caller cannot serve its callers either. One dead leaf stalls every service upstream of it, and the stall runs backward up the call graph. A client-side timeout is what converts an unbounded hang into a bounded, retryable failure before the pool drains. Without it the caller cannot even tell whether the request succeeded, failed, or is merely delayed by congestion; the timeout is what bounds that uncertainty. This is why the timeout is not a nicety; it is the circuit breaker, the one control that stops the cascade.

Third, a slow dependency rather than a dead one. A feature store or a downstream service starts taking seconds instead of milliseconds. The endpoint must make an explicit decision: fail the request fast, or degrade to a cached or default prediction. The silent default (no decision) is to inherit the dependency’s full timeout on every request, so a slow dependency silently becomes the endpoint’s own latency, and its p99 blows out for a reason that is not in its code at all. The decision must be written down, because the absence of a decision is itself a decision, and it is the worst one.

What binds all three is one rule: an unavailable backend should produce a fast, typed failure, never an open-ended wait. The standard typed failure is HTTP 503 Service Unavailable, a fast error that keeps response time bounded instead of letting a queue of work the server cannot serve grow without limit. This is the same status-code rule from Lesson 1: a 5xx body is an error, not a prediction, and the caller must branch on the status code, not parse the body as a score. Fast failure is recoverable, since the caller can retry, fall back, or alert. A hang propagates.

The state machine below is the set of states the serving process moves through and what the readiness probe reports in each. It is static structure (the states and transitions) which a diagram shows exactly and prose can list but cannot show the transitions of.

[*] --> Starting
Starting --> Ready : model load ok
Starting --> Failed : model load throws
Ready --> Degraded : model dies / dependency down
Degraded --> Ready : recovered
Failed --> [*] : liveness kills, restart

note right of Starting : readiness = NOT READY\n(no traffic routed)
note right of Ready : readiness = 200\nrequest -> prediction
note right of Degraded : readiness = NOT READY\nrequest -> fast 503
note right of Failed : readiness = NOT READY\nliveness fails -> restart

Only the Ready state returns 200 from the readiness probe, so it is the only state the load balancer routes traffic to. Starting and Failed never receive traffic, which is what prevents the every-early-request-fails outcome. Degraded returns a fast 503 rather than a hang, which is what stops the cascade. The diagram makes one thing visible that a list of states hides: there is no transition that lets a not-ready instance receive a request, so the readiness probe is the single gate every failure mode routes through.

Here is the degradation control in code: a prediction wrapped in a deadline and a typed handler, so a dead or slow model produces a fast 503-shaped response instead of propagating an exception or hanging. Watch the caller get a definite answer fast in both the success and the failure path.

from dataclasses import dataclass


@dataclass
class Result:
    status: int
    body: dict


class ModelUnavailable(Exception):
    pass


def call_model(record: dict, dead: bool, deadline_s: float) -> float:
    """Stand-in for model.predict; raises if the model is dead or too slow."""
    if dead:
        raise ModelUnavailable("model server not responding")
    elapsed = 0.45  # simulated compute that exceeds a tight deadline
    if elapsed > deadline_s:
        raise TimeoutError("exceeded " + str(deadline_s) + "s deadline")
    return 0.18


def predict_handler(
    record: dict, dead: bool = False, deadline_s: float = 0.2
) -> Result:
    """Degrade to a fast typed 503 instead of hanging or crashing the request."""
    try:
        prob = call_model(record, dead=dead, deadline_s=deadline_s)
    except (ModelUnavailable, TimeoutError) as exc:
        return Result(
            status=503, body={"error": "model_unavailable", "detail": str(exc)}
        )
    return Result(status=200, body={"probability": prob, "label": prob >= 0.5})


record = {"annual_inc": 40000, "loan_amnt": 10000}

healthy = predict_handler(record, dead=False, deadline_s=1.0)
print("healthy ->", healthy.status, healthy.body)

dead = predict_handler(record, dead=True)
print("dead    ->", dead.status, dead.body)

slow = predict_handler(record, dead=False, deadline_s=0.2)
print("slow    ->", slow.status, slow.body)

The healthy path returns 200 with a prediction; the dead and the slow paths both return 503 with a typed error the caller can branch on, so none of them hangs, and none of them leaks a raw exception as if the server itself were broken. The caller reads status == 503 and decides what to do (retry, fall back, alert) in microseconds instead of holding a connection open for minutes. The deadline is the load-bearing line: without it the slow path would block, and one slow backend would become the whole latency budget.

class ModelUnavailable(Exception):
    pass


def call_model(record: dict, dead: bool) -> float:
    if dead:
        raise ModelUnavailable("model server not responding")
    return 0.22


def predict_handler(record: dict, dead: bool = False) -> dict:
    # Right now an exception escapes and the request dies with no typed answer.
    prob = call_model(record, dead=dead)
    return {"status": 200, "probability": prob}


print(predict_handler({"x": 1}, dead=False))
print(predict_handler({"x": 1}, dead=True))  # should NOT crash; return a typed 503

class ModelUnavailable(Exception):
    pass


def call_model(record: dict, dead: bool) -> float:
    if dead:
        raise ModelUnavailable("model server not responding")
    return 0.22


def predict_handler(record: dict, dead: bool = False) -> dict:
    try:
        prob = call_model(record, dead=dead)
    except ModelUnavailable as exc:
        return {"status": 503, "error": "model_unavailable", "detail": str(exc)}
    return {"status": 200, "probability": prob}


print(predict_handler({"x": 1}, dead=False))
print(predict_handler({"x": 1}, dead=True))

Where the endpoint stops scaling

When throughput stops climbing under load, the reflex is to reach for async. Async is the word everyone associates with high-concurrency Python servers, so the instinct is to mark every route async def and expect the numbers to rise. For a model service the numbers do not move, and the reason exposes the actual scaling ceiling.

The correct model starts with one fact about CPython: the GIL (the global interpreter lock) lets only one thread execute Python bytecode at a time within one process. A model.predict is CPU work, pure bytecode, so a single worker process cannot use more than one core for prediction no matter how many concurrent requests arrive or how many threads the framework’s pool has. This is the through-line from Lesson 2. There, a plain def handler was the safe choice because FastAPI runs it in a thread pool so a blocking call cannot stall the event loop. The thread pool buys responsiveness, keeping the loop free to accept new connections, but the GIL still serializes the actual prediction across those threads, so it does not buy parallel compute. Responsiveness and throughput are different ceilings, and the thread pool raises only the first.

Async raises neither for CPU work, and understanding why is the whole point. Async parallelizes waiting, not computing. A coroutine blocked on the network or disk hits an await, yields the event loop, and lets another coroutine run while it waits: real concurrency for I/O-bound work, because the GIL is released during I/O. A matrix multiply has nothing to yield: there is no await point inside model.predict, so an async def doing prediction holds the loop for the full compute and blocks every other request on it. Marking a CPU-bound route async does not raise throughput; on a busy loop it lowers it.

The numbers make the ceiling concrete. Below, the same per-request CPU cost is run against a target throughput, and the worker arithmetic falls straight out: required workers follow from the cost and the rate, not from intuition. Watch how a single process caps at one core’s worth of work.

import math

target_rps = 120.0  # requests per second we must sustain
per_request_cpu_s = 0.040  # measured CPU seconds per prediction (NOT wall time)
cores_available = 8

# One process executes Python bytecode on one core because of the GIL.
# Throughput of a single worker, fully busy on one core:
single_worker_rps = 1.0 / per_request_cpu_s
print("single worker ceiling:", round(single_worker_rps, 1), "req/s")

# Workers needed = total CPU-seconds of demand per second, rounded up.
workers_needed = math.ceil(target_rps * per_request_cpu_s)
print("workers needed       :", workers_needed)
print("fits on one box      :", workers_needed <= cores_available)

# Adding async changes NONE of these numbers; the work is CPU-bound under the GIL.
print("async would change   :", "nothing, there is no I/O wait to overlap")

One worker tops out at 25 req/s here because 40 ms of CPU per request is 25 requests per CPU-second, and one process gets one core. Sustaining 120 req/s needs five workers, five processes, because each OS process has its own interpreter and its own GIL, which is the only way past one core. The arithmetic is workers ≈ target_rps × per_request_cpu_s, capped by core count; it is composition, not a measured constant, so the per-request CPU cost must be the measured number from Lesson 4’s instrumentation, never a guess.

That arithmetic names the real scaling levers, and each addresses a different ceiling. Worker processes, one per core, get past the GIL to N-way parallel compute on one box. Batching (Lesson 4) amortizes per-request overhead so each prediction does less framework work. Horizontal replicas behind a load balancer add boxes once a single fully-cored box still saturates, and they only work cheaply if the process is stateless, model loaded per replica with no per-request server state, because a stateful service forces the balancer to pin each client to one replica (sticky sessions) and that pinning destroys even load distribution. None of these substitutes for another: workers fix compute, batching fixes overhead, replicas fix box-level saturation.

The non-obvious staff move is restraint. At a startup, one box is the correct answer until it actually saturates. Standing up horizontal replicas and a load balancer on day one buys operational cost (a balancer to run, replica health to track, stateless discipline to enforce) against a ceiling not yet hit. The signal that the single box is finally the bottleneck is specific and measurable: CPU pinned at one core’s worth of work while requests queue. So the first diagnostic question is never “should I add async or replicas?” but “which resource is pinned, CPU, memory, or connections?”, answered by reading a monitor, not by reflex. Scaling the wrong axis is the failure: adding async to a CPU-bound service leaves one core pinned and the rest idle while throughput stays flat, and a glance at a per-core CPU monitor would have shown the pinned core before a single route was touched.

import math  # noqa: F401 -- needed once you compute workers_needed with math.ceil

target_rps = 200.0
per_request_cpu_s = 0.030
cores_available = 16

workers_needed = 0  # compute it from the demand
print("workers needed:", workers_needed)
print("fits on one box:", workers_needed <= cores_available)
# In a comment: why does adding `async def` to the route not change workers_needed?

import math

target_rps = 200.0
per_request_cpu_s = 0.030
cores_available = 16

workers_needed = math.ceil(target_rps * per_request_cpu_s)
print("workers needed:", workers_needed)
print("fits on one box:", workers_needed <= cores_available)
# async parallelizes WAITING (I/O), not COMPUTING. model.predict is CPU-bound bytecode
# with no await point to yield on, so under the GIL one process still uses one core.
# Only separate processes (each with its own GIL) add parallel compute; async adds none.

Summary

• Latency is a distribution, not a single number. Report p50 and p99, never the mean alone, because callers fail on the tail (the request they actually got), not on the median, and a mean-based alert shows a flat line straight through an incident.
• The tail diverges from the center for structural reasons: queueing (expected wait scales like 1/(1−ρ), so it knees up near saturation), GC pauses (per-request outliers), and cold caches (misses pay the full cold cost). They compound, and fan-out amplifies them, since the slowest of N backends sets the response time.
• The dangerous failure is the hang, not the crash. Convert it with two cheap controls: a readiness probe that reflects the model’s real state (so no traffic routes to a not-ready instance) and a per-request timeout that turns an unbounded hang into a fast typed 503 before the caller’s connection pool drains.
• A Python model service’s scaling ceiling is usually the GIL: one process uses one core for prediction. Worker processes (one per core, each with its own GIL) add parallel compute; async adds none for CPU-bound work because it parallelizes waiting, not computing. Measure which resource is pinned before scaling anything.

Check your understanding:

• Without looking back: name two forces that can double p99 while p50 stays flat, and state which percentile a caller’s 200 ms timeout actually depends on.
• A model server runs out of memory and is killed but keeps accepting connections. What is the dangerous default behavior, and which two controls convert it into a survivable failure?
• An endpoint’s throughput is flat under load and one CPU core is pinned at 100% while the rest idle. What is the likely bottleneck, and why would adding async def to every route not fix it?