Detect Drift: When the Inputs Stop Looking Like Training

I once owned a scorer that ran clean for weeks while it was quietly getting worse. The health check was green the whole time, the p99 latency never moved, the error count stayed at zero, and every log line was a tidy 200. The dashboard I had built in the previous two lessons reported nothing wrong because nothing it watched had changed. What had changed was the data: a new product had shifted the applicant population, the live inputs no longer looked like the rows the model trained on, and the model was returning confident numbers for a population it had never seen. I found out from the business, the default rates climbed, months after the monitor could have told me, if only it had been watching the right thing. The machine was instrumented. The data never was.

Uptime, latency, and errors are all answers to one question: is the box working. None of them answers a different question that has no exception attached, namely whether the inputs still look like training. The previous lesson hardened what a log line cannot show: the rates and latencies across many requests, the path of one slow request through its spans. Every one of those signals inspects the request’s machinery, whether it completed, how fast, and with what status. This lesson closes the failure none of them can see: the request’s values walking away from what the model was trained on. That question raises nothing. It must be asked on a schedule, against a stored baseline, or it is never asked at all.

This is the module’s central tension landing in its first measurable form. A service can return 200 OK and be quietly wrong. The drift this lesson detects is the on-ramp to the full silent-failure case in the final lesson; the thresholds it sets are what the next lesson on alerting has to keep quiet on normal day-to-day variation.

Drift is a silent failure: the inputs left training and nothing threw

The mental model a competent engineer carries out of the last two lessons is reasonable and wrong: a healthy service is one with green health checks, a flat p99, and a zero error count. That model is correct for the box and silent on the model. A trained model fits the relationship between inputs and outcome on the data it saw, and then that relationship is frozen at deploy time. The non-obvious consequence is that the model carries no notion of “this input is unfamiliar”; it returns a confident number for any in-range vector, including vectors from a population it never saw. When the live input distribution shifts away from the training distribution, the model is extrapolating into data it has no fit for, and it does so silently: the prediction is still a float, the request still completes, the status is still 200.

Watch the wrong model break. The serving distribution of annual_inc has shifted right against the training distribution, and the only thing the prior signals can tell you is that the box is fine.

# The dashboard from the last two lessons. Every signal is green.
health_check()        # -> "OK"
p99_latency_ms()      # -> 12.0    (flat all week)
error_count_today()   # -> 0
# Meanwhile, the live inputs:
train_mean_income = 75_000
serving_mean_income_this_week = 95_000   # a new product shifted the applicants
# The model scored every one of those at 200 OK. Nothing raised. Nothing logged
# a problem. The accuracy you care about will not move for months (the loans
# have to run their term). The model is already worse, today, and invisible.

Nothing in that block throws. There is no except clause that catches “the world moved,” because moving is not an error; it is a distribution. The staff move is to stop treating “no errors, fast, healthy” as “working” for an ML system and start asking the question none of those signals answer, on a schedule, against a stored baseline. Data drift is the production input distribution shifting away from the training distribution, and it is detected the only way it can be: by comparing two distributions, the training (reference) distribution against a recent production window, on an ongoing basis. A model trained on one distribution degrades when it scores another, and nothing will raise that for you.

Quantify how far today moved from training

Detection means quantifying “how far has today’s feature distribution moved from the training baseline,” per feature. There is more than one way to measure that distance, and the reason a staff monitor uses more than one is that they catch different drifts: a measure excellent at shape change inside a range is blind to a uniform slide, and the reverse. The principle is that a distribution distance is a choice about which kind of movement you can see, so pick the measure to match the failure you are most exposed to, and run more than one when you are exposed to both.

The PSI formula is the lesson’s one real equation, and it earns its place because a shifted histogram is ambiguous without a single number to rank features by. The Population Stability Index bins both distributions on edges fixed from training (commonly the training deciles), takes each bin’s proportion in each distribution, and sums a per-bin divergence:

$$\text{PSI} = \sum_{i=1}^{B} (a_i - e_i),\ln!\frac{a_i}{e_i}$$

where $e_i$ is the proportion of the training (expected) distribution in bin $i$, $a_i$ is the proportion of the actual serving distribution in the same bin, and $B$ is the number of bins. The term $(a_i - e_i)$ is the raw mass that left or arrived in bin $i$; the $\ln(a_i / e_i)$ term weights that movement by how surprising it is relative to what training expected there. The failure mode lives inside the formula: when a bin that had training mass goes to near-zero in the actual sample, $\ln(a_i / e_i) \to -\infty$, and the whole score blows up to infinity or NaN. An empty actual bin must be floored to a small epsilon or the number is meaningless. This is not a numerical nicety; it is the difference between a usable gauge and a NaN on the dashboard the first quiet day a thin bin empties out.

Here is PSI hand-rolled the way it belongs on day one: a few lines, runs inside the existing monitor, and the epsilon floor explicit so the ln cannot diverge. Watch the quiet-week score sit near zero while the drifted-week score lands two orders of magnitude higher.

import numpy as np


def population_stability_index(
    expected: np.ndarray,
    actual: np.ndarray,
    bins: int = 10,
    eps: float = 1e-4,
) -> float:
    """PSI of `actual` against `expected`, bins fixed on `expected` deciles."""
    edges = np.quantile(expected, np.linspace(0, 1, bins + 1))
    edges[0], edges[-1] = -np.inf, np.inf  # catch tails outside training range
    e_counts, _ = np.histogram(expected, bins=edges)
    a_counts, _ = np.histogram(actual, bins=edges)
    e = e_counts / e_counts.sum()
    a = a_counts / a_counts.sum()
    # Floor empty bins so ln(a/e) cannot diverge to -inf / NaN.
    e = np.clip(e, eps, None)
    a = np.clip(a, eps, None)
    return float(np.sum((a - e) * np.log(a / e)))


rng = np.random.default_rng(0)
train = rng.normal(75_000, 22_000, 20_000).clip(10_000, 300_000)  # annual_inc
quiet_week = rng.normal(75_000, 22_000, 4_000).clip(10_000, 300_000)
drift_week = rng.normal(95_000, 24_000, 4_000).clip(10_000, 300_000)

print(f"quiet-week PSI: {population_stability_index(train, quiet_week):.4f}")
print(f"drift-week PSI: {population_stability_index(train, drift_week):.4f}")

The quiet week scores near 0.003 and the drifted week near 0.72: the same code, the same baseline, two orders of magnitude apart. The non-obvious cost of PSI is the binning itself: coarse bins smear a pure location shift. A distribution that slides uniformly can leave per-bin proportions almost unchanged and barely move PSI, because the mass that left a bin on one side arrived from the bin next to it. That is exactly the drift the next measure catches.

When the failure you fear is a whole distribution sliding rather than reshaping, the Kolmogorov–Smirnov two-sample test is the right tool, because it never bins. KS is the maximum absolute vertical gap between the two empirical cumulative distribution functions, the largest disagreement at any point about “what fraction of the data is below this value.” A uniform slide lights it up because the CDFs separate everywhere. You do not hand-roll this one; scipy.stats.ks_2samp supplies the D statistic and a p-value. Watch the quiet week return a tiny D and the drifted week return a large one.

import numpy as np
from scipy import stats

rng = np.random.default_rng(1)
ref = rng.normal(75_000, 22_000, 4_000)  # training annual_inc
quiet = rng.normal(75_000, 22_000, 4_000)  # same distribution
drift = rng.normal(95_000, 24_000, 4_000)  # slid right

d_quiet, p_quiet = stats.ks_2samp(ref, quiet)
d_drift, p_drift = stats.ks_2samp(ref, drift)
print(f"quiet  D={d_quiet:.3f}  p={p_quiet:.3g}")
print(f"drift  D={d_drift:.3f}  p={p_drift:.3g}")

The quiet week returns D≈0.012 and the drifted week D≈0.337, so the D statistic separates them cleanly. The failure mode of KS is in the p-value, not the D. A p-value collapses with sample size: at high request volume any tiny, operationally-irrelevant difference becomes statistically significant, so a monitor that fires on p < 0.05 fires constantly on real-but-harmless variation. The next block makes that concrete with two distributions that differ by a hundredth of a standard deviation.

import numpy as np
from scipy import stats

rng = np.random.default_rng(2)
a = rng.normal(0.00, 1.0, 500_000)
b = rng.normal(0.02, 1.0, 500_000)  # means differ by 0.02 sd -- nobody cares
d, p = stats.ks_2samp(a, b)
print(f"D={d:.4f}  p={p:.3g}")  # tiny D, but p is wildly 'significant'
print("significant?", p < 0.05)  # True -- and operationally meaningless

The D statistic is under one percent, an effect nobody would page on, yet the p-value is on the order of 1e-13, screaming significance. This is why a KS-based drift alert gates on the effect size (the D statistic), not on significance: at the request volumes a real scorer sees, significance is free and meaningless. The other hard boundary on KS is dimensionality: two-sample tests work better on low-dimensional data, and KS in particular is one-dimensional only. Run it per feature on low dimensions, never on the whole feature vector at once.

The third measure is the categorical-drift gauge that the frozen encoder from the data-wrangling module set up for free. PSI and KS on one-hot columns are awkward; the sharper, cheaper signal is the unknown-category rate, the fraction of requests whose category value was not in the encoder’s frozen training vocabulary. The encoder you trained maps an unseen category to all-zeros instead of crashing, which is exactly what lets those requests pass silently, and a climbing unknown rate means the live category mix is drifting from training. The block below feeds the frozen encoder a quiet batch and then a batch carrying a new term it never saw. Watch the unknown rate stay near zero on the familiar mix and climb sharply once the unseen term arrives.

def unknown_category_rate(values: list[str], vocab: set[str]) -> float:
    """Fraction of requests whose category was never seen at training time."""
    if not values:
        return 0.0
    unknown = sum(1 for v in values if v not in vocab)
    return unknown / len(values)


train_vocab = {"36_months", "60_months"}
# Week 1: every term is in vocabulary.
week1 = ["36_months", "60_months", "36_months", "36_months", "60_months"]
# Week 5: a new product introduces a term the encoder never saw.
week5 = ["36_months", "84_months", "84_months", "60_months", "84_months"]

print(f"week1 unknown rate: {unknown_category_rate(week1, train_vocab):.2f}")
print(f"week5 unknown rate: {unknown_category_rate(week5, train_vocab):.2f}")

The unknown rate climbs from 0.00 to 0.60: the new 84_months term has no column in the encoder, so those rows go in as all-zeros. The gauge has two failure modes worth naming. First, it sees only the categories outside the vocabulary: a shift in the proportions among known categories (more of one existing term, less of another) does not raise it at all and needs a separate frequency check. Second, and more subtly, because the all-zeros rows carry no categorical signal, a rising unknown rate is itself silent model degradation on exactly those requests. The gauge climbing is not a warning before the harm; it is already the harm. Those loans were scored with one feature blanked to zero.

One classification matters before leaving this section, because the PSI and KS checks above catch only one of three distinct ways a model rots, and a reader who stops here will monitor the inputs and miss the other two. Input drift (also called data drift or covariate shift) is what this section detects: the distribution of the features moves, live rows land in regions the training data barely covered, and PSI or KS on each feature surfaces it. Concept drift is the dangerous one the input monitor cannot see at all: the relationship between inputs and outcome changes while the inputs themselves look identical to training. A borrower with the same income, the same debt-to-income ratio, the same grade defaults at a different rate than they did a year ago — not because the feature distribution moved, but because a recession or a policy change altered what those features mean for repayment. Every per-feature PSI reads zero and the model is quietly wrong, because the mapping it learned no longer holds. Label drift is the third: the base rate of the outcome itself shifts (the overall default rate climbs), which moves what a calibrated probability and a sensible threshold should be even if neither inputs nor relationship changed. The reason this taxonomy is operational and not academic is that the three have different detectors: input drift is caught by feature monitors with no labels, label and concept drift are only confirmable once outcomes arrive, and concept drift specifically is the failure mode that survives a clean feature-drift dashboard — which is exactly why the next section’s prediction-drift proxy and, ultimately, a real accuracy read against late-arriving labels are not optional extras but the only instruments that see it. ML Production Systems draws the same line: data drift is when “the statistical properties of the input features change,” concept drift when “the relationship between model inputs and outputs changes over time… [which] can invalidate the mapping found during training.”

The honest staff framing to leave this section with: ML monitoring metrics are a field still in flux. There is no settled standard the way there is for HTTP status codes, in part because ML production tooling is still young, and in larger part because classical and LLM models need fundamentally different drift metrics. PSI and KS on tabular feature distributions are meaningful for a loan-default scorer and do not transfer to a large language model, where “drift” means output-distribution shift, semantic variability, or hallucination rate, none with an agreed measure, and the standard scoring curves become noisy enough that single-point metrics stop being reliable. The durable skill is not memorizing today’s tool but understanding what a drift metric is for: catching the failure where live inputs leave the training distribution with no exception thrown. Hand-rolling PSI and KS is the right call on day one: a few lines, runs inside the existing monitor, teaches what the number means. A dedicated drift library is the option you reach for when per-feature drift across many models and dashboards stops fitting in a hand-rolled check.

import numpy as np


def psi_no_floor(expected: np.ndarray, actual: np.ndarray, bins: int = 10) -> float:
    edges = np.quantile(expected, np.linspace(0, 1, bins + 1))
    edges[0], edges[-1] = -np.inf, np.inf
    e = np.histogram(expected, bins=edges)[0] / len(expected)
    a = np.histogram(actual, bins=edges)[0] / len(actual)
    # BUG: no epsilon floor -- an empty actual bin makes ln(a/e) diverge.
    return float(np.sum((a - e) * np.log(a / e)))


rng = np.random.default_rng(0)
train = rng.normal(75_000, 22_000, 20_000).clip(10_000, 300_000)
# A drifted week that leaves an upper training bin empty:
drift = rng.normal(40_000, 8_000, 2_000).clip(10_000, 300_000)
print(psi_no_floor(train, drift))  # predict this first

import numpy as np


def psi(
    expected: np.ndarray, actual: np.ndarray, bins: int = 10, eps: float = 1e-4
) -> float:
    edges = np.quantile(expected, np.linspace(0, 1, bins + 1))
    edges[0], edges[-1] = -np.inf, np.inf
    e = np.histogram(expected, bins=edges)[0] / len(expected)
    a = np.histogram(actual, bins=edges)[0] / len(actual)
    e = np.clip(e, eps, None)  # floor so ln cannot diverge
    a = np.clip(a, eps, None)
    return float(np.sum((a - e) * np.log(a / e)))


rng = np.random.default_rng(0)
train = rng.normal(75_000, 22_000, 20_000).clip(10_000, 300_000)
drift = rng.normal(40_000, 8_000, 2_000).clip(10_000, 300_000)
print(f"floored PSI: {psi(train, drift):.4f}")

A drift score is meaningless without a reference window and a threshold

The obvious next instinct, once a per-feature PSI check works, is to point it at yesterday versus today and alert on any movement. Predict what that alert volume looks like over a normal week with no real drift.

import numpy as np


def psi(
    expected: np.ndarray, actual: np.ndarray, bins: int = 10, eps: float = 1e-4
) -> float:
    edges = np.quantile(expected, np.linspace(0, 1, bins + 1))
    edges[0], edges[-1] = -np.inf, np.inf
    e = np.clip(np.histogram(expected, bins=edges)[0] / len(expected), eps, None)
    a = np.clip(np.histogram(actual, bins=edges)[0] / len(actual), eps, None)
    return float(np.sum((a - e) * np.log(a / e)))


rng = np.random.default_rng(7)
# Seven quiet days, all drawn from the SAME stable distribution.
days = [rng.normal(75_000, 22_000, 1_500) for _ in range(7)]
fired = 0
for i in range(1, 7):
    score = psi(days[i - 1], days[i])  # yesterday vs today
    print(f"day {i + 1} vs day {i}: PSI={score:.4f}")
    if score > 0:  # 'any movement' alert
        fired += 1
print(f"alerts fired on a quiet week: {fired}/6")

The check fires on every single day of a week with no drift at all. Two samples drawn from the same underlying distribution are never identical (finite-sample noise alone produces a nonzero PSI) so an “any movement” alert is pure noise. A drift score is a distance, and a distance is only interpretable once you fix two things: what it is measured from, and how far is too far. The number alone is not a signal. The number plus a reference window plus a threshold is.

The reference window is a single decision, not a menu: compare against a stable, known-good baseline, never against the moving recent window. The baseline is the training distribution, or a frozen long production window known to have performed well, and today’s detection window is what you compare against it. Comparing against “yesterday” is the trap, and the mechanism is specific. A slow gradual drift moves so little day over day that each day looks fine relative to the last, while the cumulative distance from training grows large, so the drift hides inside the baseline you keep resetting. The block below makes the trap explicit: the same drifting series is measured two ways.

import numpy as np


def psi(
    expected: np.ndarray, actual: np.ndarray, bins: int = 10, eps: float = 1e-4
) -> float:
    edges = np.quantile(expected, np.linspace(0, 1, bins + 1))
    edges[0], edges[-1] = -np.inf, np.inf
    e = np.clip(np.histogram(expected, bins=edges)[0] / len(expected), eps, None)
    a = np.clip(np.histogram(actual, bins=edges)[0] / len(actual), eps, None)
    return float(np.sum((a - e) * np.log(a / e)))


rng = np.random.default_rng(3)
train = rng.normal(75_000, 22_000, 20_000)
# A slow drift: the mean creeps up 2k per day for 14 days.
days = [rng.normal(75_000 + 2_000 * d, 22_000, 1_500) for d in range(14)]

vs_yesterday = psi(days[12], days[13])  # moving reference
vs_training = psi(train, days[13])  # fixed reference
print(f"day 14 vs day 13 (moving baseline): PSI={vs_yesterday:.4f}")
print(f"day 14 vs training (fixed baseline): PSI={vs_training:.4f}")

Day 14 against yesterday barely registers, while day 14 against training shows the real accumulated shift. The moving baseline absorbs the drift one quiet day at a time; the fixed baseline accumulates it. This is why the reference is the training distribution and not the immediately-prior window.

Tune the window and the threshold as response curves

The detection-window size and the threshold are the two tunables, and neither has a default to memorize; each is a response curve with a failure at both extremes. The principle is that a drift score is only as honest as the window it is computed over and the line it is compared against: size the window to the natural variation of the feature, and set the threshold above that natural variation but below real drift. Get either wrong and the same check is either deaf or screaming.

The specific PSI cutoffs quoted elsewhere come with a caveat: a common credit-scoring rule of thumb treats PSI above roughly 0.2 as a material shift worth investigating, with roughly 0.1 as a watch level. Treat those as starting points, not constants. No authoritative source pins them, and lifting a stranger’s 0.2 without checking the local feature’s quiet-week variation imports their false-alarm rate. The discipline is to calibrate against history: a quiet-week backtest that fires often means the window is too short or the threshold too tight; a known historical drift the monitor should have caught but stayed silent on means the window is too long or the threshold too loose. Calibrate on local data, and remember the asymmetry: abrupt shifts forgive a long window, gradual shifts punish it.

import numpy as np


def psi(
    expected: np.ndarray, actual: np.ndarray, bins: int = 10, eps: float = 1e-4
) -> float:
    edges = np.quantile(expected, np.linspace(0, 1, bins + 1))
    edges[0], edges[-1] = -np.inf, np.inf
    e = np.clip(np.histogram(expected, bins=edges)[0] / len(expected), eps, None)
    a = np.clip(np.histogram(actual, bins=edges)[0] / len(actual), eps, None)
    return float(np.sum((a - e) * np.log(a / e)))


rng = np.random.default_rng(3)
train = rng.normal(75_000, 22_000, 20_000)
days = [rng.normal(75_000 + 2_000 * d, 22_000, 1_500) for d in range(14)]

# Starter: moving reference only. Add the fixed-baseline comparison.
moving = psi(days[12], days[13])
fixed = 0.0  # replace with the comparison against `train`
print(f"moving={moving:.4f}  fixed={fixed:.4f}")

import numpy as np


def psi(
    expected: np.ndarray, actual: np.ndarray, bins: int = 10, eps: float = 1e-4
) -> float:
    edges = np.quantile(expected, np.linspace(0, 1, bins + 1))
    edges[0], edges[-1] = -np.inf, np.inf
    e = np.clip(np.histogram(expected, bins=edges)[0] / len(expected), eps, None)
    a = np.clip(np.histogram(actual, bins=edges)[0] / len(actual), eps, None)
    return float(np.sum((a - e) * np.log(a / e)))


rng = np.random.default_rng(3)
train = rng.normal(75_000, 22_000, 20_000)
days = [rng.normal(75_000 + 2_000 * d, 22_000, 1_500) for d in range(14)]

moving = psi(days[12], days[13])  # day-over-day
fixed = psi(train, days[13])  # against the frozen training baseline
print(f"moving={moving:.4f}  fixed={fixed:.4f}")

Prediction drift is the early proxy while ground truth is still in the mail

With the per-feature input-drift monitor running against a calibrated baseline, the obvious next instinct is to monitor the thing that actually matters: accuracy. For the loan-default scorer that instinct walks into a wall. The label arrives months to years after the prediction, because the loan has to run its term before it resolves as paid or charged-off, so an accuracy dashboard is always reporting on a cohort that is already old. The feedback delay, the gap between making a prediction and learning its true outcome, is structural here, not a tooling gap, and it makes accuracy a lagging signal by construction. By the time accuracy moves, the damage is months done.

The sketch below lays the timeline out so the lag is unmistakable: the score happens today, the label resolves years later. Watch why an accuracy number computed today can only describe a cohort whose fate was sealed long before any recent failure began.

# The accuracy dashboard you wish you could watch:
score_a_loan(applicant)        # today
# ...the loan runs its 36-month term...
observe_outcome(loan_id)       # the label, ~3 years later
# Accuracy today can only be computed on loans that have RESOLVED, i.e. the
# cohort from years ago. A model that started failing this morning will look
# perfectly accurate on this dashboard until this morning's loans mature.

Production accuracy requires labels, and a model that started failing today cannot move an accuracy number until today’s cohort matures. The output distribution has no such lag: the prediction exists the moment the model scores. So a shift in the distribution of predictions (the predicted-default rate jumps) is observable immediately, while the labels are still in flight. Prediction drift is the model’s output distribution moving over time, and it is a proxy for input drift: holding the model’s weights fixed, the only things that can move the output distribution are the input distribution moving or a feature pipeline breaking, both worth investigating now, neither requiring a label.

The hard boundary, and the part a careless reader gets wrong: prediction drift signals that something changed, not that the model is wrong. A genuine shift in the world can justify a new output distribution. It buys time before labels arrive; it does not substitute for them. The block below tracks the positive-prediction rate over the calibrated baseline window (no new formula, the rate is a mean) and shows it jumping the instant a feature pipeline breaks.

import numpy as np


def positive_rate(scores: np.ndarray, threshold: float = 0.5) -> float:
    """Fraction of predictions flagged positive -- a mean over an indicator."""
    return float(np.mean(scores >= threshold))


rng = np.random.default_rng(11)
# Baseline window: the model flags a minority of applicants as likely default.
baseline = rng.beta(2, 5, 5_000)
# Today: an upstream join silently dropped a strong feature -> scores climb
# toward the middle, and the flag rate jumps. No exception was raised.
today = rng.beta(4, 5, 5_000)

print(f"baseline positive rate: {positive_rate(baseline):.3f}")
print(f"today's positive rate:  {positive_rate(today):.3f}")
print(f"ratio: {positive_rate(today) / positive_rate(baseline):.1f}x")

The positive-prediction rate jumps several-fold the day the pipeline breaks, with no exception and no label required. The diagnostic when prediction drift fires has exactly two branches worth chasing first, and they are distinguished by the input monitor from the first section. Either the inputs drifted (run the per-feature PSI check, and a feature’s PSI corroborates the output shift) or a feature pipeline broke upstream, where a transform started emitting a constant or a join started dropping rows. The second case produces an output shift with no corresponding natural input drift, and that mismatch is itself the tell. The failure this guards against is the worst shape in the module: a serving pipeline silently feeds a broken feature on day one, accuracy looks fine for months because no loan has resolved, and the prediction-drift monitor catches the collapsed flag rate months and many bad decisions before the accuracy monitor ever could.

def diagnose(
    output_rate_ratio: float, input_psi: float, psi_threshold: float = 0.2
) -> str:
    """Output shifted. Decide if an input feature corroborates it."""
    # An output shift WITH corroborating input drift -> the world moved.
    # An output shift WITHOUT it -> the pipeline broke.
    return "unknown"  # replace with the branching logic


print(diagnose(output_rate_ratio=3.0, input_psi=0.45))  # expect: input drift
print(diagnose(output_rate_ratio=3.0, input_psi=0.01))  # expect: pipeline break

def diagnose(
    output_rate_ratio: float, input_psi: float, psi_threshold: float = 0.2
) -> str:
    """Output shifted. Decide if an input feature corroborates it."""
    if input_psi >= psi_threshold:
        return "input drift"  # the world moved; the input monitor agrees
    return "pipeline break"  # output moved with no input drift -> upstream


print(diagnose(output_rate_ratio=3.0, input_psi=0.45))  # input drift
print(diagnose(output_rate_ratio=3.0, input_psi=0.01))  # pipeline break

Summary

• A model is a frozen snapshot of a relationship, and it returns a confident number for any in-range vector, so an input-distribution shift degrades it silently, with no exception, no latency change, and a 200 status. Every machine signal from the prior lessons can be green while the model decays.
• Drift is detected by comparing two distributions: a stable reference (the training distribution) against a recent detection window. PSI bins on training deciles and sums $(a_i - e_i)\ln(a_i/e_i)$ with empty bins floored to an epsilon; KS is the max CDF gap and must be gated on the D statistic, not the p-value, because significance is free at high volume. The unknown-category rate is a free categorical gauge, and a climbing rate is already harm, not a warning.
• A drift score is a distance, and a distance needs a fixed reference and a calibrated threshold to be a signal. Compare against training, never against the moving “yesterday,” or a slow gradual drift hides in the baseline you keep resetting. Window size and threshold are response curves: too short or too tight screams, too long or too loose averages real drift away.
• Prediction drift is the early proxy watched while ground truth is still in the mail: the output distribution is available the instant the model scores, while the label lags months to years. It signals change, not wrongness, and its diagnostic splits on whether the input monitor corroborates: corroborated means the world moved, uncorroborated means a pipeline broke.

Check your understanding:

• Without looking back: every signal from the last two lessons is green and the model is getting worse. What kind of failure is that, and why can none of those signals see it?
• A KS test fires with p < 1e-8 on the highest-traffic feature every single day. Is that drift? What number should the alert actually gate on, and why?
• Why does comparing today’s distribution against yesterday’s miss a slow gradual drift that comparing against training would catch?
• The predicted-default rate tripled overnight but no input feature’s PSI moved. Which of the two branches is that, and what is the root cause it points to?