Service Up ≠ Model Right: Catching the Silent Failure

Every dashboard was green. The health check was passing, p99 was flat, the error rate was zero, and no drift alert had fired. The service was, by every signal I had built, perfectly healthy, and it was returning garbage to every request. A feature transform had silently changed on the serving path and the model was scoring on nonsense, but nonsense produces a confident number and a 200, not an exception. I found out from a colleague who eyeballed a batch of scores and said “these all look the same.” That was the day I understood that “the service is up” and “the model is right” are two different questions, and almost everything I had built only answered the first.

The four lessons before this one built monitoring that runs. In get-monitoring-working you put a structured log line on every request, keyed by a correlation id, and split the health check into liveness and readiness, the first crack of up is not the same as working. In make-it-measurable you turned raw logs into p99 and an error rate. In detect-drift you watched the input distribution for shift. In alert-on-the-right-signal you turned those signals into alerts that page only when a human should act. By every machine measure the monitoring is now complete: uptime, latency, error rate, drift, and a quiet pager.

And it is still lying. Every signal in that list answers one question, whether the box is alive and responding, and none of them answers the question a model exists to answer: are the predictions any good. At three users a wrong-but-200 prediction is a confusing demo. At the volume the M4-trained, M6-served default scorer runs in production it is thousands of corrupted loan decisions an hour reaching customers, with a green dashboard overhead. This lesson closes that gap. It covers why infrastructure health and model correctness are independent axes, the label-free proxies that catch correctness the instant a prediction is made, and how to reconstruct one offending prediction after the fact from its correlation id alone.

A 200 OK is a claim about the request, not the answer

The mental model a competent engineer brings to a green dashboard is that the signals form one scale: uptime high, latency low, errors zero, no drift alert, therefore the system is healthy and the predictions are good. The signals are read as a single thermometer for system goodness, and the model is assumed to ride along on top of the infrastructure it runs on.

That model is wrong in a specific and dangerous way. Infrastructure health and model correctness are not two ends of one scale; they are two independent axes. An ML serving path has a defining property: its inputs are almost always type-valid even when they are semantically wrong, so the failures that hurt you do not raise. A model is a pure function from a numeric vector to a number; it has no notion of “this input is nonsense,” so it returns a confident answer for any in-range vector handed to it. Uptime, latency, and error monitoring measure everything about a request except its value. A service can sit at the top of the infra axis and at zero on the correctness axis at the same time, and nothing in the first axis will move.

Here is the failure that breaks the one-scale model. The serving path applies a feature transform, calls the model, and logs a clean 200. Watch the corrupted transform produce a number in the same plausible range as the correct one, and watch every machine signal stay green while the prediction is garbage.

# WRONG MENTAL MODEL: "if it returned 200 with no error, the answer is fine."
# A transform regression sends raw income where the model was trained on
# log-income. Nothing throws. The dashboard shows the request as a success.

import math

def transform_correct(income: float) -> float:
    return math.log1p(income)        # what training did

def transform_broken(income: float) -> float:
    return income                    # serving drifted to raw dollars

def score(model_weight: float, feature: float) -> float:
    # a stand-in linear model squashed to 0..1; any real number scores fine
    z = model_weight * feature - 4.0
    return 1.0 / (1.0 + math.exp(-z))

income = 65000.0
ok = score(0.5, transform_correct(income))     # feature approx 11.08
bad = score(0.5, transform_broken(income))      # feature = 65000.0

print("correct score:", round(ok, 3))           # ~0.82, a moderate, sane score
print("broken score: ", round(bad, 3))          # 1.0, saturated, confidently wrong
print("http status:  ", 200)                     # both requests: 200 OK
print("exception:    ", None)                    # both requests: nothing raised

The broken transform handed the model a feature thousands of times larger than it ever saw in training, the sigmoid saturated to a confident 1.0, and the request still completed with a 200 and no exception. A wrong number is still a number. The status line describes the HTTP exchange, not the value inside the body, so it cannot distinguish a moderate 0.82 from a saturated 1.0.

The stand-in above isolates the mechanism; the same failure on the real M4-trained, M6-served scorer looks identical. The example below loads the actual artifact, scores one real applicant row, then introduces a unit bug (annual_inc accidentally divided by 1000, the kind of off-by-a-scale mistake a serving refactor produces) and scores the corrupted row. Watch the prediction move while the machine signals (status, latency, error rate, exception) stay exactly where they were.

"""Lesson 5.1 Show — the 200 OK that is quietly wrong.

A corrupted feature transform (here, a unit bug: income read in thousands) feeds the
scorer. It returns 200, normal latency, no exception, and a confident probability —
every machine signal is green. But the prediction is wrong. Uptime, latency, and error
rate cannot see this; "service up" is not "model right."
"""

import time

from de_refs import load_model, model_frame


def main() -> None:
    model = load_model()
    X, _ = model_frame()
    row = X.iloc[[0]].copy()

    start = time.perf_counter()
    good = float(model.predict_proba(row).iloc[0])
    latency_ms = (time.perf_counter() - start) * 1000

    # Silent corruption: annual_inc accidentally divided by 1000 (a unit bug).
    corrupted = row.copy()
    corrupted["annual_inc"] = corrupted["annual_inc"] / 1000
    wrong = float(model.predict_proba(corrupted).iloc[0])

    print("machine signals (all green):")
    print("  status: 200    latency: normal    error_rate: 0    exception: none")
    print(f"  measured latency: {latency_ms:.1f} ms")
    print(f"\ncorrect prediction:   {good:.4f}")
    print(
        f"corrupted prediction: {wrong:.4f}  <- confidently wrong, all signals still green"
    )
    print("uptime/latency/error cannot catch this — service up != model right")


if __name__ == "__main__":
    main()

The model returns a different probability on the corrupted row (the correct row scores around 0.23, the unit-bugged row around 0.29), and every infrastructure signal is identical to the correct request: 200, normal latency, no exception. The shift is modest here because one applicant’s mis-scaled income only nudges the score, but across a batch that nudge is thousands of loan decisions made on a fabricated feature, and nothing in the infra panel moved. This is the toy block’s lesson on the real artifact the curriculum built, and it is the failure the lesson opened with.

Why is this structural and not a bug you patch once? Because the answer determines what monitoring can ever catch it. Throwing requires a value the code rejects, and the silent-failure class produces values nothing rejects. A float is a float whether it is dollars or thousands-of-dollars. A feature vector is the right shape whether its columns are in training order or shuffled. A missing field replaced by a default is a real number, not a None that would blow up. Every member of the class produces the same shape (valid input, valid-looking output, HTTP 200) and none of them move a machine signal: the process is up, the request completed, latency is normal, no exception was thrown. The wrongness lives entirely in the numeric value, which is the one thing infra monitoring never inspects.

This class has named members, and they all collapse to that one shape. Unit or scale drift: serving sends raw income, training expected log-income, a plausible number and a garbage prediction, the case above. Column reindexing: two features swap positions in the vector; every value is in range, every score is wrong. Silent default substitution: a missing field is filled with 0 or the training mean, so the row scores on a fabricated value rather than failing. Stale or wrong artifact: an old model or preprocessing object is loaded after a deploy and quietly scores every request. Train/serve transform skew: the serving feature code drifts from the training feature code. This is exactly the train/serve contract from the data-wrangling module (M2) breaking, where the boundary between what the model was trained on and what it is served diverges, and it is the silent failure I opened this lesson with: the serving transform that changed underneath a model still returning 200. Five different root causes, one symptom: a confident number and a 200.

The serving path with two dashboards over it shows why the infra panel can never turn red on this failure. Scroll through the request as the transform breaks and watch the two panels move on independent axes.

The drift alert from detect-drift did not fire in the opening incident either, and the reason is the same orthogonality one level down. Drift detection watches the input distribution and fires when the marginal distribution of a feature moves. The transform corruption produced in-range values, so the input distribution had not moved; it was the mapping from input to output that broke, not the distribution of the inputs. A check on the marginal distribution is blind to a corruption that preserves the marginal and breaks the mapping. The failure was invisible to everything I had built; only a check on the outputs could have seen it, and I had not built one. That is the next concept.

import math


def transform(income: float, broken: bool) -> float:
    # training applied log1p; a serving regression skips it
    return income if broken else math.log1p(income)


def score_request(income: float, broken: bool) -> dict[str, object]:
    feature = transform(income, broken)
    z = 0.5 * feature - 4.0
    prob = 1.0 / (1.0 + math.exp(-z))
    # placeholder: fill in the status and exception the request would report
    return {"prob": round(prob, 4), "status": 0, "exception": "?"}


good = score_request(65000.0, False)
bad = score_request(65000.0, True)
print("good:", good)
print("bad: ", bad)

import math


def transform(income: float, broken: bool) -> float:
    return income if broken else math.log1p(income)


def score_request(income: float, broken: bool) -> dict[str, object]:
    feature = transform(income, broken)
    z = 0.5 * feature - 4.0
    prob = 1.0 / (1.0 + math.exp(-z))
    return {"prob": round(prob, 4), "status": 200, "exception": None}


good = score_request(65000.0, False)
bad = score_request(65000.0, True)
print("good:", good)
print("bad: ", bad)
print("same status?   ", good["status"] == bad["status"])
print("same exception?", good["exception"] == bad["exception"])

Proxies for correctness without ground truth

Once the green-but-wrong incident has happened, the natural next move is to monitor accuracy. The reasoning is sound on its face: correctness is accuracy, accuracy is correct_predictions / total, so compute it on a rolling window and alert when it drops. The trouble is the word correct: it requires the ground-truth label, the actual outcome, and for a loan-default model the outcome is whether the loan repaid or defaulted, which will not be known for months.

That interval between serving a prediction and learning whether it was right is the feedback-loop length, and it is set by the problem, not by your monitoring. For a loan-default model it is months; the accuracy dashboard, if you built one, would always be reporting on a cohort of loans whose fate was decided long before the transform broke. The wrong hypothesis that follows is “no ground truth yet means nothing to monitor until the labels come in,” so you wait, and the silent failure runs for weeks.

A correct mental model treats accuracy as the only thing that needs labels. Correctness has several proxies that are observable the instant a prediction is made, each one a necessary condition the right answer would satisfy. None of them confirms the model is right; each confirms it is not obviously wrong along one dimension. A layered set catches far more of the silent-failure taxonomy than any single check, because each proxy fails when a different part of the path breaks. The diagnosis discipline after a green-but-wrong incident is exactly this question: which proxy should have fired and did not? That gap is the next monitor to add.

The principle is that you cannot confirm the answer without the label, but you can bound it: a proxy is a check the correct answer must pass, so a failed proxy is proof of wrongness even though a passed proxy is not proof of rightness. Each proxy guards one slice of the path and is blind to others, which is why they are layered, not chosen.

Among the three, the output-distribution proxy is the one that would have caught the opening incident, and the mechanism is precise. With the model weights fixed, the function mapping input to output does not change, so a shift in the prediction distribution is a usable proxy for a shift in the input distribution: the predictions can only move if their inputs moved. Watch the corrupted transform from the last section push the whole batch of scores toward 1.0, and watch a reference-window check fire on that shift with no labels in hand.

import math
import statistics


def transform(income: float, broken: bool) -> float:
    return income if broken else math.log1p(income)


def score(income: float, broken: bool) -> float:
    z = 0.7 * transform(income, broken) - 7.0
    return 1.0 / (1.0 + math.exp(-z))


# a reference window captured when the model was known-good
reference = [score(i, False) for i in range(20000, 90000, 2500)]
ref_mean = statistics.mean(reference)

# today's batch, scored through the corrupted serving transform
live = [score(i, True) for i in range(20000, 90000, 2500)]
live_mean = statistics.mean(live)

print("reference mean score:", round(ref_mean, 3))
print("live mean score:     ", round(live_mean, 3))
shift = abs(live_mean - ref_mean)
print("mean shift:          ", round(shift, 3))
print("output-dist proxy fires:", shift > 0.15)  # with zero labels

A reference window sat near 0.64 with real spread across the income range; the corrupted batch collapsed to 1.0, and the mean moved more than a third of the [0, 1] interval past the band. The proxy fired on a number that exists the instant the prediction is made, with no label and no waiting for loans to mature. What it cannot tell you is why the distribution moved: a genuine shift in applicant income would move it the same way, which is why this proxy reports “something changed,” never “the model is wrong.”

A canary closes the slice the distribution check leaves open. The output-distribution proxy is blind to a corruption that happens to leave the aggregate distribution roughly unchanged, and it cannot separate a broken model from a shifted population. A canary record (one fixed applicant whose correct score is known and stable) does both: its “right answer” is fixed by construction, so when its score leaves the band, the population did not change, the path did. Watch a known-good record drop out of its band the moment the transform breaks.

import math


def transform(income: float, broken: bool) -> float:
    return income if broken else math.log1p(income)


def score(income: float, broken: bool) -> float:
    z = 0.7 * transform(income, broken) - 7.0
    return 1.0 / (1.0 + math.exp(-z))


# the canary: a fixed record whose score was stable for months
CANARY_INCOME = 65000.0
CANARY_BAND = (0.65, 0.71)  # the band the score has held in, known-good


def canary_in_band(broken: bool) -> bool:
    s = score(CANARY_INCOME, broken)
    lo, hi = CANARY_BAND
    return lo <= s <= hi


print("healthy serving -- score:", round(score(CANARY_INCOME, False), 3))
print("healthy in band?        ", canary_in_band(False))
print("broken serving -- score: ", round(score(CANARY_INCOME, True), 3))
print("broken in band?         ", canary_in_band(True))

No label is needed because the canary’s answer is fixed: the same record must score the same number every run, so a transform or artifact change that moves it out of band fires immediately, in minutes rather than the weeks a population-level signal might take to confirm. The cost is coverage, because one canary exercises one path through the feature space, so a corruption affecting a feature the canary does not stress slips past it. Coverage is a set of canaries chosen to stress different features, not one record.

Across all three proxies the non-obvious cost is the false-positive surface, and it is why these signals feed the alert tiers from alert-on-the-right-signal rather than paging directly. A legitimate population shift moves the output distribution; a deploy that retrains the model legitimately moves the canary’s band. A proxy wired straight to a pager will fire on every benign shift, the responder will mute the channel, and a muted proxy catches nothing, the exact actionability failure that lesson was about. Proxies detect; the tiering decides whether detection is worth interrupting a human.

import math


def transform(amount: float, broken: bool) -> float:
    # training scaled cents to dollars (/100); a regression skips the divide
    return amount if broken else amount / 100.0


def score(amount_cents: float, broken: bool) -> float:
    z = 0.002 * transform(amount_cents, broken) - 1.84
    return 1.0 / (1.0 + math.exp(-z))


CANARY_AMOUNT = 5000.0  # cents
CANARY_BAND = (0.10, 0.20)


def canary_fires(broken: bool) -> bool:
    s = score(CANARY_AMOUNT, broken)
    # placeholder: return whether the score is OUTSIDE the band
    return False


print("healthy fires?", canary_fires(False))
print("broken fires? ", canary_fires(True))

import math


def transform(amount: float, broken: bool) -> float:
    return amount if broken else amount / 100.0


def score(amount_cents: float, broken: bool) -> float:
    z = 0.002 * transform(amount_cents, broken) - 1.84
    return 1.0 / (1.0 + math.exp(-z))


CANARY_AMOUNT = 5000.0
CANARY_BAND = (0.10, 0.20)


def canary_fires(broken: bool) -> bool:
    s = score(CANARY_AMOUNT, broken)
    lo, hi = CANARY_BAND
    return not (lo <= s <= hi)


print("healthy score:", round(score(CANARY_AMOUNT, False), 3))
print("healthy fires?", canary_fires(False))
print("broken score: ", round(score(CANARY_AMOUNT, True), 3))
print("broken fires? ", canary_fires(True))

Reconstruct one prediction: the rubric, closed

Detection is where most monitoring stops, and it feels like enough: the canary fired, the output distribution shifted, the model is wrong. The assumption underneath is that knowing that the model failed is the same as being able to act on it. It is not. A monitor that detects a problem but cannot reconstruct the offending prediction can tell you that the model failed and never why, so a disputed score becomes unresolvable and the post-mortem stalls on “we cannot reproduce it.”

This module’s real payoff is the ability to answer, after the fact, “why did the model return that for this request.” That answer requires the full causal record of one prediction (the inputs it actually saw, the artifact that scored it, the contract that was in force, and the output it produced) tied together by one key. Reconstruction is a join, and the join only works if every field was written at emit time. You cannot reconstruct what you did not log; collection can only store what emission produced. This is mechanical work, but it is decided at instrumentation time, not incident time: by the time a prediction is disputed, the missing field is already gone, and you learn what you forgot to log during the exact incident that needs it.

Holding the join together is the correlation id from get-monitoring-working, the per-request identifier written on every log line for that request, which ties the log line, the inputs, the artifact hash, and the output into one story; without it a prediction’s many records cannot be re-associated among millions. Each other field answers a question that becomes unanswerable if it is missing, and the static record below shows what each field buys and what its absence costs.

[correlation id] as cid
[logged inputs] as inputs
[model version / artifact hash] as model
[contract version] as contract
[output] as output

cid --> inputs : "can I re-run it?"
cid --> model : "which artifact scored it?"
cid --> contract : "valid at the time?"
cid --> output : "what did it return?"

inputs --> output : replay must match
model --> output : replay must match

Each edge from the correlation id is a question, and dropping the field at the other end makes that question unanswerable. Without the logged inputs, you can see what came out but never reproduce why, so you cannot distinguish a model bug from a bad input. Without the model version / artifact hash, you cannot tell which of several deployed artifacts scored the request: a blue/green or canary fleet runs more than one model version at once, and “which model produced this” is the first question in any regression. Without the contract version, you cannot tell whether the input was valid at the time it was scored, since a contract that tightened after the prediction would wrongly flag a then-legal input. The replay proves it: load the recorded artifact by its hash, feed it the logged inputs, and confirm the re-computed score matches the logged output.

Here is the full record for one prediction and the replay that closes the loop. Watch the join pull every field by the correlation id, re-run the model on the logged inputs, and confirm the recomputed score matches what was logged.

import math


# the deployed artifacts, keyed by hash -- a fleet runs more than one at once
def model_v1(feature: float) -> float:
    return round(1.0 / (1.0 + math.exp(-(0.9 * feature - 4.0))), 4)


def model_v2(feature: float) -> float:
    return round(1.0 / (1.0 + math.exp(-(0.7 * feature - 3.0))), 4)


ARTIFACTS = {"a1b2c3": model_v1, "d4e5f6": model_v2}

# the prediction log: every field captured at emit time, keyed by correlation id
PREDICTION_LOG = {
    "req-7f3a": {
        "inputs": {"log_income": 11.08},
        "model_hash": "a1b2c3",
        "contract_version": "v3",
        "output": model_v1(11.08),
    }
}


def reconstruct(correlation_id: str) -> dict[str, object]:
    rec = PREDICTION_LOG[correlation_id]  # the join, by the one key
    artifact = ARTIFACTS[rec["model_hash"]]  # which model scored it
    replayed = artifact(rec["inputs"]["log_income"])  # re-run on the logged inputs
    return {
        "correlation_id": correlation_id,
        "model_hash": rec["model_hash"],
        "contract_version": rec["contract_version"],
        "logged_output": rec["output"],
        "replayed_output": replayed,
        "record_complete": replayed == rec["output"],
    }


result = reconstruct("req-7f3a")
for key, value in result.items():
    print(key + ":", value)

Re-running the named artifact on the logged inputs matched the logged output exactly, which proves two things at once: the record carries every field the score depended on, and the model is deterministic for that input. A match is the proof the record is complete; a mismatch means either a field is missing or the artifact is not the one that actually scored the request.

The toy record above carries one feature to keep the join visible. A real prediction log carries the full feature row, and the same reconstruction runs on the real M4 artifact. The example below holds a log record with all nine loan features the model actually scores on (loan_amnt, int_rate, annual_inc, dti, revol_util, total_acc, purpose, home_ownership, term), loads the real artifact, and replays the score from the logged inputs to confirm it matches what was recorded.

"""Lesson 5.3 Show — reconstruct one prediction from its logs.

The closing rubric: given a request id, join the logged inputs + model version +
contract version + recorded output, then re-run the score and confirm it matches.
If it matches, you can explain any single prediction after the fact — the property
that makes a model auditable. The fields you need (inputs, versions, output) are
exactly what Lessons 1-5 logged.
"""

import pandas as pd

from de_refs import load_model

# A log record as it would have been written at serve time.
LOG_RECORD = {
    "request_id": "req-eval-007",
    "model_version": "loan-scorer-1.0",
    "contract_version": "v1",
    "inputs": {
        "loan_amnt": 10000,
        "int_rate": 13.5,
        "annual_inc": 45000,
        "dti": 18.0,
        "revol_util": 55.0,
        "total_acc": 12,
        "purpose": "credit_card",
        "home_ownership": "RENT",
        "term": "36 months",
    },
    "output": None,  # filled below from the original score
}


def main() -> None:
    model = load_model()
    # Pretend this was the score logged at serve time.
    original = float(model.predict_proba(pd.DataFrame([LOG_RECORD["inputs"]])).iloc[0])
    LOG_RECORD["output"] = round(original, 4)

    # Reconstruct: re-run the score from the logged inputs + version.
    rerun = round(
        float(model.predict_proba(pd.DataFrame([LOG_RECORD["inputs"]])).iloc[0]), 4
    )

    print(f"request_id:       {LOG_RECORD['request_id']}")
    print(f"model_version:    {LOG_RECORD['model_version']}")
    print(f"logged output:    {LOG_RECORD['output']}")
    print(f"reconstructed:    {rerun}")
    print(f"reconstruction matches: {rerun == LOG_RECORD['output']}")
    print(
        "inputs + versions + output logged -> any prediction is explainable after the fact"
    )


if __name__ == "__main__":
    main()

The replay reproduces the logged probability exactly from the real artifact and the full feature row, which is what makes any single production prediction explainable after the fact. The fields the replay needs (the inputs, the model version, the contract version, the output) are precisely the ones the earlier lessons logged, so the audit trail was built before the dispute, not during it.

One non-obvious constraint governs the whole technique: replay only works if scoring is deterministic given inputs and artifact. Any hidden nondeterminism breaks it, whether a random seed, a clock-dependent feature, or a config value read from the environment rather than logged. The named failure mode is the unlogged dependency: the replay produces a different score than the log, the post-mortem concludes the record is corrupt, and the real cause is a feature that read datetime.now() or an environment variable that was never written to the log line. The fix is the discipline the constraint forces: log every input the score depended on, including the ones that feel like configuration, because a “config” value the model reads is an input by another name.

This is the redemption of the correlation id you put on every log line in get-monitoring-working. There it was a field that tied a request’s log lines together; here it is the join key that turns a disputed prediction into a one-query answer: pull the id, load the named artifact, replay the inputs, confirm the score. A monitoring system that detects without reconstructing can raise the alarm and never resolve the dispute. One that reconstructs closes it.

import math


def model_v1(feature: float) -> float:
    return round(1.0 / (1.0 + math.exp(-(0.9 * feature - 4.0))), 4)


def model_v2(feature: float) -> float:
    return round(1.0 / (1.0 + math.exp(-(0.7 * feature - 3.0))), 4)


ARTIFACTS = {"a1b2c3": model_v1, "d4e5f6": model_v2}

# this record is missing the model_hash -- two artifacts are deployed
PREDICTION_LOG = {
    "req-9c2b": {
        "inputs": {"log_income": 10.5},
        "contract_version": "v3",
        "output": 0.8176,
    },
}


def can_reconstruct(correlation_id: str) -> bool:
    rec = PREDICTION_LOG[correlation_id]
    # placeholder: return whether the record has the field needed to pick the artifact
    return True


print("can reconstruct?", can_reconstruct("req-9c2b"))

import math


def model_v1(feature: float) -> float:
    return round(1.0 / (1.0 + math.exp(-(0.9 * feature - 4.0))), 4)


def model_v2(feature: float) -> float:
    return round(1.0 / (1.0 + math.exp(-(0.7 * feature - 3.0))), 4)


ARTIFACTS = {"a1b2c3": model_v1, "d4e5f6": model_v2}

PREDICTION_LOG = {
    "req-9c2b": {
        "inputs": {"log_income": 10.5},
        "contract_version": "v3",
        "output": 0.8176,
    },
}


def can_reconstruct(correlation_id: str) -> bool:
    rec = PREDICTION_LOG[correlation_id]
    return "model_hash" in rec


rec = PREDICTION_LOG["req-9c2b"]
print("has model_hash?  ", "model_hash" in rec)
print("can reconstruct? ", can_reconstruct("req-9c2b"))
print("v1 replay:", model_v1(rec["inputs"]["log_income"]))
print("v2 replay:", model_v2(rec["inputs"]["log_income"]))
print("logged output:", rec["output"])

Summary

• A 200 OK describes the HTTP exchange, not the value in the body: infrastructure health and model correctness are independent axes, and a service can sit at the top of one and zero on the other at the same time.
• The silent-failure class (unit drift, column reindexing, default substitution, stale artifact, train/serve skew) collapses to one shape: type-valid input, valid-looking output, HTTP 200, so no machine signal moves and only a check on the output or the input’s meaning can catch it.
• Accuracy needs labels that arrive on the feedback-loop length (months for loan default), but correctness has label-free proxies (the input contract, the output-distribution check, the canary) each guarding a different slice of the path.
• A failed proxy proves wrongness; a passed proxy does not prove rightness. Proxies feed the alert tiers rather than paging directly, because each has a benign-shift false-positive surface that mutes the channel if wired to a pager.
• Reconstruction is a join keyed by the correlation id: log the inputs, the model/artifact hash, the contract version, and the output at emit time, and replay proves the record complete by re-running the artifact on the logged inputs.

Check your understanding:

• Why can every infrastructure signal be green while the model is 100% wrong, and which class of signal would have to be wrong to catch it?
• Name two label-free proxies for correctness. For the green-but-wrong transform incident, which one would have fired and why was the input-drift alert silent?
• What fields must a log line carry to reconstruct one prediction, and which single field tells you which model produced it when a fleet runs more than one version at once?