What You Built in This Module

The worst failure this module exists for does not crash, does not slow down, and does not return an error — it returns 200 OK with a confident wrong answer while every infrastructure signal stays green. A green dashboard is a claim, and the signals you built are what let you check it instead of trust it. What you can operate now tells you not merely that the service is running but whether the model is right, watches its own inputs for the drift that decays a model quietly, and pages a human only when one is actually needed.

What you completed

The smallest real observability came first; each lesson after it added a signal the layer below it could not see, ending on the failure none of the standard ones catch:

• The three pillars recover different things, and you only learn which when you need them. Logs, metrics, and traces are not interchangeable — one reconstructs a single request, one trends a rate, one ties a path together — and a health check that means ready is a different claim from one that means the process is alive. Standing up queryable logs and an honest health check is what makes the later signals interpretable rather than decorative.
• Averages hide the tail and counters lie across restarts. A latency average smooths over the p99 that pages you, a raw counter resets to zero on restart so only rate() reads it correctly, and a high-cardinality label can quietly exhaust a metrics system’s memory. These are the traps that make a dashboard confidently wrong, and turning activity into metrics you can reason about means knowing each one is there.
• No infrastructure signal can see a model decaying. Error rate, latency, and health stay green while the live input distribution walks away from training, which is why drift has to be measured on the data itself with PSI and the KS statistic against a reference window. The window length is the judgment: too short cries wolf, too long dilutes a real shift away before it crosses a threshold.
• An alert that fires on noise is an alert you have trained yourself to ignore. Paging on causes and single spikes buries the one page that matters under ones that do not, so the discipline is alerting on symptoms a user actually feels and sustained breaches rather than transients. Every page should be something a human can act on, or it should not page.
• 200 OK is a claim about the request, not the answer. With no immediate ground truth you build proxies for correctness — prediction-distribution checks, score sanity, agreement over time — because “the service is up” and “the model is right” are two different questions and only the first is easy. Reconstructing one prediction end to end is what shows you exactly where a silent wrong answer would hide.

Check your understanding

Work these without scrolling back up. Each one is a new service, not the loan-scoring system the lessons used — the point is to apply the judgment, not recall the case. Try to answer before opening the hint.

1. An energy-grid demand-forecasting service is in production with no observability beyond a health check that returns 200 whenever the process is running. You need to be able to reconstruct a single forecast request when it goes wrong. Name the three observability pillars and what each one recovers, and explain why a health check that means alive is a weaker claim than one that means ready.

   Hint

   The pillars are not interchangeable — one reconstructs a single request, one trends a rate, one ties a path together. Think about which pillar lets you replay one forecast, and why "the process is up" is a different claim from "it can actually serve."

2. The demand-forecast dashboard shows an average latency that looks healthy, a request counter that drops to zero whenever a pod restarts, and a per-customer-id label that is slowly exhausting the metrics store. Explain the trap in each of those three and what you would do instead for each.

   Hint

   Each is a different way a dashboard goes confidently wrong. Think about what an average hides about the tail, why a raw counter needs `rate()` to read it across restarts, and why an unbounded-cardinality label is a memory hazard.

3. The demand-forecast service stays green on error rate, latency, and health, yet its forecasts have been drifting worse for weeks as the live load pattern moves away from what it trained on. Explain why no infrastructure signal catches this, what you would measure on the data itself, and the judgment call in choosing the reference window.

   Hint

   Infrastructure signals cannot see a model decaying — the inputs walk away while everything stays green. Think about measuring the input distribution against a reference window with PSI and the KS statistic, and why too short a window cries wolf and too long dilutes a real shift away.

4. The on-call engineer for the demand-forecast service is paged so often by single CPU spikes and an upstream cause alert that they have started ignoring the pager. Explain what makes these bad alerts, and the discipline that decides what should page a human versus what should not.

   Hint

   An alert that fires on noise is one you have trained yourself to ignore. Think about paging on symptoms a user actually feels rather than causes, and on sustained breaches rather than transient spikes — every page should be something a human can act on.

5. The demand-forecast service returns 200 OK on every request, but a downstream team reports its forecasts have been quietly wrong for days with no error anywhere. There is no immediate ground truth to check against. Explain why “the service is up” and “the model is right” are different questions, and the proxies you would build for correctness.

   Hint

   `200 OK` is a claim about the request, not the answer. With no ground truth, think about prediction-distribution checks, score sanity, and agreement over time, and why reconstructing one prediction end to end shows where a silent wrong answer hides.

What it adds up to

A green dashboard is a claim, not a proof — and this module built the signals that turn the claim into something you can actually check. The system you can now operate tells you not just that it is running but whether it is right, watches its own inputs for the drift that decays a model quietly, and pages a human only when a human is needed. That is the difference between a deployed model and a model someone can rely on without watching it by hand.

What comes next

You can now see when the model is going wrong. What you cannot yet do is fix it safely. A drift alert tells you the model needs to change; it does not tell you how to ship a new one without betting the whole system on it. The final module closes the loop: continuous integration that tests data and models, not just code, and a retraining pipeline that rolls out a new model version behind a gate — so the signals you built here become the trigger for a safe, automatic improvement instead of a 2 a.m. scramble.