Project — Monitoring, Observability & Reliability

An observability layer that distinguishes “the service is up” from “the model is still right” for the Adult / Census Income dataset — the monitoring around a running income-prediction service that tells you the model is quietly wrong before a user does.

Start here

1. Fork the starter → dutchengineer-org/phase3-starter, clone it, and install it, then run python train.py and start serve.py to have a live service to watch
2. Work the tasks below in order — each maps to a lesson you just finished
3. Check yourself against the rubric, then push and submit

By the end, when a teammate asks “is the model okay right now?” you can answer from your dashboard and logs, not from a guess: structured request logs, a latency and error-rate dashboard, a drift check on the inputs, and alerts that fire on the signals that matter and stay silent the rest of the time.

The tasks

Do these in order; each maps to a lesson you just finished. The service is already up; your work is the monitoring around it.

1. Get monitoring working on the live service (from Lesson 1)

• Add structured (JSON) logging to the request path: every prediction logs a request id, the input fields, the prediction, and the latency: one line, machine-parseable.
• Expose a real health check endpoint (returns the service’s readiness, not just 200) and track an error count so a spike in failed requests is visible, not buried.
• Hit the live service a few times and confirm each call produces a complete log line.

2. Make it measurable: metrics, traces, and a dashboard (from Lesson 2)

• Instrument the service to emit latency metrics and surface p50 and p99 (not just the average; the tail is where users feel it).
• Stand up a dashboard (Grafana, the host’s built-in metrics view, or equivalent) that shows request rate, error rate, and p50/p99 latency over time.
• Add enough trace/correlation context that you can follow one slow request end to end.

3. Detect drift: when the inputs stop looking like training (from Lesson 3)

• Write a drift check that compares recent live inputs against the training distribution using PSI (Population Stability Index) on the key features.
• Track the unknown-category rate (the share of categorical values the model never saw in training) and surface it alongside PSI.
• Run the check against a deliberately shifted batch and confirm it reports the drift.

4. Alert on the right signal (and stay quiet otherwise) (from Lesson 4)

• Define alert rules on the signals that mean real trouble: error-rate spike, p99 blowing past its budget, PSI crossing a threshold, unknown-category rate climbing.
• Tune each rule so it does not fire on normal noise: choose thresholds and a for-duration window, not a hair-trigger on a single bad data point.
• Document, per alert, what it means and the first thing the on-call person should check.

5. Service up ≠ model right: catch the silent failure (from Lesson 5)

• Add a check that catches a silent failure: the service returns 200s on time, but the predictions have quietly gone wrong (e.g. all one class, a collapsed score distribution, or input drift the health check cannot see).
• Confirm your monitoring distinguishes this case from a healthy service: the health check stays green, but a drift/quality signal goes red and an alert fires.

Rubric — your project is done when

This is the standard the module holds you to (each bar maps to the lesson that taught it):

• Logs reconstruct a single prediction after the fact — given a request id, your structured logs recover that call’s inputs, output, latency, and outcome. (Lesson 1)
• The service is measurable, not just alive — a dashboard shows request rate, error rate, and p50/p99 latency over time. (Lesson 2)
• It detects an injected input/prediction drift — feeding a shifted batch trips your PSI and/or unknown-category check; a healthy batch does not. (Lessons 3, 5)
• An alert fires on the right signal and stays quiet otherwise — your rules trigger on real trouble and survive a replay of normal traffic without crying wolf. (Lesson 4)
• “Service up” is distinguished from “model right” — when predictions go silently wrong, the health check stays green and a quality/drift alert fires. (Lesson 5)

Run the silent-failure drill from Lesson 5 against your own service before you submit: inject a drift, confirm the health check stays green, and confirm the right alert fires. If nothing fires, name which signal you are still missing.

Submit

Use the branch workflow from the M1 git lesson, not commits straight to main. Branch off main (git checkout -b m10-monitoring), build this module’s piece there, and open a pull request to merge it back once it meets the rubric. main stays the last-good version of the product you carry forward, so a half-finished module never breaks what later modules build on.

When your repo meets every rubric line, merge your branch to main, push it to GitHub, and submit the repository URL here. (Submission coming soon.)