Project — MLOps: CI/CD & Retraining

A CI pipeline that tests data and models, not just code, plus a retraining runbook for the Adult / Census Income dataset — the automation loop that keeps the service you have carried through the track honest as its data drifts.

Start here

1. Fork the starter → dutchengineer-org/phase3-starter, clone it, and install it. Already forked the Phase 3 starter in M10? Keep building in that fork
2. Work the tasks below in order — each maps to a lesson you just finished
3. Check yourself against the rubric, then push and submit

By the end, a teammate should be able to push a commit that corrupts the data and watch CI go red, then follow your runbook to promote a new model version through a shadow → canary → full rollout, without betting the whole service on it.

The tasks

Do these in order; each maps to a lesson you just finished.

1. Get a CI pipeline going green (from Lesson 1)

• Add a .github/workflows/ci.yml that triggers on: push, runs on a clean runner, installs from your lock file, runs your code tests, and builds the model artifact.
• Order the stages cheapest-failure-first (lint → unit test → data tests → model gate → build) so a trivial failure costs seconds, not a full training run.
• Push a commit and confirm a green check appears on it.

2. Make CI catch bad data (from Lesson 2)

• Add data tests that run in the pipeline as their own stage (after install, before train): a schema check (column names, dtypes, nullability), range checks (e.g. age >= 0, hours-per-week within bounds, label in the valid set), and a leakage check that asserts no post-outcome / serve-time-unavailable column is in the feature set.
• Prove it: push a bad-DATA commit (corrupt a dtype or inject an out-of-range value) and confirm CI fails on the data assertion, not a code error.

3. Gate the model and version it (from Lesson 3)

• Add a model gate: the new model must beat a baseline on a metric that survives Adult/Census’s ~24% positive class (recall or PR-AUC, not raw accuracy), measured on a held-out set it never trained on. A degenerate majority-class model must fail.
• Make every promoted model a version: write a manifest next to the artifact recording the four reproducibility inputs {data_hash, git_commit, hyperparameters, seed} (plus the gate metrics for the record) so any version is reproducible and addressable, and rollback means picking a version, not retraining from memory. The seed is not optional: drop it and a stochastic learner gives a different fit from byte-identical inputs, exactly the Lesson 3 trap.

4. Write the retraining runbook (from Lesson 4)

• Write RUNBOOK.md: the retraining loop, the trigger (drift detected by your M10 detectors), and, critically, when NOT to retrain. State that drift is a trigger, not a verdict: a drift signal opens an investigation, it does not auto-fire training.
• Document the poisoned-data guard: before retraining, validate the new data against your Lesson 2 data tests, so a drift alarm caused by corrupted upstream data never pulls that corruption into the next model.

5. Roll out a new model safely (from Lesson 5)

• Implement (or script) shadow → canary → full promotion: a new version first runs in shadow (scores live traffic, serves nothing), then canary (a small slice of real traffic), then full, with a documented criterion to advance or roll back at each gate.
• Ensure the feature transform is identical in training and serving (train/serve skew is exactly what the shadow exists to catch), so the shadow model sees the same inputs the live model does. Document the rollback path.

Rubric — your project is done when

This is the standard the module holds you to (each bar maps to the lesson that taught it):

• CI runs green on every push — on: push, clean runner, installs from the lock file, stages ordered cheapest-failure-first. (Lesson 1)
• CI fails on a bad-DATA commit, not just bad code — schema, range, and leakage data tests run in the pipeline and turn it red on corrupted data with no code error. (Lesson 2)
• The model is gated against a baseline — a new model must beat the baseline on a metric that survives class imbalance (recall / PR-AUC) on held-out data; a majority-class model fails. (Lesson 3)
• Every model version is reproducible — the artifact is bound to a manifest of data hash, code commit, hyperparameters, and seed (the four inputs that reproduce the fit), with the gate metrics recorded alongside; rollback is selecting a version, not retraining from memory. (Lesson 3)
• The runbook states when NOT to retrain — drift is documented as a trigger, not a verdict, with a guard against retraining on poisoned data. (Lesson 4)
• A new model rolls out shadow → canary → full — promotion is staged with a documented advance/rollback criterion at each gate, and the train/serve transform matches so the shadow comparison is honest. (Lesson 5)

Run your own pipeline against each bar before you submit: push the bad-DATA commit and the strictly-worse model, and confirm CI stops both: if either ships green, name which lesson’s failure mode you reintroduced.

Submit

Use the branch workflow from the M1 git lesson, not commits straight to main. Branch off main (git checkout -b m11-ci-retraining), build this module’s piece there, and open a pull request to merge it back once it meets the rubric. main stays the last-good version of the product you carry forward, so a half-finished module never breaks what later modules build on.

When your repo meets every rubric line, merge your branch to main, push it to GitHub, and submit the repository URL here. (Submission coming soon.)