A Prediction Is Not a Decision

I rendered the model’s probability next to the label because more information seemed better, and the credit lead read that decimal as the fraction of those borrowers who would default. She started declining applications on it before anyone caught what the number actually was. It meant nothing of the sort: roughly a fifth of the loans in that book charged off, the model was ranking applicants by relative risk, and the decimal on the screen was a relative score, not a calibrated frequency. The same week, wiring up authentication, I opened my own app.js and found the API key sitting in the source, shipped to every browser that had ever loaded the page, because I had hardcoded it weeks earlier and forgotten. No one reviewed that change; it was a small team and the page worked perfectly the whole time. Rotating the key and re-deploying everything that used it cost the better part of a day. The dashboard worked and was misleading on two fronts at once, and here the stakes were a person being denied credit on a number that did not mean what it looked like.

The previous lesson made the dashboard degrade honestly: it checked response.ok before reading the body, replaced any stale prediction with an explicit unavailable state on failure, and disabled the action that depended on a fresh score. That hardening was about what the page does when the call breaks. This lesson is about what the page communicates when the call succeeds, which is where an ML frontend does its quietest damage. A probability that arrives correctly and renders correctly can still walk a human into a wrong decision, because the number does not mean what an untrained reader assumes, the cutoff that turns it into an action is a business choice nobody made on purpose, and the credential that fetched it shipped to the browser in plain sight. Those are the three rubric lines of the module project, and they are the difference between a working dashboard and an honest one.

This is also where the translation job ends up. Back in the first module, the data scientist’s notebook was how the model got explained to people; that explaining role had nowhere to live once the notebook became a package, and it has been quietly travelling downstream ever since. It lands here. The dashboard is the model’s stakeholder view reborn for a new audience — one that cannot read a calibration plot or a confusion matrix and will simply act on whatever the screen says. So the question this lesson really asks is the translation question: what does this user do with the number, and what will they wrongly assume they can do? Getting that right is the last and most human translation in the whole path, and it is the one no library performs for you.

A raw probability shown to a non-ML human misleads

The reasonable mental model is that the model returns a probability, a probability is a chance out of one hundred, so rendering 0.73 tells the reader there is a 73% chance this borrower defaults. That model is wrong on two independent counts, and a non-ML reader cannot see either of them from the digits. The first is that the number may not be a frequency at all. The second is that even a true frequency means nothing without the base rate the reader never sees.

A classifier’s predict_proba output is a number in the interval from zero to one, but calibration (whether that number matches observed reality) is a separate property the model may or may not have. A model is calibrated when its scores match observed frequencies: among every record it scored near 0.7, roughly 70% should actually turn out positive. You measure it by building a reliability curve: bin the predictions by score, plot each bin’s mean predicted score against the fraction of that bin that turned out positive, and a perfectly calibrated model lies on the diagonal. Calibration is not automatic because most training objectives optimize ranking or class separation, not the absolute magnitude of the score. Gradient-boosted trees, the model stack carried through this track, and support vector machines are the classic offenders: boosting minimizes a loss that pushes scores toward the extremes to drive the classes apart, so the reliability curve comes out S-shaped, with confident scores too confident and mid-range scores pulled away from the truth, and a 0.73 from such a model can correspond to an observed default rate nowhere near 73% among the records it scored there. The fix at the model layer is a calibration step fit on held-out data — the classical-ML module built exactly that, a monotonic remap on a slice the model never trained on — but the dashboard does not get to assume the model it calls has one.

Stacked on top of miscalibration is base-rate neglect. The base rate is the prior, the share of the whole population that is actually positive, and on this loan book roughly a fifth of borrowers charge off. A reader who treats the score as an absolute chance ignores that prior entirely, so even a correctly-ranked high-risk applicant reads as near-certain when the true probability sits far lower. The wrong notional machine is “the number on the screen equals the chance the thing happens.” The actual machine produces a relative, possibly-uncalibrated score whose meaning depends on a prior the screen does not show. The frontend is the last place that gap can be communicated honestly or hidden.

The reliability curve formalizes calibration as a comparison between two quantities at each score bin:

$$ \text{calibrated} \iff \text{mean predicted score in bin} \approx \frac{\text{positives in bin}}{\text{records in bin}} $$

The left side is the digits the model emits; the right side is the fraction of that bin that actually turned out positive. A model is calibrated only when those two agree across every bin, the diagonal of the reliability diagram. When they diverge, the score on the screen is not the frequency the reader reads into it.

This block prints the verified base rate of the loan book and then shows how the same score reads against it. Watch the gap between the digits a reader sees and the rate the population actually carries.

def base_rate(labels: list[int]) -> float:
    """Fraction of the population that is actually positive (charged off)."""
    return sum(labels) / len(labels)


# A representative slice of the loan outcomes. On the full vendored Lending Club
# table, y.mean() == 0.183 -- roughly a fifth of loans charge off. This slice is
# built to land on that same prior so the lesson's number is the real one.
outcomes: list[int] = [1] * 183 + [0] * 817  # 1 = charged off, 0 = repaid
prior: float = base_rate(outcomes)
print("base rate (charge-off):", round(prior, 3))

# A reader sees this score and reads it as 'a 73% chance'.
score: float = 0.73
print("score on screen:", score)
print("reader's wrong reading:", f"{score:.0%} of these borrowers default")
print("population that actually defaults:", f"{prior:.0%}")

At 0.183, the verified figure for the full table, the base rate sits far below where a reader anchors 0.73, which overshoots the true population rate by roughly four times. The score may rank this borrower above most others correctly while still carrying a true default probability far below the digits suggest: the rank is meaningful, the magnitude is not. Why does a perfectly correct ranking still mislead when rendered as a raw decimal? Because rank locates this applicant relative to the rest, while the reader is reading the decimal as an absolute frequency anchored to nothing.

The named failure mode is the opening incident: a raw probability rendered beside the label, read as a real-world default rate, turned into a string of declined applications that were off the mark. On a small team there was no analyst layer between the model and the person acting on it; the page was the interface, so the misread went straight into a credit decision. The fix is not hiding the number; it is framing it so the reader’s mental model matches what the model does. The same scores rendered as a likelihood band read honestly where the raw decimals read as certainty.

def render_raw(score: float) -> str:
    """The misleading rendering: a bare decimal a reader takes as a frequency."""
    return f"default probability: {score}"


def render_banded(score: float) -> str:
    """The honest rendering: a relative likelihood band, not a guaranteed rate."""
    if score >= 0.66:
        band = "HIGH relative likelihood"
    elif score >= 0.33:
        band = "MEDIUM relative likelihood"
    else:
        band = "LOW relative likelihood"
    return f"{band} (relative score, not a guaranteed rate)"


scores: list[float] = [0.73, 0.41, 0.08]
for s in scores:
    print("raw:   ", render_raw(s))
    print("banded:", render_banded(s))
    print()

Identical model output reads as near-certainty in the raw framing and as “high relative likelihood” in the banded one. The band discards precision the model never earned, since the difference between 0.73 and 0.71 is noise on an uncalibrated score, and keeps only the ordering the model is actually good at. That is the trade: a band gives up the false exactness of a decimal to stop the reader inventing a frequency that is not there.

def render_banded(score: float) -> str:
    """Return a likelihood band, not a raw decimal. Replace the placeholder."""
    # Map the score to LOW / MEDIUM / HIGH relative likelihood.
    return "REPLACE ME"


for s in (0.91, 0.50):
    print(s, "->", render_banded(s))

def render_banded(score: float) -> str:
    """Return a likelihood band, not a raw decimal."""
    if score >= 0.66:
        return "HIGH relative likelihood"
    if score >= 0.33:
        return "MEDIUM relative likelihood"
    return "LOW relative likelihood"


for s in (0.91, 0.50):
    print(s, "->", render_banded(s))

Who owns the threshold: the model, the API, or the frontend

Once the score is framed honestly, the next reasonable assumption is that turning it into an action is the model’s job: the model decides yes or no, and the page only shows the verdict. The cutoff that collapses a continuous score into a yes/no label is not a model property. It is a business decision about the relative cost of two different mistakes, and 0.5 is the arithmetic midpoint, not a principled choice. On a class-imbalanced problem the default 0.5 is almost always wrong, and the frontend is frequently where the action that depends on it lives.

A classifier outputs a continuous score; collapsing it to a label requires a decision threshold. Moving that threshold trades two errors against each other along a fixed curve. A lower cutoff labels more applicants “will default”: it catches more true defaulters (recall rises) but also flags more borrowers who would have repaid (precision falls, more false positives). A higher cutoff reverses the trade: fewer false alarms, but more real defaulters slip through as false negatives. No setting improves both at once for a given model; the threshold only chooses where on that curve the model sits. The right place depends on the relative cost of a false positive (declining a borrower who would have repaid, a lost good loan) versus a false negative (approving one who defaults, lost principal), and that cost ratio is a business input, nowhere in the model.

On a book where roughly four borrowers in five repay, the imbalance makes 0.5 a sharp trap. A 0.5 cutoff optimizes for raw accuracy, and a model can score high accuracy by predicting “will repay” for almost everyone, so 0.5 produces a label set that flags a trickle and misses most of the actual defaulters. That is the “the model is useless” complaint, and the cause is not the model. This block applies two thresholds to the same scores and counts what each flags. Watch the caught-and-missed counts move, not the threshold value.

scores: list[float] = [0.62, 0.55, 0.48, 0.41, 0.30, 0.22, 0.71, 0.12, 0.58, 0.44]
truth: list[int] = [1, 0, 1, 0, 0, 0, 1, 0, 1, 0]  # 1 = actually charged off


def confusion(
    scores: list[float], truth: list[int], thr: float
) -> tuple[int, int, int]:
    """Return (flagged, true_positives, false_negatives) at a threshold."""
    flagged = sum(1 for s in scores if s >= thr)
    tp = sum(1 for s, y in zip(scores, truth) if s >= thr and y == 1)
    fn = sum(1 for s, y in zip(scores, truth) if s < thr and y == 1)
    return flagged, tp, fn


for thr in (0.5, 0.4):
    flagged, tp, fn = confusion(scores, truth, thr)
    print(f"threshold {thr}: flagged {flagged}, caught {tp} defaults, missed {fn}")

At 0.5 the model flags a handful and lets real defaulters through as false negatives; dropping the threshold to 0.4 catches more of them at the cost of flagging more borrowers overall. Moving one number, not retraining, is what turns the model from “useless” to “useful,” and 0.5 was the worst choice for an imbalanced problem because it is tuned for an accuracy that the majority class already pins. The named failure mode is the opening’s second-order version: I treated probability >= 0.5 as “will default” because it was the obvious cutoff, the model flagged almost nobody, and the credit lead told me the model was useless. The model was fine. The threshold was buried in frontend code I had written myself, where no one, me included, thought to question it. I nearly retrained the thing before realizing I had to move one number.

The design decision is therefore where the threshold lives, because that determines who can change it and how visible it is. The M6 /predict response deliberately returns both label (the threshold already applied server-side) and the raw probability so a consumer can re-threshold. The choice is between three places.

The non-obvious cost is that the most convenient option, hardcoding the cutoff in the page, is the one that fails silently. A wrong threshold in app.js does not throw; it produces a confident, working dashboard that flags the wrong set of applicants, and the people who own the cost trade-off cannot even see the number that encodes their policy. The threshold belongs where the cost owner can change it without a frontend deploy, which is the server’s label by default.

scores: list[float] = [0.60, 0.52, 0.47, 0.38, 0.29, 0.66, 0.18, 0.41]
truth: list[int] = [1, 0, 1, 0, 0, 1, 0, 1]  # 1 = actually positive


def caught(scores: list[float], truth: list[int], thr: float) -> int:
    """Return how many real positives are flagged at this threshold."""
    # Count records where score >= thr AND truth == 1.
    return 0


for thr in (0.5, 0.35):
    print(f"threshold {thr}: caught {caught(scores, truth, thr)} of the real positives")
# Comment: a fraud team lowers the threshold to avoid ____ (false positives / false negatives)?

scores: list[float] = [0.60, 0.52, 0.47, 0.38, 0.29, 0.66, 0.18, 0.41]
truth: list[int] = [1, 0, 1, 0, 0, 1, 0, 1]  # 1 = actually positive


def caught(scores: list[float], truth: list[int], thr: float) -> int:
    """Return how many real positives are flagged at this threshold."""
    return sum(1 for s, y in zip(scores, truth) if s >= thr and y == 1)


for thr in (0.5, 0.35):
    print(f"threshold {thr}: caught {caught(scores, truth, thr)} of the real positives")
# A fraud team lowers the threshold to avoid false negatives -- a missed fraud
# costs far more than reviewing one extra false alarm.

No secrets in client code: an API key in the browser is already public

One comfortable assumption remains to break: that code lives somewhere private, so a key written into app.js is hidden inside the application. Anything in frontend JavaScript ships to and runs in the user’s browser, which means any key, token, or secret in that code is downloadable by every visitor. It is not hidden in the code; it is published. Minification and bundling obscure nothing; the secret is one “view source” away.

Why is there no safe hiding place? The browser must download and execute the frontend’s JavaScript to run it, so the full source, including any string literal holding a key, is present on the client by definition. There is no client-side place to store a secret safely, because the same runtime that uses the key is fully inspectable: DevTools shows the source, the network tab shows every request header, and minifiers only rename variables and strip whitespace. They do not encrypt values, so a string scan over the shipped bundle recovers the literal. This block runs that scan against a snippet of “frontend” source to prove that “in the code” means “public.” Watch the key come straight back out.

import re

# A snippet of shipped frontend source -- exactly what every visitor downloads.
app_js: str = """
const API_URL = "https://api.example.com/predict";
const API_KEY = "sk_live_9f3a2c7e8b1d4f6a0e2c9b8d7f1a3e5c";
async function predict(record) {
  return fetch(API_URL, { method: "POST",
    headers: { "Authorization": "Bearer " + API_KEY },
    body: JSON.stringify(record) });
}
"""

# The same thing any visitor's DevTools does: scan the text for a key literal.
match = re.search(r"sk_live_[a-z0-9]+", app_js)
recovered: str = match.group(0) if match else "(no key found)"
print("recovered from shipped source:", recovered)
print("minification renames API_KEY but leaves this literal intact")

One regular expression recovers the key, and minification would not help: renaming the API_KEY variable leaves the literal string untouched in the bundle. The non-obvious trap that catches experienced engineers is build-time inlining. Modern frontend builds read environment variables at build time and substitute their values as literals into the bundle: the build replaces every process.env.API_KEY (or a VITE_/NEXT_PUBLIC_-prefixed variable) with the actual string while compiling the output that ships. The mental model “it is an environment variable, so it is server-side and safe” is exactly wrong here. The variable existed on the build machine, but its value was baked into the JavaScript that gets downloaded, so it is now a public string in the bundle no differently than if it had been typed as a literal.

Vite and Next.js created their prefix conventions precisely to mark which variables are intended to be public. The Vite documentation states that VITE_-prefixed variables “are bundled into your source code at build time” and warns they “should not contain sensitive information such as API keys.” Next.js inlines a NEXT_PUBLIC_-prefixed value “into the js bundle that is delivered to the client,” while non-prefixed variables stay in the Node.js environment and are not accessible to the browser. The leak happens when a real secret is given a public-exposed prefix, or read into client code at all. The diagram below shows why the boundary is physical, not a matter of how the variable is named.

rectangle "Browser (untrusted):\neverything here is downloaded\nand inspectable" as browser {
  [app.js + bundle] as bundle
  [DevTools / network tab] as devtools
}
rectangle "Server (trusted):\nthe only place a secret can live" as server {
  [proxy / backend-for-frontend] as proxy
  [API_KEY env var] as key
}
bundle --> proxy : HTTP request (no key)
proxy --> key : reads server-side
proxy --> bundle : response

The fix is architectural, not a hiding trick: the secret lives on a server the browser never sees inside, and the page either calls a backend-for-frontend that holds the key and adds it server-side, or the endpoint is gated by a per-user auth token issued and verified server-side. This block shows the correct shape, where the credential is read from a server-side environment variable the client never receives, and the browser’s request carries no key at all.

import os


def server_side_call(record: dict[str, float]) -> dict[str, str]:
    """Runs on the proxy, not in the browser. The key never leaves the server."""
    api_key: str | None = os.environ.get("API_KEY")  # server env, not bundled
    if api_key is None:
        return {
            "status": "no key configured -- request would not be sent with a secret"
        }
    return {"sent_header": f"Bearer {api_key[:6]}...", "key_in_browser": "no"}


# The browser's request to the proxy carries no credential at all.
browser_request: dict[str, object] = {"url": "/api/predict", "headers": {}, "key": None}
print("browser request headers:", browser_request["headers"])

# On the server, the key is read from the environment (unset here -> the safe path).
print(server_side_call({"loan_amnt": 10000.0}))

No credential rides along in the browser’s request; the key is read from the environment only on the server, where the client cannot reach it. The failure mode is the opening’s second incident: a key in app.js, shipped to every browser, that I found in my own source weeks after writing it. The code worked the entire time, which is exactly why it sat there. With a small team and no separate reviewer, the only thing between that key and the public was me happening to re-read the file. The fix cost a day of unplanned rotation work, all of it avoidable by never putting a secret in an artifact the client downloads.

# This is "frontend" code. The author believes the env var keeps the key off the client.
client_code: str = """
const API_KEY = import.meta.env.VITE_API_KEY;  // "this is an env var, so it's safe"
fetch(URL, { headers: { Authorization: "Bearer " + API_KEY } });
"""


def leaks(code: str) -> bool:
    """Return True if this code ships a credential to the browser."""
    # A VITE_-prefixed var is inlined at build time. Does its value ship?
    return False


print("leaks:", leaks(client_code))
# Rewrite: where must the Authorization header actually be added?

client_code: str = """
const API_KEY = import.meta.env.VITE_API_KEY;
fetch(URL, { headers: { Authorization: "Bearer " + API_KEY } });
"""


def leaks(code: str) -> bool:
    """Return True if this code ships a credential to the browser."""
    # VITE_-prefixed vars are inlined as literals at build time, so the value
    # ships in the bundle. Reading it into client code is the leak.
    return "VITE_API_KEY" in code and "Authorization" in code


print("leaks:", leaks(client_code))
# Correct shape: the browser calls a same-origin proxy with NO key; the proxy
# (server-side) reads API_KEY from its environment and adds the Authorization
# header before calling the model API. No credential exists in the bundle.
print("fix: browser -> /api/predict (no key) -> server adds key -> model API")

Summary

• A model’s probability is a relative, often uncalibrated score, not a real-world frequency; rendering a bare decimal invites a non-ML reader to invent a chance the number does not support, doubly so because they cannot see the base rate.
• Frame the score so the reader cannot misread it, as a likelihood band or rank, rather than handing over a decimal that reads as certainty against a prior of roughly one in five.
• The decision threshold that turns a score into a yes/no is a business choice about the cost of a false positive versus a false negative; 0.5 is the arithmetic midpoint and is almost always wrong on an imbalanced problem.
• The threshold belongs where the cost owner can change it, the server’s label by default, never hardcoded in client JS where a wrong value silently buries the model’s usefulness.
• Any secret in frontend code ships to every browser; build-time inlining means a VITE_/NEXT_PUBLIC_ env var leaks as completely as a literal. A credential must live on a server the browser never sees inside.

Check your understanding:

• A reader sees 0.73 on the page. What wrong mental model do they build, and what two independent facts about the model make that reading wrong?
• Why is a 0.5 threshold usually wrong on a class-imbalanced problem, and why is hardcoding any threshold in the page a bad place for a business decision to live?
• Without looking back: why does reading a key from an environment variable still leak it if the build inlines that variable, and where must a credential that protects the /predict endpoint actually live?