Alert on the Right Signal (and Stay Quiet Otherwise)

I once set up alerts the way the instinct tells you to: catch everything, set the threshold low so nothing slips past. I built a monitoring setup that detected everything and then alerted on everything. The drift detector from the last lesson fired on every PSI blip that crossed 0.2 and crossed back a minute later; the p99 rule fired on a single slow request. Each firing was a real measurement, so each one looked defensible. I owned that channel, and I did the only thing a human can do when a channel buzzes dozens of times a day for nothing: I muted it. Then a genuine sustained latency regression ran unacknowledged, because the alert it tripped landed in the channel I had already silenced. The customer reported the outage first.

That is the inversion this module has been building toward. Every lesson after the first opened on monitoring that reported “everything is fine” while something real was broken, the blind spot. Here the monitoring is not blind. It detected the failure. It still failed to make anyone act, because the true signal arrived in a channel its owner had learned to ignore. Detection without actionability is not a smaller win than detection-plus-action. It is zero. This lesson turns the signals from the earlier lessons, the structured log line and /health check, the p99 latency, the PSI drift score, into alerts that page a human exactly when action is needed, and, the harder half, stay quiet otherwise.

Three terms carry the lesson, and a reader who lands here mid-module needs them grounded. An alert is a rule that watches a signal (a metric, a drift score) and, when a condition holds, notifies a human: a page, a ticket, a mark on a dashboard. It is distinct from the signal itself. The metric is the measurement; the alert is the decision to interrupt someone over it. A page is the loudest notification tier: it interrupts a human now, on the assumption that something must be done immediately. The page is the scarce resource this whole lesson is about spending well. The signals being alerted on are the p99 (the latency 99% of requests come in under, from the metrics lesson) and the drift score (the PSI against a reference window, from the drift lesson).

Alert fatigue: a too-sensitive threshold is worse than no alert

A competent engineer setting up alerts reasons like this: an alert’s job is to catch incidents, so the more incidents it catches the better, so tune it sensitive. That reasoning treats detection rate as the thing to maximize. It is the wrong objective, and it fails in a specific, mechanical way. Watch what a sensitive rule does to a perfectly normal latency signal, one with no incident in it at all, only the ordinary sample-to-sample noise of a tail metric.

# A "catch everything" rule: page the instant a single sample crosses the line.
p99_samples = [180, 195, 210, 205, 198, 215, 188, 220, 192, 208]  # ms, all normal noise
THRESHOLD_MS = 200

for i, sample in enumerate(p99_samples):
    if sample > THRESHOLD_MS:
        print(f"PAGE at minute {i}: p99 = {sample}ms")
# Fires at minutes 2, 3, 5, 7, 9: five pages in ten minutes.
# Not one of these is an incident. The signal is only noisy around 200.

Five pages in ten minutes, and not one of them is an incident. The signal is jittering around the threshold, which is what real tail-latency signals do at the sample level. The rule is not wrong about any single measurement; each value above 200 is genuinely above 200. The rule is worthless, and a worthless alert does not sit there harmlessly. It actively trains the one person watching to ignore the channel.

The correct objective is the opposite of sensitivity. An alert’s value is not its detection rate; it is the probability that, when it fires, a human both can and should act. A rule tuned to miss nothing maximizes false positives, and every false positive (a firing with no incident behind it) spends a fixed amount of the responder’s trust. Spend enough and the channel is muted, at which point the rule’s true positives are worth nothing, because no one is reading them. On a small team this is sharper than the SRE literature assumes, because there is no on-call rotation to absorb the noise. The channel is one person. When that person mutes it, there is no second tier still listening.

This is the precision-versus-recall trade from classification, the one you met setting decision thresholds: lowering the alert threshold raises recall (catch more real incidents) and destroys precision (most firings are false). The non-obvious part is that for alerts the cost function is asymmetric and cumulative. A missed incident costs once. A false positive costs a little trust every time, and the costs sum until the channel dies. The optimum is therefore not the threshold that maximizes recall. It is the one that keeps precision high enough that the channel stays trusted, because a trusted channel with slightly lower recall still gets read, and a distrusted channel with perfect recall does not.

The scrolly below traces how a noisy alert destroys itself, step by step, and how a sustained window reverses it.

The two mechanical fixes for the two noise modes the scrolly shows are a sustained window and hysteresis. A sustained window, which fires only when the condition holds for N consecutive evaluations rather than for a single sample, filters on duration. Noise is uncorrelated sample-to-sample, so the probability it holds the bar for N evaluations in a row falls toward zero, while a genuine regression holds it every time. You are filtering on exactly the property that separates transient from real. The SRE book names this directly: an alert goes pending for a duration before it fires, to ensure it is not a transient state. Hysteresis fixes flapping with a dead-band: fire at a high value but only resolve after the signal drops below a separate, lower value, so a signal hovering near the high line cannot toggle state; it must travel the whole band to change. (The same dead-band a thermostat uses so it does not switch on and off every few seconds.)

Here is the same noisy series run through both rules, with the page count printed for each so the noise reduction is a number rather than an assertion.

def count_pages_instantaneous(samples: list[float], threshold: float) -> int:
    """Fire on every single sample above the threshold."""
    return sum(1 for s in samples if s > threshold)


def count_pages_sustained(samples: list[float], threshold: float, window: int) -> int:
    """Fire only when `window` consecutive samples stay above the threshold,
    then stay 'firing' until the streak breaks (one page per sustained run)."""
    pages = 0
    streak = 0
    firing = False
    for s in samples:
        if s > threshold:
            streak += 1
            if streak >= window and not firing:
                pages += 1  # page once when the run becomes sustained
                firing = True
        else:
            streak = 0
            firing = False  # the run ended; re-arm for the next one
    return pages


# A noisy normal stretch (jitter around 200) followed by a real sustained regression.
noise = [180, 210, 195, 220, 188, 215, 192, 208, 185, 205]
regression = [260, 265, 258, 270, 262, 268]  # genuinely degraded, holds high
p99 = noise + regression

THRESHOLD = 200
WINDOW = 4

inst = count_pages_instantaneous(p99, THRESHOLD)
sust = count_pages_sustained(p99, THRESHOLD, WINDOW)
print(f"instantaneous rule: {inst} pages")
print(f"sustained rule (window={WINDOW}): {sust} pages")
print(f"noise pages suppressed: {inst - sust}")

The instantaneous rule pages on every crossing in the noisy stretch and again on every sample of the real regression, a flood that buries the one event that matters. The sustained rule ignores the jitter entirely (no run of four consecutive crossings exists in the noise) and pages exactly once when the regression holds the bar. The same signal, the same threshold; the only change is that the rule filters on duration, and duration is what separates transient from real.

The window length is the one tunable that matters, and the standard advice to pick five minutes hides a real trade. The window is a response curve, not a constant.

The non-obvious cost lives in the moderate column: a sustained window does not catch incidents faster, it catches them later, by exactly the window length. You are buying precision with detection latency. A staff engineer reads that and stops treating the window as a noise filter alone; it is a noise-versus-latency dial, and the right setting depends on how much a minute of the failure actually costs.

The failure mode this section maps to is alert fatigue: a channel that fires on noise trains its one reader to ignore it, so the real signal lands in a silent room. The fix is not a better threshold value. It is a different rule shape, sustained plus hysteresis, that makes a firing mean “this held long enough to be real.” This is the rubric’s second bar in mechanical form: an alert that fires on the right signal because it filters out everything that is not one.

def count_pages_instantaneous(samples: list[float], threshold: float) -> int:
    return sum(1 for s in samples if s > threshold)


def count_pages_sustained(samples: list[float], threshold: float, window: int) -> int:
    pages = 0
    streak = 0
    firing = False
    # Fill in: increment streak on a crossing, page once when streak >= window
    # and not already firing, and reset streak + firing when the signal drops.
    return pages  # placeholder


psi = [0.18, 0.25, 0.19, 0.27, 0.21, 0.31, 0.33, 0.30, 0.34, 0.20]
THRESHOLD = 0.2
WINDOW = 3
print(count_pages_instantaneous(psi, THRESHOLD))
print(count_pages_sustained(psi, THRESHOLD, WINDOW))

def count_pages_instantaneous(samples: list[float], threshold: float) -> int:
    return sum(1 for s in samples if s > threshold)


def count_pages_sustained(samples: list[float], threshold: float, window: int) -> int:
    pages = 0
    streak = 0
    firing = False
    for s in samples:
        if s > threshold:
            streak += 1
            if streak >= window and not firing:
                pages += 1
                firing = True
        else:
            streak = 0
            firing = False
    return pages


psi = [0.18, 0.25, 0.19, 0.27, 0.21, 0.31, 0.33, 0.30, 0.34, 0.20]
THRESHOLD = 0.2
WINDOW = 3
inst = count_pages_instantaneous(psi, THRESHOLD)
sust = count_pages_sustained(psi, THRESHOLD, WINDOW)
print(f"instantaneous: {inst} pages")
print(f"sustained (window=3): {sust} pages")
print(f"suppressed: {inst - sust} false alarms")

Symptom vs cause: page on what a human can fix

Suppose the noise problem is solved: every alert now uses a sustained window and stops flapping. The remaining instinct is that any signal which holds long enough deserves a page, because it is real. That instinct rebuilds the fatigue of the last section from a new direction, because most signals that hold are causes, not symptoms. CPU stays high for ten minutes and then the autoscaler catches up. A replica restarts and the load balancer routes around it. One feature drifts slightly and the model absorbs it. Each is a sustained, real condition that never touches a user. Paging on it spends a human interrupt on a maybe.

The principle is that the right routing tier for a signal is set by one question: does this condition correlate with harm a user is experiencing right now? A symptom is a condition the user feels, such as error rate climbing, served p99 over the SLO, or prediction drift sustained past a material threshold. A cause is an internal condition that may or may not produce a symptom, such as high CPU, a restarted replica, or a single drifting feature. Symptom-level conditions correlate with actual harm, so interrupting a human over them is almost always justified. Cause-level conditions fire often, frequently self-heal, and may never reach anyone, so paging on them spends attention on a maybe. The SRE rule makes this concrete: page only on a condition that is undetected, urgent, actionable, and actively or imminently user-visible, four filters, and a cause-level metric typically fails at least one.

The classic version of this failure is the CPU alert. Watch a competent engineer’s reasoning break on a single line.

# "High CPU means the box is struggling, so page on it."
cpu_percent = 94          # sustained, real, holds for ten minutes
served_p99_ms = 180       # under SLO, users are fine
error_rate = 0.001        # nominal

if cpu_percent > 85:
    page("CPU high on scorer-3")   # fires
# A human is woken. They look. p99 is fine, errors are nominal, users are happy.
# The autoscaler adds a replica four minutes later and CPU drops to 60.
# The page was real, sustained, and useless: it pointed at a cause
# that never became a symptom.

The CPU reading was true and sustained, so a sustained window does not save you here. The window filtered transients; it cannot tell a cause from a symptom, because that is not a duration property; it is a meaning property. A high-CPU condition tells you nothing about whether users are impacted or whether you should act. The signal that does carry that information is the served p99 and the error rate, and both were nominal. The page interrupted a human to report a condition the system was already handling on its own.

The fix is to route signals by what they mean, not by whether they are real. The three routing tiers encode severity. Page wakes someone now. Ticket is a real condition that needs a human within hours, not now. Dashboard-only is context you read during an investigation and never get notified about. The same SRE book that names flapping prescribes exactly this split: page-worthy alerts go to the on-call responder, subcritical alerts go to a ticket queue, and everything else is retained as informational data on a status dashboard. The code below runs each candidate signal through the tribunal and prints the verdict with its one-sentence reason.

def route(user_visible: bool, urgent: bool, self_heals: bool) -> str:
    """Classify a signal into page / ticket / dashboard by the four SRE filters."""
    if user_visible and urgent and not self_heals:
        return "PAGE"
    if user_visible or (not self_heals and not urgent):
        return "TICKET"
    return "DASHBOARD"


candidates = [
    # (name, user_visible, urgent, self_heals)
    ("error rate climbing", True, True, False),
    ("served p99 over SLO (sustained)", True, True, False),
    ("CPU at 94%", False, True, True),
    ("one replica restarted", False, False, True),
    ("one feature drifting slowly", False, False, False),
    ("one slow request", True, False, True),
]

reasons = {
    "PAGE": "user-visible harm, urgent, will not self-heal",
    "TICKET": "real but not an immediate interrupt",
    "DASHBOARD": "internal cause, no user impact -- read during investigation",
}

for name, uv, urg, heals in candidates:
    tier = route(uv, urg, heals)
    print(f"{tier:10} {name:34} ({reasons[tier]})")

The two genuine symptoms, error rate and sustained p99, page. CPU at 94% and the restarted replica drop to dashboard because they self-heal and never become user-visible. The slowly drifting feature becomes a ticket: real, worth a human eventually, but not an interrupt. The single slow request is user-visible but transient, so it does not earn a page either. The verdict for most signals is “not a page,” and that is the point: the tribunal exists to keep the page scarce.

The non-obvious staff judgment is when to build the tiers at all. On day one, with one engineer and a handful of alerts a day, a single channel is correct. Splitting into page, ticket, and dashboard is not free, because every signal now needs a routing decision and a place to land, and premature tiering is its own complexity. You have earned the tiers when one channel forces you to mute to survive (the fatigue from the last section), or when a second engineer joins and “everything pings everyone” stops scaling. Below that line, tiers are overhead; above it, a single channel guarantees the symptom scrolls past buried in causes. The standard advice to “tier your alerts” is wrong on day one and right by the time you have a second responder; the skill is knowing which side of that line you are on.

The named failure here is burying the symptom in the causes. Once the team grew past one engineer, the single channel pinged on every replica restart, a cause that auto-healed in seconds. Meanwhile a sustained error-rate climb, the one signal that should have interrupted them, scrolled past in the same stream. Attention was spent on harmless causes while the real symptom went unnoticed. Splitting the stream, so symptoms interrupt and causes go to a dashboard checked on its own schedule, fixed both ends, and that split was the point at which the tiers had actually been earned.

def route(user_visible: bool, urgent: bool, self_heals: bool) -> str:
    if user_visible and urgent and not self_heals:
        return "PAGE"
    if user_visible or (not self_heals and not urgent):
        return "TICKET"
    return "DASHBOARD"


# Fill in (user_visible, urgent, self_heals) for each candidate:
candidates = [
    ("prediction error rate doubled", True, True, False),
    ("disk at 70% and climbing slowly", True, True, False),
    ("model latency p99 over SLO for 10 min", True, True, False),
    ("garbage-collection pause spiked once", True, True, False),
]
for name, uv, urg, heals in candidates:
    print(f"{route(uv, urg, heals):10} {name}")

def route(user_visible: bool, urgent: bool, self_heals: bool) -> str:
    if user_visible and urgent and not self_heals:
        return "PAGE"
    if user_visible or (not self_heals and not urgent):
        return "TICKET"
    return "DASHBOARD"


candidates = [
    # (name, user_visible, urgent, self_heals)
    ("prediction error rate doubled", True, True, False),
    ("disk at 70% and climbing slowly", False, False, False),
    ("model latency p99 over SLO for 10 min", True, True, False),
    ("garbage-collection pause spiked once", False, False, True),
]
for name, uv, urg, heals in candidates:
    print(f"{route(uv, urg, heals):10} {name}")

Tie the threshold to the budget: alert on burn rate

The threshold question keeps returning: what value of p99 or error rate is “too high”? The instinct is to pick a number, “page if error rate exceeds 1%.” That number is pulled from the air, and it restarts the noise problem from the other side. It has no relationship to whether users are being harmed enough to matter, so it is either too jumpy (a brief 1.2% spike pages, even though it harmed almost no one) or too lax (a steady 0.5% rate never pages, even though it is quietly destroying the reliability you promised). A hand-picked threshold misses both magnitude and duration. Here is the lax half failing.

# "Page when error rate is above 1%." Looks reasonable. Misses the slow burn.
error_rate = 0.005          # 0.5%, comfortably under the 1% threshold
THRESHOLD = 0.01

if error_rate > THRESHOLD:
    page("error rate high")
# else: silent, no page.
# But the promise to users was 99.9% success (a 0.1% budget over the week).
# A steady 0.5% rate is FIVE TIMES the allowed failure. It will blow the
# weekly budget by Friday. The threshold sleeps through it because 0.5 < 1.0.

The 0.5% rate is below the threshold, so the rule stays silent, and it is five times the failure the promise to users allowed. The threshold has no idea what was promised. It is a raw number compared against a raw number, with no notion of how much failure is acceptable over what window. A threshold means nothing until you have stated that.

The principle is that you define the allowed failure first, then alert on the rate at which you are spending it. A service-level objective (SLO) is a target on a measured indicator: “99.9% of requests succeed over a moving 30-day window.” The error budget is the complement, the failure the SLO permits, computed as $1 - \text{SLO}$: a 99.9% SLO allows 0.1% of requests to fail over the window. The judgment that anchors every alert in this lesson is that you do not page when the error rate crosses an arbitrary line. You page on the burn rate, how fast you are consuming the budget relative to the window, because that is the quantity that actually predicts whether you will breach the objective users were promised.

Burn rate folds magnitude and duration into one number, which is exactly the transient-versus-sustained distinction the first section reached for with windows, now expressed as a rate against a budget. A 5% error rate that lasts thirty seconds barely dents a 30-day budget. The same rate sustained for a day will blow it. The burn rate is the multiple of the budget you would consume if the current rate continued for the whole window:

$$ \text{burn rate} = \frac{\text{observed failure rate}}{\text{budget}} = \frac{\text{observed failure rate}}{1 - \text{SLO}} $$

A burn rate of 1 means you are spending the budget exactly on pace to exhaust it at the end of the window. A burn rate of 10 means you will exhaust it in a tenth of the window: three days into a thirty-day budget, not the full month. The process that produces a defensible alert has four steps, and skipping any one of them removes the anchor.

The code below derives a budget from an SLO and runs two error streams through a burn-rate check: a brief spike and a sustained low rate. The point is that the spike spends negligible budget and does not page, while the sustained low rate is on track to breach and does, the opposite of what the hand-picked threshold did.

def budget(slo: float) -> float:
    """Allowed failure fraction = 1 - SLO."""
    return 1.0 - slo


def burn_rate(observed_failure_rate: float, slo: float) -> float:
    """How many budgets per window the current rate would consume."""
    return observed_failure_rate / budget(slo)


def decide(observed: float, slo: float, page_at: float, ticket_at: float) -> str:
    rate = burn_rate(observed, slo)
    if rate >= page_at:
        return f"PAGE  (burn {rate:.1f}x -- exhausts budget fast)"
    if rate >= ticket_at:
        return f"TICKET (burn {rate:.1f}x -- slow but will breach)"
    return f"none  (burn {rate:.1f}x -- negligible)"


SLO = 0.999  # 99.9% success promised over the window
print(f"budget = {budget(SLO):.4f} ({budget(SLO) * 100:.1f}% allowed to fail)")

# A brief spike: 5% errors, but only for a sliver of the window.
spike_window_fraction = 0.0003  # ~30 seconds of a 30-day window
spike_effective_rate = 0.05 * spike_window_fraction
print("brief 5% spike:", decide(spike_effective_rate, SLO, page_at=10, ticket_at=2))

# A sustained low rate: 0.5%, under any hand-picked 1% threshold, all window long.
print("steady 0.5% rate:", decide(0.005, SLO, page_at=10, ticket_at=2))

The brief spike spends a burn rate near zero (five percent of a thirty-second sliver of a thirty-day window is nothing against the budget) so it does not alert at all, which is correct, because almost no users were harmed. The steady 0.5% rate burns at five times the budget pace and tickets (above the 2x ticket bar, below the 10x page bar), which the 1% threshold never did. The same arithmetic that silences the noise the first section fought also catches the slow failure a raw threshold sleeps through. The threshold finally means something a human can defend: it fires when the failure is consuming the promised reliability fast enough to matter.

The fast-burn-page versus slow-burn-ticket split is the symptom-versus-cause tiering from the last section, applied to the rate instead of the raw signal. A burn rate of 10 empties a thirty-day budget in about three days, fast enough to page. A burn rate of 2 takes about two weeks, so it is a ticket: real, worth fixing, but not tonight. The non-obvious cost is that burn-rate alerting requires you to have committed to an SLO at all, which is a product and reliability decision, not a monitoring one. A team that has never stated “how much failure is acceptable” cannot compute a budget, so it falls back to hand-picked thresholds and the noise that comes with them. The hardest part of burn-rate alerting is not the arithmetic; it is getting the organization to name the number in step 2.

For an ML scorer the indicator does not have to be HTTP success. It can be a correctness proxy: the served p99, the sustained-drift fraction, or a canary score staying in band. The burn-rate machinery is identical; only the SLI changes. That is the bridge to the next lesson, where the SLI becomes “is the model right,” not only “is the box up.” The whole module turns on the distinction that a service returning 200 OK can be quietly wrong, and an SLO on a correctness proxy is how you put a budget on wrong rather than only on down.

def budget(slo: float) -> float:
    return 1.0 - slo  # allowed failure fraction


def burn_rate(observed_failure_rate: float, slo: float) -> float:
    # Fill in: burn rate is the observed rate as a multiple of the budget.
    return 0.0  # placeholder


outcomes = ["ok"] * 970 + ["fail"] * 30  # 1000 requests, 30 failures
SLO = 0.99
observed = outcomes.count("fail") / len(outcomes)
rate = burn_rate(observed, SLO)
print("observed failure rate:", observed)
print("burn rate:", rate)
# Decide: PAGE if rate >= 10, TICKET if rate >= 2, else no action.

def budget(slo: float) -> float:
    return 1.0 - slo


def burn_rate(observed_failure_rate: float, slo: float) -> float:
    return observed_failure_rate / budget(slo)


def decide(rate: float, page_at: float, ticket_at: float) -> str:
    if rate >= page_at:
        return "PAGE"
    if rate >= ticket_at:
        return "TICKET"
    return "no action"


outcomes = ["ok"] * 970 + ["fail"] * 30  # 1000 requests, 30 failures
SLO = 0.99
observed = outcomes.count("fail") / len(outcomes)
rate = burn_rate(observed, SLO)
print(f"budget: {budget(SLO):.3f} (1% allowed to fail)")
print(f"observed failure rate: {observed:.3f}")
print(f"burn rate: {rate:.1f}x")
print(f"decision: {decide(rate, page_at=10, ticket_at=2)}")

Summary

• A too-sensitive alert is worse than no alert, because each false positive spends a fixed amount of the responder’s trust and the costs are cumulative: spend enough and the channel is muted with the real signal inside it. Tune for actionability, not detection rate.
• The two noise modes have two mechanical fixes: a sustained window (fire only when the condition holds for N consecutive evaluations) filters transients by filtering on duration; hysteresis (a dead-band between the fire and resolve thresholds) stops a signal near the line from flapping.
• The window length is a noise-versus-latency dial, not a constant: a longer window filters more noise but means a real incident is older by exactly the window length before it pages.
• Page on symptoms (conditions the user feels) and route causes (internal conditions that self-heal and may never reach a user) to a dashboard. Most real signals are causes, so the verdict for most signals is “not a page.” Build the page/ticket/dashboard tiers only once a single channel forces muting or a second responder joins.
• A hand-picked threshold misses both magnitude and duration. Anchor it to an error budget ($1 - \text{SLO}$) and alert on burn rate (the multiple of the budget the current failure rate would consume over the window) so a brief spike spends negligible budget (no page) while a slow steady burn pages before it breaches. The SLI can be a correctness proxy, not only HTTP success, the bridge to the next lesson.

Check your understanding:

• Why is a too-sensitive alert worse than no alert, and what two rule-shape changes turn a noisy signal into an actionable page?
• What distinguishes a symptom from a cause, and why should most detected problems not page? When have you earned the page/ticket/dashboard tiers?
• Without looking back: what is an error budget, and why does a burn-rate alert catch a steady 0.5% failure rate that a hand-picked 1% threshold sleeps through, while also ignoring a brief 5% spike?