Make It Measurable: Metrics, Traces, and a Dashboard

The working monitoring from the last lesson said everything was fine. Every log line was a clean 200, the health check was green, the average latency across the day looked fast. Then a handful of users reported that the scorer was “sometimes slow.” The logs were no help: each one looked instant on its own, and the mean latency over the day’s requests had not moved at all. The problem only surfaced when I stopped looking at single events and looked at the distribution across thousands of them, and then at where inside one slow request the time actually went.

The structured logs and the health check from the last lesson answer one question well: what happened to this one request. That is a per-event view, and the failures that page you do not live in a single event. They live in two places a log line cannot show. The first is the distribution across many requests, the rates and latencies that a dashboard, not a log, is built to hold. The second is the path within one request, where, across feature lookup, model call, and downstream, the time disappeared. This lesson hardens both: the right metric type for a quantity, the percentile that describes the worst-served user instead of the typical one, the trace that localizes a slow stage, the one label that quietly takes down the metrics store, and the dashboard that assembles the survivors into a question list.

Why the average latency lies and the p99 does not

The dashboard reports a mean latency of 20 milliseconds for the scorer. The question that looks settled: with a 20-millisecond average, is anyone being served slowly? The obvious answer is no, since 20 milliseconds is fast, and a fast average means a fast service.

The obvious answer is wrong, and the way it is wrong is the whole point of this section. The same batch of requests that averages 20 milliseconds can time out for one user in a hundred, and the mean never moves when it happens. The number anyone would put on a status page is the number that hides the failure.

The mechanism is structural, not a quirk of one dataset. A latency average answers “what does a typical request feel like,” and that is the wrong question for monitoring, because nobody is paged about the typical request; they are paged about the slow slice. The number on the dashboard has to describe the experience of the worst-served requests, because that experience is the one that becomes a timeout, a retry storm, or a user filing a report. Choosing which statistic represents “the latency” is therefore a decision about whose experience the dashboard is willing to be blind to.

Latency distributions are right-skewed: a dense cluster of fast requests with a long thin tail of slow ones, where the tail comes from transient contention or cold paths such as a garbage-collection pause, a cold cache, a contended lock, or a degraded downstream. The mean is the sum over the count, so a handful of large values barely move it while the majority stay small. It tracks the common case and is mathematically incapable of surfacing the tail. A percentile does the opposite operation: it does not average, it orders. The p99 is the value that 99 percent of requests come in under: sort the latencies and read the value at rank $\lceil 0.99 \times N \rceil$. Because it is a position near the top of the sorted order, it tracks the tail directly and is untouched by how fast the fast requests were.

$$ p99 = x_{(k)}, \quad k = \lceil 0.99 \times N \rceil $$

Here $x_{(k)}$ is the $k$-th value once the $N$ latencies are sorted ascending. The mean collapses the ordering into one number by summing; the percentile keeps the ordering and reads a rank. This is also why a percentile cannot be recovered from a mean after the fact: both have already thrown the ordering away, but the mean threw away more, which is the bridge to the next section on metric types.

The code below builds a distribution that looks healthy by its mean and is broken at the tail. Watch the gap between the mean and the p99, and watch how many requests cross a caller’s 200-millisecond timeout while the mean stays well under it.

import numpy as np


def summarize_latency(latencies_ms: np.ndarray, timeout_ms: float) -> dict[str, float]:
    ordered = np.sort(latencies_ms)
    n = len(ordered)
    p50_rank = int(np.ceil(0.50 * n)) - 1
    p99_rank = int(np.ceil(0.99 * n)) - 1
    timed_out = int((ordered > timeout_ms).sum())
    return {
        "mean": float(ordered.mean()),
        "p50": float(ordered[p50_rank]),
        "p99": float(ordered[p99_rank]),
        "max": float(ordered[-1]),
        "timed_out": timed_out,
        "timed_out_pct": 100.0 * timed_out / n,
    }


rng = np.random.default_rng(0)
fast = rng.normal(18.0, 3.0, size=9_900)  # the common case: ~18 ms
slow = rng.normal(450.0, 80.0, size=100)  # 1 in 100: a degraded tail
latencies = np.clip(np.concatenate([fast, slow]), 1.0, None)

stats = summarize_latency(latencies, timeout_ms=200.0)
for name, value in stats.items():
    print(f"{name:>14}: {value:.2f}")

The mean lands near 22 milliseconds, the number that goes on the status page, while the max shows a request near 590 and one percent of requests blow through the 200-millisecond timeout. The mean moved by only a few milliseconds when one in a hundred requests started taking more than twenty times longer, because ninety-nine hundredths of the sum is still the fast cluster. A dashboard watching the mean draws a flat line straight through this incident. There is a sharp subtlety in the p99 here worth pausing on: with the tail at exactly one percent of requests, the p99 reads about 28 milliseconds — just under where the slow cluster begins — not 400, because the 99th-percentile rank lands on the last fast request and the slow ones sit at the 99.1st percentile and above. The p99 caught that something was off (28 ms is well above the 18 ms median) but did not reveal how bad the tail actually is; the max and a p999 do. The lesson is that the percentile you watch has to sit past the fraction of requests that are slow: a one-percent tail hides from p99 and only shows in p999 or max.

The scrolly below shows why the gap opens. The distribution is one tight spike at low load; as utilization climbs and bursts arrive, a queue forms and the p99 lifts off the p50 while the median request still finds the server free.

Which percentile to watch is a tunable, so the rule is a response curve, not one default. Watch the median when you want the typical-request baseline or are comparing it against the p99 to gauge how heavy the tail is, since a widening p50-to-p99 gap is the tail growing. Watch the p99 when you want the worst-served one percent, because that is the experience that pages you. Watch the max only when an SLA has a hard ceiling or you are hunting a single pathological request, because one outlier dominates it entirely and a single thirty-second pause makes the max useless as a trend.

The signal that tells you which way to look: alert on the p99, diagnose with the p50 and the p99 together, never the p99 alone. At volume the tail is not a rounding error. On a high-traffic inference service the p99 describes tens of thousands of real requests a day, not a handful; the slow slice is a population, not noise. Which signal pages you, versus sits on the dashboard for context, is decided two lessons from now.

def report_latency(latencies_ms: list[float]) -> float:
    ordered = sorted(latencies_ms)
    # currently returns the mean -- replace with the p99
    return sum(ordered) / len(ordered)


data = [20.0] * 95 + [800.0] * 5
print(round(report_latency(data), 2))  # predict this, then make it the p99

import math


def report_latency(latencies_ms: list[float]) -> float:
    ordered = sorted(latencies_ms)
    n = len(ordered)
    rank = math.ceil(0.99 * n) - 1
    return ordered[rank]


data = [20.0] * 95 + [800.0] * 5
print("mean:", round(sum(data) / len(data), 2))
print("p99: ", round(report_latency(data), 2))

Counter vs gauge vs histogram: the type decides which question you can still answer

Every metric you record forces one decision before you know which question the incident will ask: is this quantity a running total, a point-in-time level, or a distribution? The decision feels like a formatting choice, three ways to write down the same number.

It is not a formatting choice, and treating it as one is how the data you need during an incident gets deleted before the incident starts. A collector reads each metric on a schedule, a scrape, and whatever happened between two scrapes survives only in the form the metric type preserved. The type is therefore a decision about which future questions stay answerable. Storing latency as a single current value, as a running total, or as a bucketed distribution are three different commitments, and only one of them lets you ask for a percentile after the fact. You are not choosing a style; you are choosing what to throw away.

A counter is a cumulative value that only ever increases, or resets to zero on restart: total requests, total errors. Its instantaneous value is meaningless; you read its rate, the difference between two scrapes divided by the elapsed time. The subtlety that makes counters survivable is monotonicity: when a redeploy resets a counter to zero, the collector sees the new value is lower than the last scrape and treats it as a reset rather than computing a giant negative rate. Use a counter for a quantity that legitimately goes down and that same reset logic corrupts every rate it computes.

A gauge is a single value that goes up and down: in-flight requests, queue depth, loaded-model memory. It stores only the last value written, so it has no memory between scrapes: a spike that rises and falls inside one scrape interval leaves no record it ever happened. A gauge also cannot safely hold a cumulative total, because a restart sets it to zero and the collector cannot distinguish that reset from a real drop, so the history is silently wrong from then on.

A histogram is for a quantity you will want to slice into percentiles later, such as request latency or payload size. It keeps a set of bucket counters: one monotonic counter per latency range, plus a total count and a running sum, exposed as the _bucket, _count, and _sum series. Percentiles are reconstructed after the fact by walking the buckets, and because the buckets are themselves counters, percentiles even aggregate across instances by summing matching buckets, something you cannot do by averaging averages.

The reconstruction below makes the loss concrete. Each object is handed the same stream of latencies; only the histogram can answer for a percentile afterward, because only it kept enough of the ordering. Watch the gauge return whatever it saw last and the counter return a rate but no distribution.

import math


class Counter:
    def __init__(self) -> None:
        self.total = 0

    def observe(self, _value: float) -> None:
        self.total += 1  # counts events; the value itself is discarded

    def rate(self, seconds: float) -> float:
        return self.total / seconds


class Gauge:
    def __init__(self) -> None:
        self.last = 0.0

    def observe(self, value: float) -> None:
        self.last = value  # only the most recent write survives

    def current(self) -> float:
        return self.last


class Histogram:
    def __init__(self, bounds: list[float]) -> None:
        self.bounds = bounds
        self.buckets = [0] * (len(bounds) + 1)
        self.count = 0

    def observe(self, value: float) -> None:
        self.count += 1
        for i, upper in enumerate(self.bounds):
            if value <= upper:
                self.buckets[i] += 1
                return
        self.buckets[-1] += 1

    def quantile(self, q: float) -> str:
        target = math.ceil(q * self.count)
        seen = 0
        lower = 0.0
        for i, upper in enumerate(self.bounds):
            seen += self.buckets[i]
            if seen >= target:
                return f"between {lower:.0f} and {upper:.0f} ms"
            lower = upper
        return f"above {self.bounds[-1]:.0f} ms (overflow bucket)"


stream = [18.0, 21.0, 19.0, 17.0, 480.0, 20.0, 22.0, 19.0, 18.0, 510.0]
counter, gauge = Counter(), Gauge()
hist = Histogram(bounds=[25.0, 50.0, 100.0, 250.0, 500.0])
for v in stream:
    counter.observe(v)
    gauge.observe(v)
    hist.observe(v)

print("counter rate (req/s over 2s):", counter.rate(2.0))
print("gauge current value         :", gauge.current())
print("histogram p90               :", hist.quantile(0.90))

The counter recovered a rate and nothing about the shape. The gauge returned 510, the last value it happened to see, and could not tell you a percentile, a rate, or even that two slow requests existed. Only the histogram located the p90 in a bucket. Store latency as a gauge and the percentiles from the previous section are gone forever; there is no later query that brings them back.

The histogram’s precision is capped by its bucket boundaries, and that is the one cost that bites in production. A percentile is reported as “somewhere in the bucket it falls into,” so if every slow request lands in one wide top bucket (a single >500ms bucket) the reported p99 is somewhere above 500, and you cannot tell 1.1 seconds from 10 seconds. The Prometheus documentation makes this concrete: with a coarse bucket layout, an estimated p95 can come back tens of milliseconds off the true value, because the error is bounded by the width of the bucket the quantile lands in — a too-wide tail bucket blurs exactly the number you most need to read. Bucket layout is a design decision, not a default, and the failure mode is a calm-looking p99 that is precise only to within an order of magnitude.

The per-bucket cost has a second edge worth previewing here, because every bucket boundary is its own stored series. Watch one histogram on a handful of endpoints expand into dozens of series before a single extra label is added, the on-ramp to the cardinality section two sections down.

# The cardinality trap, previewed: each histogram bucket is its own series.
# A 12-bucket histogram on a metric with 5 endpoints is 60 series before you
# add a single other label, the on-ramp to the cardinality section.
http_request_duration_seconds_bucket{endpoint="/score", le="0.025"}
http_request_duration_seconds_bucket{endpoint="/score", le="0.05"}
# ... one series per bucket per endpoint ...

A counter recovers a rate but not a distribution; a gauge recovers neither, only “what was it at the last scrape”; a histogram recovers a distribution, and therefore any percentile, at the cost of one counter per bucket. Choosing a gauge for an inherently cumulative or distributional quantity does not look slightly off; it permanently deletes the information the next incident will need. The per-bucket cost is also the first hint of the cardinality problem two sections from now.

class Counter:
    def __init__(self) -> None:
        self.total = 0

    def observe(self, value: int) -> None:
        self.total += value  # accumulates -- wrong for a level

    def read(self) -> int:
        return self.total


class Gauge:
    def __init__(self) -> None:
        self.last = 0

    def observe(self, value: int) -> None:
        return  # placeholder -- make this store the level

    def read(self) -> int:
        return self.last


depth_readings = [3, 5, 2, 7, 4]  # queue depth at five scrapes
metric: Counter | Gauge = (
    Counter()
)  # wrong type -- replace with Gauge and finish observe()
for d in depth_readings:
    metric.observe(d)
print("reported queue depth:", metric.read())

class Gauge:
    def __init__(self) -> None:
        self.last = 0

    def observe(self, value: int) -> None:
        self.last = value  # a level: overwrite with the most recent reading

    def read(self) -> int:
        return self.last


depth_readings = [3, 5, 2, 7, 4]
metric = Gauge()
for d in depth_readings:
    metric.observe(d)
print("reported queue depth:", metric.read())

One trace to follow a slow request through the system

The scorer’s p99 spiked. The metrics from the last two sections tell you that it is slow and roughly how slow across the fleet. They cannot tell you where inside one request the time went. The natural hypothesis is “the model got slow,” and that hypothesis costs you hours of profiling a model that turns out to be fine.

A metric is an aggregate by construction, which is precisely why it cannot answer the where. It tells you the fleet’s p99 doubled at 2pm. It cannot tell you, for one slow request, that the feature lookup took 400 milliseconds while the model took 8, because every component logs independently, with its own clock and no shared key, so there is no way to reassemble one request’s per-stage timing from the pile of logs. A trace fixes this with metadata minted at the entry point and carried forward. Each unit of local work becomes a span that records its start time, end time, and a pointer to its parent. Every span in one request shares a trace id; each span carries its own span id and a parent span id that names the span it ran inside. The collector needs no global clock and no ordering guarantee: it groups spans by trace id and links each to its parent by the parent pointer, rebuilding a tree from child-to-parent edges the same way any tree is reconstructed from its edges.

That tree turns the slow stage from a guess into a subtree you can see. The diagram below is the trace for one scorer request: the request span is the root, and the feature-lookup span nested inside it holds almost all of the elapsed time while the model span is tiny.

[request 450ms] as req
[feature lookup 400ms] as feat
[model predict 8ms] as model
[response encode 2ms] as resp
req --> feat : child span
req --> model : child span
req --> resp : child span
note bottom of feat : the slow subtree,\nnot the model

Read top-down, the trace says the model was never the problem. The feature lookup owned the request. The hypothesis “the model got slow” was wrong, and the trace falsifies it in one read instead of after an afternoon of profiling. This maps directly onto the request-to-handler-to-response path from the deployment module: each arrow in that path is a span boundary, and the trace is what makes the boundaries timeable.

The code reconstructs the tree from a flat list of spans, the exact job the collector does, and finds the slowest stage by walking parent pointers. The spans arrive in no particular order and with no shared clock; only the ids relate them. Watch the reconstruction find the slow child without ever being told the order the spans were emitted in.

from dataclasses import dataclass


@dataclass
class Span:
    span_id: str
    parent_id: str | None
    name: str
    duration_ms: float


# Emitted out of order, no shared clock -- only the ids relate them.
spans = [
    Span("c", "a", "model_predict", 8.0),
    Span("a", None, "request", 450.0),
    Span("d", "a", "response_encode", 2.0),
    Span("b", "a", "feature_lookup", 400.0),
]


def slowest_child(spans: list[Span]) -> tuple[str, float]:
    root = next(s for s in spans if s.parent_id is None)
    children = [s for s in spans if s.parent_id == root.span_id]
    worst = max(children, key=lambda s: s.duration_ms)
    return worst.name, worst.duration_ms


name, ms = slowest_child(spans)
print(f"root holds {len(spans) - 1} child spans")
print(f"slowest stage: {name} at {ms} ms")

The reconstruction names feature_lookup at 400 milliseconds without any timestamp ordering: the parent pointers alone were enough to build the tree and find the subtree that owned the request. That is the structural difference from metrics: a metric averaged the stages away, a trace kept the edges, so the where survives.

Propagation is the fragile part and the reason traces break in practice. The trace id travels in band, as a request header on each outgoing call (the W3C traceparent header) or as an explicit argument threaded down the call chain. Any boundary that fails to forward it starts a new trace id: a library that strips headers, a queue that drops metadata, a thread or async hop that loses context. Every span past that boundary is filed under a different trace, the original tree ends there, and the request looks like it finished at the boundary, so the time spent downstream becomes unattributable. A broken trace most often comes from two sides using different propagation formats, or one hop simply not forwarding context.

How much you keep is a genuine choice, so sampling is a tunable curve. Head-based sampling at a low rate decides at trace start to keep, say, one in a hundred: cheap and low-overhead, but the decision is made before the request reveals whether it was interesting, so the one slow request you actually need was probably not sampled. Head-based at a high rate keeps most or all traces, which is affordable only at low volume, because storage and per-request metadata scale with the kept fraction and tracing every request at full volume is the cost you cannot carry at scale. Tail-based sampling waits until the trace finishes, then keeps it if it was slow or errored, at the cost of buffering every in-flight trace in memory until it completes, which is more moving parts but means the trace you want is the trace you keep.

The signal: if the question is “why is this request slow,” uniform low-rate sampling will usually have thrown it away, so sample by tail latency so the slow ones survive. The trace localized the where; the next section is about the one piece of trace detail that must never migrate onto a metric.

from dataclasses import dataclass


@dataclass
class Span:
    span_id: str
    parent_id: str | None
    name: str
    duration_ms: float


spans = [
    Span("y", "w", "model_predict", 12.0),
    Span("w", None, "request", 310.0),
    Span("x", "w", "downstream_call", 270.0),
]


def diagnose(spans: list[Span]) -> tuple[float, str]:
    root = None  # find the span whose parent_id is None
    children: list[Span] = []  # collect spans whose parent_id == root's id
    # return (root.duration_ms, slowest child's name)
    return 0.0, "unknown"


total, stage = diagnose(spans)
print(f"request held {total} ms; investigate {stage} first")

from dataclasses import dataclass


@dataclass
class Span:
    span_id: str
    parent_id: str | None
    name: str
    duration_ms: float


spans = [
    Span("y", "w", "model_predict", 12.0),
    Span("w", None, "request", 310.0),
    Span("x", "w", "downstream_call", 270.0),
]


def diagnose(spans: list[Span]) -> tuple[float, str]:
    root = next(s for s in spans if s.parent_id is None)
    children = [s for s in spans if s.parent_id == root.span_id]
    worst = max(children, key=lambda s: s.duration_ms)
    return root.duration_ms, worst.name


total, stage = diagnose(spans)
print(f"request held {total} ms; investigate {stage} first")

Labels and cardinality: the metric that quietly takes down the metrics store

Adding a dimension to a metric, a label like status or endpoint, lets you slice it to ask sharper questions, such as the error rate per endpoint instead of across everything. More labels feel like more insight, and for bounded labels they genuinely are: a status label with five values and an endpoint label with tens are rich and almost free.

The illusion is that this generalizes, that if a few labels help, a high-resolution label like request_id or user_id would help more. It does the opposite, and the damage is delayed and shared. A metric in a dimensional store is not one number; it is one time series per unique combination of label values. The series http_requests_total{endpoint="/score", status="200"} and the same metric with status="500" are two distinct series, each with its own retained history. The total series count is the product of every label’s distinct-value count, so cardinality is multiplicative: two labels with 5 and 10 values is 50 series, and adding a third with 1,000 values is 50,000. The store keeps every active series resident in memory to answer queries fast, so cost scales with cardinality, not with traffic, and a low-traffic service can still detonate the store with one bad label.

The code makes the multiplication concrete. It counts the series a metric mints under a bounded label set, then under one unbounded label, on the same request volume. Watch the series count decouple from the request count entirely.

def series_count(label_value_sets: dict[str, int]) -> int:
    total = 1
    for distinct_values in label_value_sets.values():
        total *= distinct_values  # multiplicative, not additive
    return total


requests_served = 1_000_000

bounded = {"endpoint": 10, "status": 5, "model_version": 3}
unbounded = {"endpoint": 10, "status": 5, "request_id": 1_000_000}

print("requests served       :", requests_served)
print("bounded label series  :", series_count(bounded))
print("unbounded label series:", series_count(unbounded))
print("blow-up factor        :", series_count(unbounded) // series_count(bounded))

The bounded label set is 150 series and stays 150 no matter how much traffic arrives. The instant request_id becomes a label, the series count is fifty million (a new series minted on every distinct request) and the store’s memory climbs with the request id count, not with anything that would alert someone to slow down. The blow-up factor is in the hundreds of thousands against the bounded baseline.

This is the structural line between logs and metrics, not advice. A bounded label whose value set is small and fixed (status has around five values, endpoint has tens) is fine forever. An unbounded label is a slow-motion outage: putting request_id on a metric mints a brand-new series on every single request, the series count grows without limit, and the store’s memory climbs until it runs out and the process is killed. Because metrics backends are usually shared infrastructure across teams, that out-of-memory kill blinds every team at once, during the one window nobody can afford to be blind. The named failure shape is a calm dashboard, no traffic spike, memory creeping up for hours, and then every service’s monitoring going dark together, the root cause a single user_id label someone added last week to “slice errors per user.”

The per-request detail that tempted the label, the request id and the exact inputs, belongs in logs and traces, which hold one record per event: the structured logs from the last lesson and the spans from the previous section. Metrics aggregate; their labels must be low-cardinality by construction. Reaching for a metric label to capture per-request identity is using the aggregating tool for the job the high-resolution tool exists to do.

The number of labels is the tunable, and both extremes break. No labels at all means a single global counter you cannot slice, so “error rate on /score specifically” is unanswerable and you are blind to which part of the service is failing. A few bounded labels is the target: status, endpoint, model_version, rich and cheap, with essentially no failure mode as long as each value set stays bounded as the service grows. An unbounded label is the third extreme and is never correct on a metric.

The signal: before adding a label, ask how many distinct values it can take, ever. If the answer is unbounded or unknown, it is not a label; it is a log field. With the labels disciplined, the surviving low-cardinality metrics are ready to be assembled into a dashboard.

def series_count(label_value_sets: dict[str, int]) -> int:
    total = 1
    for distinct_values in label_value_sets.values():
        return distinct_values  # placeholder -- make this multiply all of them
    return total


safe = {"status": 5, "endpoint": 12}
risky = {"status": 5, "endpoint": 12, "loan_id": 1_000_000}

print("safe series :", series_count(safe))
print("risky series:", series_count(risky))

def series_count(label_value_sets: dict[str, int]) -> int:
    total = 1
    for distinct_values in label_value_sets.values():
        total *= distinct_values
    return total


safe = {"status": 5, "endpoint": 12}
risky = {"status": 5, "endpoint": 12, "loan_id": 1_000_000}

print("safe series :", series_count(safe))
print("risky series:", series_count(risky))
print("loan_id must move to the log line: it is an unbounded label")

A dashboard is a question list, not a wall of graphs

The metrics and traces from the last four sections now exist. The remaining step is the view, the dashboard a human reads during an incident. The instinct is to put every metric the service emits onto it, on the theory that more visibility is better visibility.

Completeness is the trap, and it is the one that costs you the incident. A dashboard that shows everything shows nothing, because the eye cannot find the one bad signal among forty flat ones, so the responder learns the service is down from a user instead of from the dashboard built to tell them. A dashboard is not a place to display every metric you happen to emit; it is the answer surface for one question, read under pressure: is the service healthy for users right now. The discipline is to start from the questions a responder asks in the first thirty seconds of an incident and put exactly those signals at the top, in the order they get asked.

The canonical starting set is the four golden signals (latency, traffic, errors, saturation) because if you can measure only four things about a user-facing system, those four describe its health. Latency is the time to service a request, reported with successful and failed requests separated, because a failed request can return misleadingly fast and a fast-failing 500 storm would otherwise improve the latency panel while the service burns. Traffic is the demand on the system. Errors is the rate of failed requests. Saturation is how full the most-constrained resource is. Everything richer, such as per-feature drift or per-stage trace breakdowns, is a drill-down you open after one of the four tells you where to look.

This section composes the previous four, and the dashboard is where those decisions become visible or wasted. The latency panel must show the p99, not the mean, or it inherits the opening incident’s blindness. Each signal needs the right metric type: errors as a counter rate, in-flight requests as a gauge, latency as a histogram so the p99 is reconstructable. Every slice on the dashboard uses only bounded labels, or the dashboard itself is what takes the store down. The code lays the four signals out as a priority-ordered list and flags the panels a responder reads first.

from dataclasses import dataclass


@dataclass
class Panel:
    signal: str
    metric_type: str
    priority: int  # 1 = symptom (user feels it), 2 = context (why)


dashboard = [
    Panel("error_rate", "counter (rate)", 1),
    Panel("latency_p99", "histogram", 1),
    Panel("traffic_rps", "counter (rate)", 2),
    Panel("saturation", "gauge", 2),
]

ordered = sorted(dashboard, key=lambda p: (p.priority, p.signal))
print("dashboard, top to bottom:")
for p in ordered:
    tier = "SYMPTOM" if p.priority == 1 else "context"
    print(f"  [{tier:>7}] {p.signal:<12} via {p.metric_type}")

The two symptom panels, error rate and p99 latency, sort to the top where the eye lands first; traffic and saturation sit below as context for why a symptom is happening. Each panel also names its metric type, which is the metric-type decision made visible: the p99 panel is a histogram because a gauge could never reconstruct it.

Lay the dashboard out top-to-bottom by what a degradation means: the symptom signals a user feels at the top, the demand and capacity signals below as the explanation. Two failure modes bound the design. The too-sparse failure is a dashboard missing one golden signal, blind to a whole class of failure, and the common version is a missing saturation panel, so the resource exhaustion that is about to cause the latency spike goes unseen until the latency spike arrives with no warning. The too-dense failure buries the error-rate panel under thirty cause-level graphs, so the responder scrolls past the one panel that mattered while the page is burning.

Which of these signals should page you, versus sit on the dashboard for context, is the next lesson’s decision. And the failure where every golden signal is green while the model is quietly wrong, a 200 OK on a prediction that is garbage, is the spine of the last lesson in this module. This dashboard can show that the service is up; it cannot yet show that the model is right, and that gap is seeded here on purpose.

from dataclasses import dataclass


@dataclass
class Panel:
    signal: str
    metric_type: str
    priority: int


draft = [
    Panel("error_rate", "counter (rate)", 1),
    Panel("latency_mean", "histogram", 1),  # wrong statistic for the symptom
    Panel("traffic_rps", "counter (rate)", 2),
    Panel(
        "gc_pause_count", "counter (rate)", 2
    ),  # a cause-level drill-down, not golden
    # one golden signal is missing -- add it
]


def fix_dashboard(panels: list[Panel]) -> list[Panel]:
    # 1. replace the mean latency panel with p99
    # 2. add the missing golden signal (which one is absent?)
    return panels


for p in sorted(fix_dashboard(draft), key=lambda p: (p.priority, p.signal)):
    print(p.priority, p.signal, "via", p.metric_type)

from dataclasses import dataclass


@dataclass
class Panel:
    signal: str
    metric_type: str
    priority: int


draft = [
    Panel("error_rate", "counter (rate)", 1),
    Panel("latency_mean", "histogram", 1),
    Panel("traffic_rps", "counter (rate)", 2),
    Panel("gc_pause_count", "counter (rate)", 2),
]


def fix_dashboard(panels: list[Panel]) -> list[Panel]:
    fixed = []
    for p in panels:
        if p.signal == "latency_mean":
            fixed.append(Panel("latency_p99", "histogram", 1))  # symptom needs the tail
        else:
            fixed.append(p)
    fixed.append(Panel("saturation", "gauge", 2))  # the missing golden signal
    return fixed


for p in sorted(fix_dashboard(draft), key=lambda p: (p.priority, p.signal)):
    tier = "SYMPTOM" if p.priority == 1 else "context"
    print(f"[{tier:>7}] {p.signal:<13} via {p.metric_type}")

Summary

• The mean is structurally blind to a right-skewed tail because a handful of large values barely move a sum dominated by the fast majority, so alert on the p99 (a rank in the sorted order, $\lceil 0.99 \times N \rceil$) and diagnose with p50 and p99 together.
• The metric type decides which question survives a scrape: a counter recovers a rate, a gauge recovers only the last value, a histogram recovers any percentile at the cost of one counter per bucket. Storing a distribution as a gauge deletes the percentile permanently.
• A trace localizes where inside one request the time went by rebuilding a span tree from child-to-parent edges, but only if the trace id survives every boundary; any hop that drops it orphans every downstream span and the request looks finished at the boundary.
• An unbounded label mints one time series per distinct value, and cardinality is multiplicative, so a request_id label is an out-of-memory outage of the shared metrics store with a delay. Per-request identity belongs in logs and traces, not metric labels.
• A dashboard is the four-golden-signals question list (latency, traffic, errors, saturation), symptoms on top and context below, not a wall of every graph. It shows the service is up; it cannot yet show the model is right.

Check your understanding:

• A service reports a mean latency of 15 ms and a p99 of 900 ms over 100,000 requests. Roughly how many real requests is the p99 describing, and why did the mean not move when they slowed down?
• You stored loaded-model memory as a counter for a month. What can you no longer recover from that data, and what should the type have been?
• Without looking back: a trace shows the request span ending at the feature-lookup boundary with no downstream spans, even though a downstream call definitely happened. What broke, and where is the downstream time now filed?
• Someone proposes a customer_segment label (about 8 values) and a session_id label (one per session) to “slice errors better.” Which one is safe on a metric, which one moves to the log line, and what is the failure if it does not?