Bound the Cost Before It Bounds You

I deployed a decision-tree scorer once, demoed it, and walked away. It worked flawlessly. Months later a finance ping asked what a small recurring charge was for, and I had to go look: the service had served almost no traffic the entire time and had still billed every single hour it existed. An always-on instance at full size, a load balancer, and a reserved static IP had each been running their own meter the whole time. No alert fired, no health check went red, nothing in the deploy ever surfaced the waste. The only thing still running was the clock.

The previous lesson left this service recoverable: a one-step rollback to the previous known-good image digest, the content-addressed identifier of the exact container that was tested. The deploy is now reachable at a real URL over TLS, the tested artifact is the shipped one, the name resolves, and a bad version is one command from gone. The service is correct under traffic and recoverable. The one thing it is not yet is bounded in cost. Every earlier lesson hardened a dimension that has traffic in it: a stranger reaching the service, a name resolving, a version rolling back. This lesson hardens the one dimension with no traffic at all: an idle deploy keeps a clock running, and cost is the only failure that accrues silently while every health check stays green. The work is to make idle cost near-zero or capped, pick the billing model from where the service sits on the traffic curve, and state the expected and worst-case monthly figure out loud.

Bill for what is reserved, not for what is used

The mental model a competent engineer carries into a first cloud deploy is the metered-utility model: a service that gets steady traffic costs roughly what its instance-hours cost, busy means expensive, quiet means cheap, the way electricity bills track use. Under steady traffic that model is close enough to be harmless. The trouble is that an ML service deployed by one person is almost never steady; it is bursty and mostly idle, a demo and a handful of users with long nights and weekends at zero requests. Predict the bill for a service that ran flawlessly for five weeks and served almost nothing, and the utility model says “almost free.” The meter says otherwise.

Here is the prediction, made concrete. The utility model expects cost to track the work actually done, so the two columns below should differ enormously.

# Two deploys, both on the SAME always-on instance, both up 24/7 for a month.
# Service A: a forgotten demo — a trickle of requests.
# Service B: real usage — orders of magnitude more requests.
# The utility ("busy = expensive") model predicts B costs far more than A.

hours_per_month = 730            # ~365 * 24 / 12
instance_rate   = 0.05           # $/hour for the reserved instance

def monthly_instance_cost(requests_served: int) -> float:
    # The wrong model: cost is a function of requests.
    return requests_served * 0.0001   # made up per-request price

print(monthly_instance_cost(requests_served=300))        # Service A, forgotten
print(monthly_instance_cost(requests_served=3_000_000))  # Service B, busy
# Predicts $0.03 vs $300 — a 10,000x gap that tracks traffic.

That model is wrong because requests_served never enters the real formula. Cloud providers bill for the capacity they hold for you, not the work you ask of it. On-demand compute is priced per instance-hour, metered from the moment an instance starts until it is stopped or terminated — the meter reads wall-clock time, not request count. Two services at the same 24/7 uptime cost nearly the same regardless of how many requests each served, because request count is not a variable in the equation. Here is what the meter actually computes.

"""Why an always-on instance costs the same idle as busy: cost is hours x rate.

The reserved instance bills for every hour it is *up*, regardless of how many requests
it serves. A forgotten demo and a heavily-used service that run the same hours cost the
same — traffic does not appear in the formula. That is why a forgotten deployment is
dangerous: zero users, full bill.
"""


def instance_monthly_cost(hourly_rate: float, hours_running: float) -> float:
    return hourly_rate * hours_running


if __name__ == "__main__":
    hours_per_month: float = 730.0  # 365 * 24 / 12, the standard cloud-billing month
    rate: float = 0.05  # $/hour for one reserved instance (illustrative)

    # Same hours, wildly different traffic — and the bill is identical, because traffic
    # is not in the formula. The forgotten demo costs exactly what the busy service does.
    cost = instance_monthly_cost(rate, hours_per_month)
    print(f"forgotten demo (near-zero traffic): ${cost:.2f}")
    print(f"busy service (heavy traffic):       ${cost:.2f}")
    print("difference: $0.00 — an idle always-on instance bills like a busy one")

The difference is zero. The quiet weeks cost exactly what the busy weeks cost, because the lever that moves an always-on bill is reserved hours, never request count, and the only way to lower reserved hours is to downsize, shut down off-hours, or scale to zero. Systems Performance states the corollary: “Cost savings can be realized immediately when they downsize.” Why is the saving immediate? Because the meter is per-hour: the moment a reserved hour stops being reserved, it stops being billed. There is no settling period because there was never a per-request accumulator to drain.

A scale-to-zero platform, a serverless container host that runs a container only while a request is in flight, attacks exactly that. Foundations of Scalable Systems describes the property: serverless platforms “do not require any compute resources to be statically provisioned,” so nothing is reserved during idle hours and the meter stops. The cost it trades for that saving is a cold start: the first request after an idle gap pays for the platform to pull the image, boot the container, and run startup before it can serve. For an ML service the model load is the startup, so cold-start time scales with model size. Designing Distributed Systems is explicit that re-loading a model on start “can add minutes of time to your application startup,” which is why the canonical fix is to cache the model locally rather than re-download weights on every cold worker. A decision-tree scorer is on the order of seconds; a large model is minutes.

The trap that produced the opening incident is the third meter, the one the instance price never names. Scaling the container to zero does not zero the resources attached to it. A load balancer, a reserved static IP, and egress bandwidth each bill on their own continuous meter, independent of the compute. A managed load balancer typically bills per hour it is running, regardless of traffic. A reserved public IP address is commonly billed per hour even when it is idle — attached to nothing — so it costs money for merely existing. These are the line items that survive “I turned the service off,” and they must be enumerated explicitly because nothing in the headline instance price mentions them. The failure mode is named: a zombie attached-resource bill, where compute is scaled to zero, every dashboard is green, and a load balancer plus a static IP quietly bill a service that handles no traffic, discovered only on an invoice.

The three meters compose differently depending on which billing model the compute is on, and the cleanest way to see it is side by side.

The compute row is the only one scale-to-zero changes; the bottom three are orthogonal to the billing model entirely. That separation is the whole reason the opening bill was a surprise: the engineer reasoned about the row that moves and never enumerated the three that do not.

"""Try It: a staging deploy that is "basically free because nobody uses it."

Predict the monthly bill, then fill in the calculation. One argument does not
belong in the formula.
"""


def staging_monthly_cost(
    instance_rate: float,
    hours_running: float,
    lb_monthly: float,
    static_ip_monthly: float,
    requests_served: int,
) -> float:
    """Return the monthly bill. One of these arguments does not belong in the formula."""
    # TODO: sum the meters that actually bill. Leave out the one that does not.
    return 0.0  # placeholder -- replace


if __name__ == "__main__":
    print(staging_monthly_cost(0.05, 730.0, 16.0, 3.6, requests_served=12))

"""Solution: an always-on bill is a sum of clocks; requests_served is not one."""


def staging_monthly_cost(
    instance_rate: float,
    hours_running: float,
    lb_monthly: float,
    static_ip_monthly: float,
    requests_served: int,
) -> float:
    """Sum the reserved-time meters. requests_served is intentionally ignored."""
    compute = instance_rate * hours_running
    return compute + lb_monthly + static_ip_monthly


if __name__ == "__main__":
    total: float = staging_monthly_cost(0.05, 730.0, 16.0, 3.6, requests_served=12)
    print(f"monthly bill: ${total:.2f}")
    print("requests_served (12) contributed: $0.00 -- it is not in the formula")

Pick the billing model from where the service sits on the traffic curve

The first section showed scale-to-zero stopping the idle meter, and the reflexive next move is to treat it as the obviously-better choice and reach for it by default: serverless sounds modern, idle cost goes to near-zero, done. That reflex is wrong because it reads the two models as a quality ranking when they are positions on a response curve. The cost-optimal answer flips as sustained traffic rises, and picking by which model sounds better is how a service ends up on the expensive side of its own traffic.

The plausible-but-wrong assumption is “scale-to-zero is cheaper, period.” Watch it break as concurrency climbs.

# Wrong model: scale-to-zero always wins on cost.
# Reality: as sustained concurrency rises, the platform keeps an instance warm
# continuously (there is always a request in flight), so you pay a warm instance
# PLUS per-request overhead — which can cross above a flat always-on rate.

always_on_monthly = 0.05 * 730            # flat: ~$36.50
def scale_to_zero_monthly(requests: int) -> float:
    warm_instance = 0.05 * 730            # once traffic never drops to zero, it stays warm
    per_request   = requests * 0.0004     # metered overhead on top
    return warm_instance + per_request

print(scale_to_zero_monthly(1_000))       # bursty: warm rarely — overstated, but illustrates
print(scale_to_zero_monthly(5_000_000))   # steady: warm + millions of request units > flat

The model breaks because the deciding variable is not the platform’s name, it is how much of the time an instance would otherwise sit idle. Designing Distributed Systems states the crossover plainly: the pay-per-request model “is great if you only have a few requests per minute or hour,” but “as a service grows… the economics of a pay-per-request model start to become bad” while a long-running instance’s cost “generally decreases as you add more cores.” The mechanism behind the crossover is that scale-to-zero only saves money during hours it actually reserves nothing; once sustained concurrency means there is always a request in flight, the platform keeps an instance warm continuously anyway, and the bill becomes a permanently-warm instance plus per-request overhead against what would have been a flat rate. Idle hours are the entire saving, so when idle hours vanish, so does the reason to use it.

The correct model has two axes, and the staff move is naming the line between them rather than reaching for one option by reflex. The cost axis is governed by the fraction of idle hours; the latency axis is the cold start from the last section, and it is orthogonal to cost. A service can sit firmly on the cheap side of the cost curve and still be disqualified from scale-to-zero because a human is waiting synchronously on a first request that pays a multi-second model load.

The block below computes both bills across the curve so the crossover is a number, not an intuition. Watch the same scale-to-zero model that wins at bursty traffic cross above the flat always-on rate once concurrency keeps an instance warm continuously.

"""Scale-to-zero wins while idle, but crosses above flat once an instance stays warm."""


def always_on_monthly(rate: float, hours: float) -> float:
    return rate * hours


def scale_to_zero_monthly(
    rate: float, hours: float, requests: int, warm: bool
) -> float:
    """If sustained traffic keeps an instance warm, you pay the warm instance PLUS overhead."""
    warm_instance: float = rate * hours if warm else 0.0
    per_request_overhead: float = requests * 0.0004
    return warm_instance + per_request_overhead


if __name__ == "__main__":
    rate: float = 0.05
    hours: float = 730.0
    flat: float = always_on_monthly(rate, hours)

    bursty: float = scale_to_zero_monthly(rate, hours, requests=2_000, warm=False)
    steady: float = scale_to_zero_monthly(rate, hours, requests=3_000_000, warm=True)

    print(f"always-on (flat):              ${flat:.2f}")
    print(f"scale-to-zero, bursty/idle:    ${bursty:.2f}  -> cheaper")
    print(f"scale-to-zero, steady/warm:    ${steady:.2f}  -> crosses above flat")

At bursty traffic scale-to-zero wins decisively because the warm-instance term is zero: the bill is only the requests handled. At steady traffic the same model pays a warm instance plus millions of request units and crosses above the flat rate. The cheaper option at low traffic became the more expensive one at high traffic, and nothing about the platform changed except where the service sat on its own curve.

Because the deciding variable is a tunable position rather than a fixed default, the choice is read off the curve in both directions.

The same scorer is the right candidate for scale-to-zero on the day it is deployed and the wrong one if it ever takes sustained traffic. Picking once and never re-reading the curve is the failure: the decision is not made at deploy time and frozen, it is re-checked against the warm-instance signal as traffic changes.

"""Try It: recommend a billing model from the crossover signal.

Warm-continuously OR metered-cost-above-flat means switch to always-on.
"""


def recommend_billing_model(
    warm_continuously: bool,
    metered_monthly: float,
    always_on_monthly: float,
) -> str:
    """Return 'always-on' or 'scale-to-zero' from the crossover signal."""
    # TODO: switch to always-on when the instance never sleeps OR metered cost > flat rate.
    return "scale-to-zero"  # placeholder -- does not yet read the signal


if __name__ == "__main__":
    print(
        recommend_billing_model(
            warm_continuously=True, metered_monthly=92.0, always_on_monthly=36.5
        )
    )

"""Solution: either trigger alone is enough to switch to always-on."""


def recommend_billing_model(
    warm_continuously: bool,
    metered_monthly: float,
    always_on_monthly: float,
) -> str:
    """Switch to always-on if the instance never sleeps OR metered cost exceeds the flat rate."""
    if warm_continuously or metered_monthly > always_on_monthly:
        return "always-on"
    return "scale-to-zero"


if __name__ == "__main__":
    steady: str = recommend_billing_model(True, 92.0, 36.5)
    bursty: str = recommend_billing_model(False, 4.0, 36.5)
    print(f"warm + metered $92 vs flat $36.50 -> {steady}")
    print(f"idle + metered $4 vs flat $36.50  -> {bursty}")

Convert worst-case cost from unknown to stated

Whichever billing model the service lands on, the instinct is to assume the choice itself bounds the cost: scale-to-zero feels safe because idle is near-zero, always-on feels safe because it is a flat rate. Both assumptions miss the same thing: an autoscaler, the controller that adjusts how many instances run in response to load, sitting on either model. Reliable Machine Learning describes it precisely: “Autoscaling dynamically adjusts the number of instances provisioned for a model in response to changes in the workload… we can configure the minimum and maximum scaling capacity and a cool-down period to control scaling behavior and price.” An autoscaler with no upper bound is an open-ended financial liability, because it is built to convert load into spend and it cannot tell load worth serving from load that is attacking the service.

Watch the uncapped case turn a weekend scraper into a bill.

# Uncapped autoscaler: instance count tracks concurrency to whatever height load reaches.
# A scraper, a retry storm, and a real spike all register identically as concurrency.

def instances_for(concurrency: int, target_per_instance: int = 80) -> int:
    return concurrency // target_per_instance  # NO max — rides load to any height

def overnight_cost(concurrency: int, rate: float = 0.05, hours: float = 10.0) -> float:
    return instances_for(concurrency) * rate * hours

print(overnight_cost(concurrency=200))      # normal: a couple of instances
print(overnight_cost(concurrency=160_000))  # scraper over a weekend: hundreds of instances
# The autoscaler faithfully converts the attack into instances, fastest exactly
# when something is already going wrong — and nobody is watching the console.

That breaks because instances_for has no ceiling: instance count rides concurrency linearly to any height, and cost rides instance count. The autoscaler is working correctly, and that is the problem. It maps load to instances exactly as designed, and a scraper, a retry storm, and a genuine spike are indistinguishable as concurrency, so it converts all three into instances and the instances into spend, fastest precisely when something is already wrong. The named failure mode is a weekend cost blowout: an uncapped autoscaling service hit by a scraper scales to dozens of instances and bills overnight, with no ceiling, no alert, and nobody watching the console, so the number is only visible on Monday’s invoice. The spike, the scale-out, and the spend are all complete and in the past by the time a human looks.

Bounded cost is not a vibe that a deploy is “cheap enough.” It is a configured ceiling plus a stated number. The ceiling is a max-instances cap, a hard upper bound on instance count. Scale-to-zero platforms expose this cap as an explicit cost-control lever: you set a maximum-instances value low enough to protect the budget from an unexpected spike, and requests beyond that capacity are rejected rather than served. That rejection is the trade: a cap makes the worst case a known quantity by giving up availability past the cap (excess load is shed) in exchange for a hard cost bound. For a single-owner ML service, that trade is almost always right, because an outage past the cap is recoverable and an unbounded bill is not.

The cap is what makes worst-case cost a formula instead of a guess. The two figures the owner must be able to state are computed here:

"""A max-instances cap turns worst-case cost from a guess into a finite formula."""


def expected_monthly(
    typical_instances: float, hourly_rate: float, hours: float, attached: float
) -> float:
    """Expected: the instances usually running, plus the attached-resource meters."""
    return typical_instances * hourly_rate * hours + attached


def capped_worst_case(
    max_instances: int, hourly_rate: float, hours: float, attached: float
) -> float:
    """Worst case: the cap pinned at the ceiling for the whole month, plus attached."""
    return max_instances * hourly_rate * hours + attached


if __name__ == "__main__":
    rate: float = 0.05
    hours: float = 730.0
    attached: float = 19.6  # load balancer + static IP monthly, from the first section

    expected: float = expected_monthly(1.2, rate, hours, attached)
    worst: float = capped_worst_case(
        max_instances=3, hourly_rate=rate, hours=hours, attached=attached
    )

    print(f"expected monthly:          ${expected:.2f}")
    print(f"capped worst-case monthly: ${worst:.2f}")

The worst-case line is finite only because max_instances is finite. Remove the cap and that term has no upper bound (it is whatever the attack reaches), which is why the uncapped case has no worst-case figure to state at all. Notice the attached-resource meters from the first section appear in both formulas: the cap bounds compute, not the load balancer or the static IP, so those clocks are still in the total.

The cap is itself a tunable, and the cost is the direction of the error.

The cap bounds the worst case; it does nothing while the spend is happening. That is the other half: a budget alert, a nonzero threshold on accumulated spend that fires while the spend is still in progress and still stoppable, instead of after. This is the burn-rate-alert pattern from observability applied to a cost budget. Observability Engineering describes the pattern for an error budget, “track your error budget burn rate… pick a nonzero threshold on which to alert,” and the move here is to apply it to dollars: alert before the budget is fully consumed, not when the invoice arrives. Advanced SQL names the surprise the alert exists to prevent: “Model costs have a way of surprising teams. A prompt that costs pennies in development can cost hundreds of dollars when it runs against a full production table.” The threshold converts that invoice surprise into a mid-month page.

"""Burn-rate alert: fire at a nonzero fraction of budget, before it is fully consumed."""


def budget_alert_should_fire(
    spend_so_far: float, monthly_budget: float, threshold: float
) -> bool:
    """Burn-rate alert: fire at a nonzero fraction of budget, before it is fully consumed."""
    return spend_so_far >= monthly_budget * threshold


if __name__ == "__main__":
    budget: float = 50.0
    threshold: float = 0.8  # page at 80% consumed -- nonzero, before the invoice

    for spend in (20.0, 38.0, 44.0):
        fired: bool = budget_alert_should_fire(spend, budget, threshold)
        print(f"spend ${spend:.2f} of ${budget:.2f} budget -> alert fires: {fired}")

The alert fires at 80% of budget, while there is still budget left and the spend is still stoppable: pull the deploy, tighten the cap, block the scraper. A zero-or-100% threshold would only ever report after the fact; the nonzero threshold is the entire point, because it opens the window in which the spend can still be acted on. The cap bounds the worst case and the alert catches the drift toward it, but the staff deliverable is the number itself: an owner who cannot write down the expected and capped-worst-case monthly figures does not control the deploy, because cost nobody can state is cost nobody is accountable for.

"""Try It: state the worst-case monthly cost an owner must produce on demand.

Return the capped worst-case figure -- and show what the same function returns
when the cap is removed (passed as None).
"""


def capped_worst_case(
    max_instances: int | None,
    hourly_rate: float,
    hours: float,
    attached: float,
) -> float:
    """Return the worst-case monthly cost. If there is no cap, the worst case is unbounded."""
    # TODO: when max_instances is None there is no finite ceiling -- return float("inf").
    # Otherwise: cap * rate * hours + attached.
    return 0.0  # placeholder -- replace


if __name__ == "__main__":
    print(capped_worst_case(3, 0.05, 730.0, 19.6))
    print(capped_worst_case(None, 0.05, 730.0, 19.6))

"""Solution: a finite cap gives a statable figure; no cap returns infinity."""


def capped_worst_case(
    max_instances: int | None,
    hourly_rate: float,
    hours: float,
    attached: float,
) -> float:
    """Capped: cap * rate * hours + attached. Uncapped: the compute term is unbounded."""
    if max_instances is None:
        return float("inf")
    return max_instances * hourly_rate * hours + attached


if __name__ == "__main__":
    capped: float = capped_worst_case(3, 0.05, 730.0, 19.6)
    uncapped: float = capped_worst_case(None, 0.05, 730.0, 19.6)
    print(f"capped worst-case:   ${capped:.2f}")
    print(f"uncapped worst-case: {uncapped}")

Summary

• An always-on instance meters reserved wall-clock time, not requests, so a forgotten idle service and a busy one at the same uptime cost nearly the same; the lever that lowers the bill is reducing reserved hours, never reducing traffic.
• Scale-to-zero stops the idle compute meter but pays a cold start on the first request after idle, and for an ML service the model load is the startup: seconds for a small scorer, minutes for a large model.
• Attached resources (load balancer, reserved static IP, egress) bill on independent continuous meters; scaling compute to zero does not zero them, which is how a service handling no traffic still produces a bill.
• The billing model is a position on a traffic curve, not a quality ranking: scale-to-zero wins while idle hours dominate, always-on wins once sustained concurrency keeps an instance warm continuously and metered cost crosses above the flat rate.
• A max-instances cap bounds worst-case spend to max_instances × rate × hours + attached by shedding load past the cap; a budget alert at a nonzero threshold pages the owner while the spend is still stoppable; the deliverable is the expected and capped-worst-case figures stated out loud.

Check your understanding:

• Two services run 24/7 on the same always-on instance, one served twelve requests this month, the other served three million. Without looking back: why do they cost nearly the same, and what is the one change that would lower either bill?
• You deployed a small scorer on scale-to-zero at launch. What single signal tells you it is time to switch it to always-on, and what is the cost you start paying the moment that signal appears?
• An autoscaler is up with no max-instances cap. What turns a weekend traffic spike into an unbounded bill, which two settings bound it, and what does capped_worst_case return until you add the cap?