Why Click-Ops Is the "Works on My Machine" of Infrastructure

The deployed Lending Club default scorer ran fine in production. Then it had to exist in a second region, and nobody could reproduce it. The original had been clicked together in a web console months earlier, across a dozen screens, by an engineer who had since changed teams. There was no record of what existed: which firewall rules, which instance size, which environment variables were set where. Standing up the duplicate took days of archaeology, and the result was still not identical to the original. The infrastructure had the exact disease the application layer cured years ago. It worked on one account, in one region, and could not be reproduced anywhere else.

That sentence, “works on one account, cannot be reproduced anywhere else,” is the infrastructure version of works on my machine. The last lesson made one resource exist as code and made the plan diff explicit, so the desired state lived in a file and applying it reconciled reality to that file. This lesson turns to the alternative most engineers have been doing implicitly for years: click-ops, provisioning infrastructure by hand in a web console with no recorded configuration. The problem comes first because the fix only makes sense once the cost is visible. Click-ops is the same class of failure containers cured for the application, moved up one layer to the resources the application runs on. The payoff of this lesson is the judgment to name when the unreproducibility cost of clicking has grown past the adoption cost of a tool.

Two terms carry the whole argument, so they are grounded here even though the last lesson introduced them. A declarative config is one where the desired end state is stated and the tool reconciles reality to it, rather than scripting the steps to get there. Reproducible-by-construction means the same input config produces the same resources every time, because the config is the input and there is nothing else to re-derive. Both matter for one reason: the failure of click-ops is that it has no input to re-run.

Click-ops produces no artifact, so it is unreproducible by construction

Clicking through a console feels like building something durable. The resource appears on the screen, it works, it serves traffic. The plausible mental model, the one a competent engineer holds after shipping plenty of working systems by clicking, is that the resource existing is the same thing as the resource being recorded. It is not. A console click mutates live infrastructure directly and leaves the result behind while destroying the recipe: the intent, the alternatives considered, the order of operations, and every field left at a default the operator never looked at.

The mental model breaks the day the thing has to exist a second time. The only “record” of a clicked environment is a sequence of actions nobody wrote down, and reconstructing it from memory looks like this (the jargon below — a VPC is a private cloud network, a CIDR is the IP-address range it spans, a firewall rule decides what traffic is allowed in — is exactly the detail that gets forgotten):

# The only "record" of the production scorer's network, reconstructed from memory
1. Open the VPC console -> create network "scorer-net"  (which CIDR? nobody remembers)
2. Add a firewall rule -> allow tcp:8080 from ... 0.0.0.0/0? or just the LB subnet?
3. Create an instance -> size? "the medium one" -> which generation? which image?
4. Set env vars on the instance -> MODEL_BUCKET=..., THRESHOLD=... (set where? when?)
5. ... several more screens, each with defaults nobody saw

# Reproduce it in region B:
$ ???   # there is no command. the recipe was the clicking, and the clicking is gone.

There is no $ ??? that reproduces a clicked environment, because the input was never captured. The output of provisioning must be a recorded description of what should exist, not merely the existence of the thing itself. Reproducibility is not a property added afterward; it is a property of having written the recipe down before reality changed. When the desired state lives as version-controlled configuration, the artifact is the recipe, applying it makes reality match the file, and a second environment is the same file with a different variable file. That is reproducible-by-construction stated concretely: the same input deterministically produces the same output.

The mechanism behind “two operators building the same thing diverge” is the field count. A single cloud resource exposes dozens to hundreds of configurable fields, and the console fills most of them with invisible defaults the operator never sees. Infrastructure as Code (Kief Morris) states the consequence directly: setting server options manually “encourages mistakes and leads to inconsistently configured servers, unpredictable systems, and too much maintenance work.” Two people building “the same thing” make different unseen choices on the very first screen, and the environments diverge from that first action. This is configuration drift, similar infrastructure elements becoming inconsistent over time, and Morris is explicit that even the same operator building a fresh server “often has differences from one built weeks or months earlier.” Drift is not a rare accident; it is the default outcome of a process with no recorded input.

The contrast lives at the level the argument turns on: what a console hands back versus what a config file hands back when the question is “what exactly is here?” The function below takes a record of an environment and reports how much of it can be reproduced from that record alone. Watch the unknown count: those are fields that existed, mattered, and are gone.

"""What a console hands back versus what a config file hands back.

The function takes a record of an environment and reports how much of it can be
reproduced from that record alone. Watch the unknown count -- those are fields that
existed, mattered, and are gone. The clicked environment is unreproducible because
there is no input to re-run; the declared one is reproducible by construction.
"""

from dataclasses import dataclass, field


@dataclass
class Environment:
    name: str
    recorded_fields: dict[str, str] = field(default_factory=dict)
    total_fields: int = 0  # every field the resource actually has
    has_artifact: bool = False  # is the desired state written down?


def reproducibility_report(env: Environment) -> str:
    recorded = len(env.recorded_fields)
    unknown = env.total_fields - recorded
    if not env.has_artifact:
        # No input to re-run: the unrecorded fields are unrecoverable.
        return (
            f"{env.name}: {recorded}/{env.total_fields} fields recorded, "
            f"{unknown} left at invisible defaults nobody can recover. "
            f"Reproducible: NO (no artifact to re-apply)."
        )
    return (
        f"{env.name}: {recorded}/{env.total_fields} fields recorded in config. "
        f"Reproducible: YES (re-apply the file)."
    )


# The clicked production environment: the operator "remembers" four fields.
clicked = Environment(
    name="prod-clicked",
    recorded_fields={
        "region": "us-1",
        "port": "8080",
        "size": "medium",
        "image": "ubuntu",
    },
    total_fields=40,
    has_artifact=False,
)

# The same environment as declarative config: every field is in the file.
declared = Environment(
    name="prod-declared",
    recorded_fields={f"field_{i}": "set" for i in range(40)},
    total_fields=40,
    has_artifact=True,
)

print(reproducibility_report(clicked))
print(reproducibility_report(declared))

The clicked environment reports 36 fields that are gone: not wrong, gone, with no input to recover them from. The declared one is reproducible not because someone was careful but because the file is the input, and re-applying it is deterministic. This is the non-obvious cost a senior engineer underprices: the clicked environment works, so the bill for the 36 missing fields does not arrive at provisioning time. It arrives the day someone needs the environment to exist twice, and by then the operator who set those defaults has changed teams. Microservices (Sam Newman) records the shape of that bill exactly: a client without version-controlled infrastructure “ended up spending over three months painstakingly trying to rebuild a mirror image of an earlier production environment by wading through emails and release notes to try and work out what was done by whom.” That is the archaeology the opening incident dramatized, paid in months.

The divergence does not happen all at once. It accumulates across the lifetime of two environments built independently, which is why prose cannot capture it in a sentence. Scroll through the two paths below: the clicked path widening into measurable difference, and the IaC path staying identical by construction.

The scrolly’s last step is the whole argument compressed: the divergence is not a discipline failure, it is structural. Two paths built from an unrecorded recipe cannot be guaranteed identical, because there is no recipe to compare against. One config with two variable files cannot diverge, because there is exactly one description of what “the scorer environment” is.

"""Try It 1 starter: what is lost when a clicked resource is deleted.

Complete the function so it returns the list of lost information categories for a
clicked environment (and an empty list for a recorded one). The config file is the
input -- anything written in it survives a delete because re-applying recreates the
resource. A click leaves no input, so everything not externally recorded is gone.
"""


def lost_on_delete(has_artifact: bool) -> list[str]:
    # A clicked environment loses categories of information a recorded one keeps.
    # Return the categories lost when has_artifact is False; [] when True.
    candidates = [
        "the intent (why each field was set this way)",
        "the values of fields left at invisible defaults",
        "the order/sequence the resource was built in",
        "a diff to review or roll back to",
        "who changed what and when",
    ]
    if has_artifact:
        return ["placeholder"]  # replace: a recorded env loses nothing reproducible
    return ["placeholder"]  # replace: a clicked env loses all of the above


print("clicked:", lost_on_delete(has_artifact=False))
print("declared:", lost_on_delete(has_artifact=True))

"""Try It 1 solution: a clicked environment loses every category; a recorded one loses none.

The file is the input, so re-applying it restores the resource and every field in
it. A click captures nothing outside the operator's memory and the now-deleted live
resource, so all five categories are unrecoverable. This is the asymmetry the rest
of the lesson builds on.
"""


def lost_on_delete(has_artifact: bool) -> list[str]:
    candidates = [
        "the intent (why each field was set this way)",
        "the values of fields left at invisible defaults",
        "the order/sequence the resource was built in",
        "a diff to review or roll back to",
        "who changed what and when",
    ]
    if has_artifact:
        return []  # the file is the input; re-apply it and everything returns
    return candidates  # no input was captured, so every category is unrecoverable


print("clicked:", lost_on_delete(has_artifact=False))
print("declared:", lost_on_delete(has_artifact=True))

A config file is a reviewable diff before it touches reality

The reflex defense of click-ops from an experienced engineer is “I am careful, and I review my own changes before I make them.” The hidden assumption is that the moment a change becomes visible and the moment it becomes live can be the same instant without losing anything. They cannot. A click collapses propose, review, and enact into one irreversible action: by the time a second person can examine what changed, it is already in production.

The breaking condition is a change that is wrong: a port opened to 0.0.0.0/0 instead of the load-balancer subnet, an instance sized ten times too large, a field set on the wrong resource. With click-ops, the only review available is forensic, and it reads like this:

# Click-ops review of "why is the scorer reachable from the public internet?"
# (runs AFTER the rule is live, as an incident)
1. Open the firewall console -> read the live rule -> "allow tcp:8080 from 0.0.0.0/0"
2. Try to remember: was it always like this? who changed it? when?
3. No history. No prior value. No diff. File an incident; guess at the rollback target.

Every line of that runs after the misconfiguration is serving traffic. The most valuable property of infrastructure-as-code is not that the config exists; it is when the change becomes inspectable. Pulling the change out as a text edit separates the proposal of a change from its enactment, and that gap is where review, approval, and rollback live. The same change as a config edit is a diff in a pull request, reviewed and approved while it is still only text:

  resource "firewall_rule" "scorer_ingress" {
    protocol = "tcp"
    port     = 8080
-   source   = "0.0.0.0/0"          # reviewer: this is the whole internet, reject
+   source   = "10.0.1.0/24"        # the load-balancer subnet only
  }

A reviewer rejects that diff in seconds, before anything mutates. Advanced SQL (Castro) names the mechanism precisely: when configuration lives in version control, a change such as opening a port or creating a schema “is reviewed and logged just like a code change, reducing accidental misconfigurations.” A clicked change has no such surface; it is enacted at the moment it is proposed, so the only review is a post-mortem. Newman adds the audit half: version-controlling infrastructure “gives you transparency over who has made changes, something that auditors love.” The diff is both the review surface and the audit log; a click is neither.

This is the same review discipline already applied to application code (a pull request) extended to the resources the code runs on. The two practices are the same shape: Container Security (Liz Rice) and GitOps treat infrastructure changes as code changes that flow through the same review and version-control process as application code. The plan step from the last lesson adds a second inspection point on top of the diff. The config diff is a review of intent: does this change read correctly as text. The plan is a review of consequence: given current reality, here is the exact set of resources the tool would create, change, or destroy.

The two inspection points are not redundant, and the gap between them is a real failure mode. The function below models a change passing through both gates. Watch the diff that reads innocently still produce a destructive plan.

"""A change passing through both gates: code review of the diff, then plan of consequence.

Watch the diff that reads innocently still produce a destructive plan. The diff is a
review of intent -- does the text read correctly. The plan is a review of consequence
-- given current reality, here is what the tool would create, change, or destroy. A
team that approves diffs without reading the plan has the surface but skips the check.
"""

from dataclasses import dataclass


@dataclass
class Change:
    description: str
    diff_looks_safe: bool  # does the text edit read as harmless to a reviewer?
    plan_destroys: int  # how many existing resources the plan would destroy


def review_gates(change: Change) -> str:
    # Gate 1: code review of the diff (intent). Gate 2: plan of consequence.
    diff_verdict = "approved" if change.diff_looks_safe else "rejected at diff"
    if not change.diff_looks_safe:
        return f"{change.description}: {diff_verdict} -- never enacted."
    plan_verdict = (
        f"plan shows {change.plan_destroys} resources DESTROYED -- stop"
        if change.plan_destroys > 0
        else "plan is additive -- safe to apply"
    )
    return f"{change.description}: diff {diff_verdict}; {plan_verdict}."


rename = Change(
    description="rename the scorer instance",
    diff_looks_safe=True,  # a one-word edit; reads harmless
    plan_destroys=1,  # but rename forces destroy+recreate of the live box
)
widen_port = Change(
    description="open port to 0.0.0.0/0",
    diff_looks_safe=False,  # caught at the diff
    plan_destroys=0,
)

print(review_gates(rename))
print(review_gates(widen_port))

The rename passes the diff review (it is one harmless-looking word) and the plan is what catches that the change forces the live instance to be destroyed and recreated. The port change never gets that far; the diff review rejects it as text. The non-obvious cost here is that the artifact alone is not the discipline: a team that approves diffs without reading the resulting plan has the review surface but skips the consequence check, and ships a destructive change that looked innocent. Click-ops has neither gate; the cost of IaC is that it has two, and both must actually be read.

"""Try It 2 starter: report the first gate that stops a change.

A reviewer needs to know the first gate that stops a change, because that is where
the cost was paid -- cheap at the diff, more expensive at the plan, catastrophic if
neither caught it. Return the gate name: "diff", "plan", or "none -- reached production".
"""

from dataclasses import dataclass


@dataclass
class Change:
    diff_looks_safe: bool
    plan_destroys: int


def stopping_gate(change: Change) -> str:
    # Return the first gate that stops this change.
    # "diff" if the text review rejects it,
    # "plan" if the diff passes but the plan shows destruction,
    # "none -- reached production" if both gates pass.
    return "placeholder"


print(stopping_gate(Change(diff_looks_safe=False, plan_destroys=0)))
print(stopping_gate(Change(diff_looks_safe=True, plan_destroys=1)))
print(stopping_gate(Change(diff_looks_safe=True, plan_destroys=0)))

"""Try It 2 solution: the gates fire in cost order.

The diff review is the cheapest gate, so it fires first and stops the obviously-wrong
change before anyone runs anything. The plan is the backstop for changes that read
fine but act destructively. A clicked change skips straight to the third line every
time -- it reaches production with no gate, and the only review left is an incident.
"""

from dataclasses import dataclass


@dataclass
class Change:
    diff_looks_safe: bool
    plan_destroys: int


def stopping_gate(change: Change) -> str:
    if not change.diff_looks_safe:
        return "diff"  # cheapest gate: rejected as text, never enacted
    if change.plan_destroys > 0:
        return "plan"  # diff passed, but consequence check caught it
    return "none -- reached production"


print(stopping_gate(Change(diff_looks_safe=False, plan_destroys=0)))  # diff
print(stopping_gate(Change(diff_looks_safe=True, plan_destroys=1)))  # plan
print(stopping_gate(Change(diff_looks_safe=True, plan_destroys=0)))  # none

Clicking is cheaper until exactly the scale where it stops being

Everything so far argues for IaC, which sets up the trap a staff engineer must avoid: concluding that IaC is always correct. It is not free, and pretending it is loses the senior engineer who has shipped plenty of working systems by clicking. The decision is not “good versus bad.” It is “pay the tool cost now” versus “pay the unreproducibility cost later,” and the staff skill is naming the signal that flips which one is larger.

The tool cost is real and named in the literature. Cloud Application Architecture Patterns records that adopting a new infrastructure approach “could take longer, and you have to maintain and govern the old systems as well as the newer cloud solutions for a long time,” raising total cost of ownership during the transition. IaC adds a tool, a state concept to learn (the tool’s record of what it believes exists, which the next lesson covers), and a workflow to set up. Infrastructure as Code (Kief Morris) frames IaC plainly as a practice you adopt, applying “the principles, practices, and tools of software engineering” to infrastructure, not a default state of the world. None of that is zero.

The reason this is a decision and not a rule is that the cost of click-ops is invisible until it is catastrophic. The clicked environment works, so the bill for not recording it does not arrive at provisioning time. The function below prices the two paths against three signals and shows where the lines cross.

"""Price the two paths against three signals and show where the lines cross.

A single operator on a single throwaway sandbox stays in "keep clicking". The moment
a second environment, a second operator, or an audit appears, the branch flips. Any
one of the three signals is enough, because each alone makes the unreproducibility
cost start compounding.
"""


def should_adopt_iac(
    environments: int,
    operators: int,
    must_reproduce_or_audit: bool,
) -> str:
    # The three signals -- any ONE is enough to flip the decision.
    signals = []
    if environments > 1:
        signals.append("multiple environments")
    if operators > 1:
        signals.append("multiple operators")
    if must_reproduce_or_audit:
        signals.append("future reproduce/review/audit need")
    if signals:
        return (
            f"ADOPT IaC now: {', '.join(signals)}. "
            f"The unreproducibility cost is already compounding; the tool has paid for itself."
        )
    return (
        "Keep clicking (for now): genuine one-off, single operator, "
        "never reproduced. The tool cost would not pay off below this scale."
    )


print(should_adopt_iac(environments=1, operators=1, must_reproduce_or_audit=False))
print(should_adopt_iac(environments=2, operators=1, must_reproduce_or_audit=False))
print(should_adopt_iac(environments=1, operators=1, must_reproduce_or_audit=True))

A single operator on a single throwaway sandbox stays in the “keep clicking” branch, because declaring elaborate infrastructure for one box slated for deletion tomorrow is its own waste. The moment a second environment or a second operator or an audit appears, even with everything else held at one, the branch flips. The crossover is driven by any one of three signals: more than one environment must exist, more than one person touches the infrastructure, or anyone will ever need to reproduce, review, or audit what exists. These three are not a quoted rule from a book; they are this lesson’s framing, composed from the IaC benefits the earlier sections verified: reproducibility serves multiple environments, version-controlled review serves multiple operators, and the audit log serves the audit need.

The non-obvious cost runs the other direction too, and it is where the standard “always use IaC” advice is wrong. The expensive case is not adopting too early; it is adopting too late, after a clicked environment has become load-bearing. Retrofitting code onto a clicked environment means reverse-engineering every invisible default the console set, because the tool’s plan will show a diff against reality it did not create and will propose to “correct” fields nobody knew existed. The clean-up cost of “I will codify it later” is strictly higher than codifying on day one, and it grows with every out-of-band tweak. A staff engineer prices the deferred unreproducibility bill at design time, when it is still a paragraph in a design doc, instead of paying it as days of archaeology after the original operator has left.

This trade-off is the reproducibility principle from the packaging module applied one layer up. Docker made the application reproducible: the same image runs the same way on any host. IaC makes the infrastructure reproducible: the same config produces the same resources in any account or region. Both rest on one mechanism: the same setup built twice resolves to two different results unless something pins the answer. A container image pins the application’s dependencies; a config file pins the infrastructure’s fields. Click-ops pins nothing, which is why it is the works-on-my-machine failure moved up a layer.

"""Try It 3 starter: decide adopt-vs-click for four scenarios.

Complete the function so it returns "adopt" when any signal is present and "click"
only when all three are absent. The rule is OR, not AND -- any single signal is
enough to flip the decision toward adoption.
"""


def decide(environments: int, operators: int, audit_or_reproduce: bool) -> str:
    # Return "adopt" if ANY of the three signals is present, else "click".
    # Signals: environments > 1, operators > 1, audit_or_reproduce is True.
    return "placeholder"


# Scenarios:
print(decide(1, 1, False))  # personal sandbox, torn down tomorrow
print(decide(1, 3, False))  # one env, but a whole team touches it
print(decide(2, 1, False))  # second region needed
print(decide(1, 1, True))  # auditor will ask what exists

"""Try It 3 solution: any single signal flips the decision to adopt.

Only the all-absent case stays on "click" -- a true throwaway. Every other row trips
a single signal and flips to "adopt", because one signal is enough for the deferred
unreproducibility cost to start growing. The judgment a staff engineer adds is
foreseeing which signal is coming.
"""


def decide(environments: int, operators: int, audit_or_reproduce: bool) -> str:
    if environments > 1 or operators > 1 or audit_or_reproduce:
        return "adopt"
    return "click"


print(decide(1, 1, False))  # click  -- genuine one-off
print(decide(1, 3, False))  # adopt  -- multiple operators
print(decide(2, 1, False))  # adopt  -- multiple environments
print(decide(1, 1, True))  # adopt  -- future audit/reproduce need

Summary

• Click-ops produces no artifact. A console click changes the world and destroys the recipe: the intent, the order, and every field left at an invisible default. There is no input to re-run, so “reproduce it” means re-deriving a state nobody recorded. This is configuration drift by construction, not by carelessness.
• A config file separates proposing a change from enacting it. The diff is reviewed and the plan is inspected while the change is still text, before reality mutates. A click collapses propose, review, and enact into one irreversible instant, so its only review is a forensic post-mortem.
• Two gates, both must be read. The diff catches wrong intent; the plan catches destructive consequence that read innocently as text. Having the artifact without reading the plan is having the surface without the discipline.
• Adopt IaC when any one of three signals appears: more than one environment, more than one operator, or any future need to reproduce/review/audit. The tool cost is real; the click-ops cost is invisible until it is catastrophic. Price the deferred bill at design time, not at incident time.
• It is the packaging-module reproducibility principle one layer up. Docker pins the application; the config file pins the infrastructure. Click-ops pins nothing, which is exactly why it is works-on-my-machine moved up a layer.

Check your understanding:

• Without looking back: why is click-ops the infrastructure version of “works on my machine”? Name the missing thing that makes it so.
• What three things does an IaC config provide that a console click does not, at provisioning time, at change time, and at audit time?
• Name the three signals that flip the click-ops-vs-IaC decision toward adopting the tool, and explain why any one of them is enough rather than all three.