State, Drift, and the apply That Destroys

I once edited a single setting on a managed database resource, a parameter group name, the kind of one-line change that reviews approve without comment, and ran the plan out of habit before applying. The plan did not say ~ update in place. It said -/+ destroy and then create replacement, with the database resource named on the destroy line. That parameter, on that provider, was create-time-only. One apply would have deleted the production database and built an empty replacement, and the only thing between the edit and the data loss was that I read the plan instead of trusting that a small edit meant a small change. I cancelled, and spent the rest of the day learning that with infrastructure as code, apply is not “save my edit.” It is “make reality match this file by whatever means, including deleting things.”

That gap, between what a change looks like and what it does, is the subject of this lesson. The previous lesson, app-vs-infra, split the two jobs you run against a system: deploying the application, which happens all day, and provisioning the infrastructure, which changes rarely and outlives every deploy. It closed on a tension: infrastructure changes are rare and dangerous, while application deploys are frequent and cheap. This lesson lands that danger. In declare-one-resource you saw plan compute a diff from config and apply execute that diff and nothing more. Here those same mechanics turn destructive, because the diff is computed against a stored model that can be wrong, and one routine apply enforces the entire config, so a problem three resources away from your edit gets corrected in the same run. This is the staff payoff of the module: IaC is its own skill, not “config instead of clicks,” because its source of truth is fallible, reality drifts away from it, and the tool will delete live infrastructure to close the gap.

This lesson teaches the principle of how IaC bites, not Terraform syntax. Terraform is the worked example because it is explicit about state and prints a clear diff, but the subject is state as a fallible model and the destructive apply, and every IaC tool has both.

Treat state as a fallible map, not as ground truth

The mid-level mental model is that the tool reads the cloud, compares it to your config, and proposes the difference, so a plan is a faithful report of “config minus reality.” Under that model, a plan that wants to change a resource nobody touched in the config is a bug in the tool. Here is the shape of that surprise: an engineer edits one resource, runs a plan for that change, and the plan also proposes to modify a second resource whose config lines were not touched at all.

# The config in git: the database tags block was last edited months ago,
# and nobody touched it in this change.
resource "cloud_database" "lending_club" {
  name = "lending-club-prod"
  tags = {
    team = "ml-platform"   # this is what git says the tag should be
  }
}

# The plan for an unrelated change to a different resource:
#   ~ cloud_database.lending_club
#       ~ tags = {
#           ~ "team" = "data-eng" -> "ml-platform"   # nobody edited this line
#         }
# The tool wants to "fix" a tag the config never changed. Tool bug? No.

The plan is not lying, and the tool is not buggy. The hypothesis “the tool is wrong” sends you reading the tool’s source instead of asking the one question that resolves it: which of the three inputs moved. The tool never re-derives the world from scratch. It keeps a state file: its stored record of what it believes it created, the real cloud object IDs it is bound to, and the last field values it observed. State is the only thing that maps your local resource name (cloud_database.lending_club) to the real cloud object ID, and every plan is computed against that stored model, not against live reality directly. So the correct mental model is not “config minus reality.” It is this: the tool reconciles three things (the config in git, the state file, and a refresh of the live cloud) and every drift bug lives in the gap between two of them.

A plan reconciles those three things, and naming the three gaps is what turns an “unexpected” plan line into a diagnosable one:

The tag surprise above is the second tab: someone edited the tag in the cloud console (the L2 click-op that leaves no artifact), state and cloud now disagree on that field, and the refresh surfaced it as a diff. The tool is faithfully correcting a gap you did not create. The diagnostic loop is the staff move here: read the plan, find the line you did not expect, and reason backward to which pair disagrees. An unexpected ~ or - is a drift question, not a tool bug; the hypothesis to test is “what moved, config or cloud, on this specific resource,” and the answer is almost never “the tool.”

The non-obvious cost lives in the third tab. Because apply enforces the whole config rather than only your edit, drift is not a problem you can defer. The named failure mode is the silent drift revert: an on-call engineer hand-edits a production resource during an incident (bumps a connection limit, widens a security rule) to stop the bleeding, and it works. Weeks later someone applies an unrelated config change. The plan, if read, shows the incident fix being reverted; if not read, the apply quietly restores the resource to what git says, and the symptom from the original incident returns with no obvious cause, because nothing in the recent change touched that resource. The root cause is that the emergency edit violated the boundary the whole module is built on: the config is the source of truth, and any change that does not go through it is a deviation the next apply will erase.

The structure of the three-way reconciliation is the thing to hold in your head, so here it is as one picture, the three inputs and the named gap between each pair:

[config in git] as config
[state file] as state
[live cloud] as cloud

config --> state : config ↔ state gap\n("what to change": healthy diff)
state --> cloud : state ↔ cloud gap\n(DRIFT: apply reverts to config)
config --> cloud : config ↔ cloud gap\n(what apply ENFORCES: whole config)

note bottom of state : the only map from\nlocal name to cloud ID\n(can fall stale silently)

Read the diagram as the diagnostic, not as architecture: every unexpected plan line is a question about which of those three edges moved. The config↔state edge is the work you intended; the state↔cloud edge is drift; the config↔cloud edge is the total blast radius. State sits in the middle as the single map from local names to cloud IDs, which is exactly why the next section treats it not as a convenience but as a liability.

"""Try It 1 starter: name which of the three gaps explains an unexpected plan line.

Given the last value state recorded for a field and the value a refresh read back
from the live cloud, predict which gap this is and return the diagnosis. Fill in the
branch that names drift -- the gap where the cloud moved out from under state.
"""


def diagnose_plan_line(config_value: str, state_value: str, cloud_value: str) -> str:
    """Name which of the three gaps explains an unexpected plan line.
    config_value: what git says the field should be
    state_value:  what state last recorded
    cloud_value:  what a refresh read back from the live cloud
    """
    if state_value != cloud_value:
        # state and reality disagree -- someone changed the cloud out-of-band
        return "TODO: name this gap"
    if config_value != state_value:
        return "config-state gap: an intended change, healthy diff"
    return "no-op: all three agree"


# The unexpected line: config says ml-platform, state recorded ml-platform,
# but a refresh read data-eng from the console.
print(diagnose_plan_line("ml-platform", "ml-platform", "data-eng"))

"""Try It 1 solution: the branch that fires is the drift gap.

When state and cloud disagree, the tool proposes to push the cloud back to what config
says, reverting the out-of-band console edit. Naming the gap is the whole diagnostic:
an unexpected plan line is a question about which input moved, not whether the tool is
broken.
"""


def diagnose_plan_line(config_value: str, state_value: str, cloud_value: str) -> str:
    """Name which of the three gaps explains an unexpected plan line."""
    if state_value != cloud_value:
        return "state-cloud gap (DRIFT): the cloud moved out-of-band; apply reverts it to config"
    if config_value != state_value:
        return "config-state gap: an intended change, healthy diff"
    return "no-op: all three agree"


print(diagnose_plan_line("ml-platform", "ml-platform", "data-eng"))
print(diagnose_plan_line("ml-platform", "data-eng", "data-eng"))
print(diagnose_plan_line("ml-platform", "ml-platform", "ml-platform"))

The source of truth is a liability proportional to trust

The reassuring mental model is that state is only a cache: a performance optimization the tool keeps to avoid re-scanning the whole cloud on every run, and if it were lost the tool would rebuild it from reality. That model makes losing the state file feel recoverable. It is not, and the gap between “cache” and “source of truth” is where the worst IaC incidents live. State earns its power by being the one authority the tool consults, which means anything that damages or duplicates it damages or duplicates the tool’s entire understanding of what exists. There is no quorum and no reconstruction from reality by default; lose the map and the tool concludes it owns nothing.

Here is the cache model breaking. Imagine state lived only on one engineer’s laptop and that laptop died. The config in git is intact, with every resource still declared. But the binding from each local name to its real cloud ID lived only in the lost state file:

# config in git: still declares the production database, fully intact.
# state file: gone (laptop died, no remote backend, no backup).

# Next plan, with an empty state model versus a full config:
#   + cloud_database.lending_club will be created
#       name = "lending-club-prod"
#       ...
# The tool does not see the existing live database; it has no binding to it.
# It proposes to CREATE one. On apply, you get a second, empty database,
# and the original keeps running, now unmanaged. The data is not "rebuilt."

A cache, lost, gets rebuilt from the source. State, lost, does not, because state is the source of the name-to-ID bindings, and nothing else in the system holds them. The next plan sees an empty model versus a full config and proposes to create the whole world fresh; on stateful resources, the live data is invisible to the tool and the proposed create is an empty replacement standing beside the original it can no longer see. The correct model is that state is a single point of failure exactly proportional to how much the tool trusts it: the more it trusts state as the sole authority, the worse the failure when state is wrong, lost, or written by two people at once.

The taxonomy of state-layer failures, ordered by how the damage to the map happens:

• State lost or deleted. The tool forgets it created anything. The next plan sees an empty model versus a full config and proposes to recreate the whole world; on stateful resources, that is mass data loss or, as above, a duplicate empty resource beside an orphaned original. The mitigation is remote, versioned, backed-up state, never a file living only on one laptop, because the failure is unrecoverable from reality alone.
• Concurrent apply against shared state. Two people apply at once and interleave writes to the one state file, corrupting the map. Every subsequent plan is then computed against a garbled model of wrong bindings and half-written field values, so the corruption spreads from one bad moment into every future run. State locking is the guard, and its reach is the trade-off below.
• State and config diverge in version control. Two engineers apply their own conflicting versions of the config one after the other. Each apply succeeds, and the second silently reverts the first’s intent, because the second engineer’s config never had the first’s change. Locking does not prevent this; it only serializes simultaneous writes to one instance, and these writes were never simultaneous.

State locking is a tunable with a reach, and reading the curve is what tells you it is necessary but not sufficient. The three settings are off, on, and on-plus-a-single-apply-path, each closing one more failure than the last:

The signal that tells you which way to move is the count of actors, not the size of the infrastructure. One actor, one sandbox: local state is fine. The second actor of any kind, human or CI, is the signal to turn on locking and route applies through one path, because locking alone leaves the conflicting-versions gap open. The named failure here is the last-writer-wins revert: two engineers branch the config, each adds a different resource, each applies their branch in turn, both applies succeed with green output, and the resource the first engineer added vanishes, because the second engineer’s config never contained it, and apply enforces the whole config. No lock was violated; the writes were sequential. The boundary it violated is the one the book states plainly: when more than one person works on an instance, applies must come from a centralized service against one authoritative config, not from individual laptops with diverging copies.

"""Try It 2 starter: model that apply enforces the whole config.

Two engineers apply conflicting versions of the config one after the other. The cloud
is the set of resources the last apply enforced. Complete apply_config so it models
the real rule -- apply makes the cloud match the config it was given, dropping
anything not in that config -- and confirm the first engineer's resource is gone.
Locking would not have changed this outcome.
"""


def apply_config(cloud: set[str], config: set[str]) -> set[str]:
    """Model one apply: the cloud is made to match the given config.
    apply enforces the WHOLE config, not a merge with what is already there.
    """
    # TODO: return the cloud state after this apply
    return cloud  # placeholder -- does not yet enforce the config


cloud: set[str] = {"database"}
# Engineer A branched and added "cache", applied:
cloud = apply_config(cloud, {"database", "cache"})
# Engineer B branched from the original (no cache), added "queue", applied:
cloud = apply_config(cloud, {"database", "queue"})
print(sorted(cloud))

"""Try It 2 solution: apply enforces the whole config, so last writer wins.

The returned cloud equals the config that was applied -- nothing is merged in from
before. Engineer B's config never contained cache, so the second apply drops it. A
lock would have serialized the two applies, which is exactly what happened anyway; it
never had a chance to prevent this. The fix is process: one centralized apply path.
"""


def apply_config(cloud: set[str], config: set[str]) -> set[str]:
    """Model one apply: the cloud is made to match the given config."""
    return set(config)


cloud: set[str] = {"database"}
cloud = apply_config(cloud, {"database", "cache"})  # Engineer A adds cache
print("after A:", sorted(cloud))
cloud = apply_config(cloud, {"database", "queue"})  # Engineer B never had cache
print("after B:", sorted(cloud))

The provider’s API decides reversibility, not your edit

The intuitive model is that the size of the edit predicts the size of the change: a one-character edit is a small, in-place, reversible change, and a destroy only happens when you explicitly remove a resource or write a scary command. That model is what made the opening incident dangerous, and it is wrong because the tool cannot perform an operation the underlying provider API does not offer. For some fields the provider exposes an in-place update (resize a disk, change a tag) and the change is cheap and reversible. For other fields the value is baked in at creation (a resource’s name, its availability zone, a primary key, an immutable network attachment) and the provider exposes no update operation. The only path to the new value is to delete the old object and create a new one.

Here is the model breaking, in the shape of the opening incident. The edit is one line; the plan is a destroy:

# Before: the database in a parameter group named "pg-v1"
resource "cloud_database" "lending_club" {
  name           = "lending-club-prod"
  parameter_group = "pg-v1"
}

# The one-line edit: rename the parameter group reference.
#   parameter_group = "pg-v2"

# The plan, NOT the "~ update in place" the edit's size suggests:
#   -/+ cloud_database.lending_club (new resource required)
#       ~ parameter_group = "pg-v1" -> "pg-v2"  # forces replacement
#       name = "lending-club-prod"
#   Plan: 1 to add, 0 to change, 1 to destroy.
# destroy-then-create. On a stateful resource, the destroy takes the data.

The edit changed one field. The provider treats that field as create-time-only (Terraform calls this a “force new” attribute) so the tool emits -/+ destroy and then create replacement with # forces replacement on the changed line. For a stateless resource that is invisible and fine; the replacement is functionally identical. For a stateful resource such as a database or a disk, the destroy takes the data with it and the create makes an empty replacement. The correct mental model is that the diff, not the config, is the thing you must learn to read: the config tells you intent, the diff tells you consequence, and only the diff knows that the provider treats the label you changed as create-time-only. Reading the config tells you what you meant; reading the plan tells you what the provider will do.

The discipline is one mechanism with one safe response, so it is principle-plus-rules, not a choice between tools:

• Always read the plan. The plan is the only place the destroy is visible before it happens; the config never shows it.
• Hunt for - and -/+ lines on stateful resources. A -/+ on a database is a stop sign; finding those lines is the single most important thing to read in a plan.
• Before applying a -/+, confirm whether the changed field is create-time-only. If it is, the replacement is not optional; you need a migration plan, not an apply.

For truly load-bearing resources there is a further guard: a deletion-protection flag the tool or the cloud refuses to destroy through. Like locking, it is a tunable with a reach, not a default to memorize, and each setting blocks one more path to the delete than the last:

The signal is the same as everywhere in this module: stateless and cheap to recreate, no flag needed; stateful and load-bearing, prevent_destroy plus, ideally, the provider-side flag, accepting that both turn an intended replacement into a deliberate two-step. The guard exists because the plan is read by humans, and humans on autopilot are the failure mode it covers. The named incident is the autopilot replacement: a routine config cleanup renames a field that happens to be create-time-only on a database, the plan shows -/+ but is approved without anyone hunting the destroy line, and apply deletes the production database to satisfy a rename. prevent_destroy is the seatbelt for exactly the day someone applies on autopilot: it converts a silent catastrophe into a failed apply and an error message.

This is the same discipline as reading a git diff before a commit, or reading a PR before approving it: read the change before letting it land. The difference is blast radius: a bad commit is reverted, while a bad apply on a stateful resource is gone. The scrolly below traces one edit becoming a destroy, from the keystroke through the provider’s capability check to the data loss, so the mechanism is visible end to end.

The scrolly is the whole mechanism in one path: the size of the edit never predicted the size of the change; the provider’s API did, and only the plan made it visible before the data was gone.

"""Try It 3 starter: block a destructive apply on a stateful resource.

Given the plan action and whether the resource is stateful, decide whether to proceed
or stop. Complete should_block_apply so it treats any replacement (-/+) or destroy (-)
of a stateful resource as a stop, and confirm the database rename is blocked while a
tag change on the same database is allowed through.
"""


def should_block_apply(action: str, stateful: bool) -> bool:
    """Return True if this plan action should stop the apply for review.
    action: one of "~" (update in place), "-/+" (replace), "-" (destroy)
    stateful: True if the resource holds data that a destroy would lose
    """
    # TODO: stop on a destroy or replacement of a stateful resource
    return False  # placeholder -- never blocks anything yet


print(should_block_apply("-/+", True))  # rename forces replace on a database
print(should_block_apply("~", True))  # tag change, in place
print(should_block_apply("-/+", False))  # replace of a stateless instance

"""Try It 3 solution: the two symbols that delete a resource are - and -/+.

Combined with stateful, those are the only cases that lose data, so those are the only
cases to block. The database rename is blocked because -/+ on a stateful resource loses
data; the tag change passes because ~ never deletes; the stateless replace passes
because there is no data to lose. This predicate is the plan-reading discipline as code.
"""


def should_block_apply(action: str, stateful: bool) -> bool:
    """Return True if this plan action should stop the apply for review."""
    destroys = action in {"-", "-/+"}
    return destroys and stateful


print(should_block_apply("-/+", True))  # True: rename replaces a live database
print(should_block_apply("~", True))  # False: in-place, no data lost
print(should_block_apply("-/+", False))  # False: stateless instance, safe to replace
print(should_block_apply("-", True))  # True: explicit destroy of a stateful resource

Summary

• State is a stored model of reality, not reality itself: the only map from local resource names to real cloud IDs. Every plan is computed against that model, so a stale model produces wrong, sometimes destructive, plans. An unexpected ~ or - is a drift question (which input moved: config, state, or cloud), not a tool bug.
• A plan reconciles three inputs (config, state, refreshed cloud) and every drift bug lives in the gap between two of them: config↔state is the healthy diff, state↔cloud is drift, config↔cloud is the full blast radius of apply.
• State is a single point of failure proportional to how much the tool trusts it. Lose it and the tool proposes to recreate the world; corrupt it with a concurrent apply and every future plan is wrong. Locking stops simultaneous writes but not two people applying conflicting config versions one after the other; that needs a centralized apply path.
• The provider’s API, not the size of your edit, decides whether a field updates in place or forces a destroy-and-recreate. A one-character edit to a create-time-only field on a stateful resource is silent data loss, surfaced only in the plan as -/+ destroy and then create replacement.
• apply enforces the entire config against reality, so drift anywhere is corrected in any run. Reading the plan and hunting -/-/+ lines on stateful resources is the guard; prevent_destroy and provider-side deletion protection are the seatbelts for the day someone applies on autopilot.

Check your understanding:

• A plan proposes to modify a resource whose config nobody touched this cycle. Which two of the three inputs disagree, and what should you do before applying?
• Two engineers apply their own branches of the config one after the other and the second silently reverts the first’s new resource. Why does state locking not prevent this, and what does?
• Without looking back: a one-line rename produces -/+ destroy and then create replacement on a production database in the plan. What decided that the rename was not an in-place update, and what is the one config-side flag that would make this apply fail instead of deleting the data?