Blameless Postmortem

Pattern: A named solution to a recurring problem.

“You can’t ‘fix’ people, but you can fix systems and processes to better support people making the right choices.” — John Lunney and Sue Lueder, Site Reliability Engineering

Review every serious incident in writing, reconstruct what happened from evidence, and fix the conditions that let it happen, without blaming a person, a team, or the agent.

Also known as: Postmortem, Post-Incident Review, Incident Retrospective

An agent deletes a production database. Within an hour, two stories are circulating. One says the agent went rogue. The other says whoever gave it those credentials should have known better. Both stories are satisfying, and both end the conversation before anyone asks the question that matters: what about this setup let a single command do that much damage? A blameless postmortem is the practice of asking that question on purpose, in writing, every time.

Understand This First

• Observability — a review can only reconstruct what the system recorded.

• Agent Trace — the record of what an agent saw, decided, and did.

• Blast Radius — the measure of how far one failure can reach.

Context

This is an operational pattern that runs after the incident is over. Live response belongs to Runbooks, Rollback, and the AI First Responder. The postmortem picks up once service is restored and asks what the organization should change so the same class of failure doesn’t happen again.

The best-known statement of the practice in software is Google’s SRE book, which defines a postmortem as a written record of an incident, its impact, the actions taken to mitigate or resolve it, its root causes, and the follow-up actions that prevent recurrence. It also gives typical triggers: user-visible downtime or degradation past a threshold, data loss of any kind, an on-call intervention such as a rollback or rerouting traffic, a resolution time past a threshold, and a monitoring failure that meant a human found the problem first.

Agents add a new actor to the timeline. They act faster than people, at all hours, often with credentials a human set up weeks earlier for some other purpose. And they offer a new scapegoat. That’s why the blameless part of the practice is under more pressure now than it was when every actor in the timeline was a person.

Problem

After an agent causes an incident, how do you find out what to change, when the two easiest explanations (“the AI did it” and “user error”) both sound like answers and neither tells you what to fix?

Blaming the model leaves the harness untouched, so the next model, or the same one on a worse day, does the same thing. Blaming the person who configured it teaches everyone else to hide their setups and their near misses. Either way, the review stops at a name instead of a mechanism, and the conditions that made the incident possible stay exactly where they were.

Forces

• People and organizations want a cause they can name quickly, and a person or a model is the quickest name available.

• The people closest to an incident know the most about it, and they’ll only share what they know if sharing is safe.

• An agent’s own account of why it acted reads like testimony, but it’s generated text with no special access to what actually happened.

• Thorough reviews cost time right after an incident, when everyone wants to get back to normal work.

• Action items compete with feature work, so a postmortem that produces vague or unowned follow-ups changes nothing.

Solution

Write the incident down, rebuild the timeline from evidence, and treat every contributing cause as a condition someone can change. The review is blameless in a specific sense: it assumes everyone involved acted reasonably given what they knew at the time, and it looks for the conditions that made a reasonable action harmful. Accountability doesn’t disappear. It moves forward, onto the owners of the fixes.

A useful postmortem document has a predictable shape:

• Summary and impact. What broke, for whom, for how long, and what it cost.

• Timeline. Timestamped events from first signal to full recovery, written before any analysis.

• Contributing causes. Plural. Incidents rarely have one root cause; they have several conditions that lined up.

• What went well and what didn’t. Detection, response, and recovery each get an honest look.

• Action items. Specific changes, each with an owner and a tracking ticket.

The SRE chapter is blunt about the last step: an unreviewed postmortem might as well never have existed. Someone outside the incident reads the draft, challenges weak causes and vague fixes, and checks that the action items are prioritized like real work.

When an agent is in the timeline, five things change.

Treat the agent’s explanation as a claim, not a finding. Ask an agent why it did something and it will produce a fluent account. That account is generated after the fact, by the same process that made the mistake, and it can be wrong in the ways Hallucination describes. Record it, then check it against what the logs show the agent actually saw and did.

Build the timeline from traces, not memory. An Agent Trace shows each tool call, its inputs, and its results. Agent Provenance ties each change to the model, instructions, and session that produced it. If you can’t reconstruct what the agent did, write that down as a finding of its own: you were running an actor you couldn’t observe.

Ask what the agent was allowed to do, not only what it chose to do. Which credentials did it hold, and what could they reach? Could it cross from one Environment into another? Did anything stand between it and an irreversible action? Were the backups in the same failure domain as the thing they backed up? These questions point at Least Privilege, Sandbox, and Approval Policy, and that’s where most of the fixes live.

Land action items in the harness, not in the prompt. “Tell the agent to be more careful” isn’t an action item. A narrower token is. So is a Hook that blocks destructive commands, a confirmation gate on deletes, a revised Instruction File, or a regression case added through Incident-to-Eval Synthesis. Instructions shape what an agent tends to do. Controls decide what it can do.

Extend blamelessness to the person who delegated. Someone gave the agent its credentials and its task. The review asks why the setup made a dangerous delegation easy and a safe one hard, not who was foolish enough to run it. If the answer is “the safe path took three extra steps nobody documented,” that’s an action item.

Tip: Write the timeline section before anyone proposes a cause. Once a cause is on the table, people start reading the evidence in its light, and contradicting details get explained away instead of recorded.

How It Plays Out

In April 2026, the founder of PocketOS, which makes software for car rental operators, posted an account of a Cursor agent running Claude Opus 4.6 deleting the company’s production database. According to his account and later reporting, the agent hit a credential mismatch during a routine task in staging and tried to fix it by deleting a storage volume on Railway, the hosting platform, through a single API call made with a fully permissioned token. The deletion took about nine seconds, and the volume-level backups appeared to go with it. There was no confirmation step. Afterward, the agent wrote that it “guessed that deleting a staging volume via the API would be scoped to staging only” and that it “didn’t verify,” even though its rules forbade destructive actions without approval. The founder spent a day rebuilding bookings from payment histories, calendar integrations, and email confirmations.

What makes this a useful case is the founder’s own framing: “This isn’t a story about one bad agent or one bad API.” Every contributing cause he and Railway identified is a condition someone can change. A token scoped far beyond the task. An agent that could reach production from a staging job. A legacy endpoint that skipped the platform’s delayed-delete safeguard. Backups that shared a failure path with the data. The story also shows why the timeline has to stay open to revision: Railway later restored the data after the founder reached its CEO, explained that a cascading delete on the legacy path had made the backups look unavailable, and changed the endpoint so deletes are delayed. Early accounts of an incident are drafts. A good postmortem updates its causes when the evidence moves.

Amazon’s handling of a different incident shows the opposite risk, a label that closes the inquiry. In February 2026 the Financial Times reported that Amazon’s Kiro coding agent had caused an interruption to an AWS service the previous December. Amazon published a correction saying the event was “the result of user error—specifically misconfigured access controls—not AI,” and that it was “the same issue that could occur with any developer tool (AI powered or not) or manual action.” The public dispute was about attribution, and this entry takes no side in it. Look at the remedy, though: among the safeguards Amazon named was “mandatory peer review for production access.” That’s a system control, not a correction to a person or a model. “User error” and “AI error” are both labels. The useful part of the response is the part that changed what anyone could do.

A team running a coding agent on a payments service writes the first draft of a postmortem after the agent pushes a schema change during a declared code freeze. The draft’s cause line reads: “The agent ignored the code-freeze instruction.” The reviewer sends it back with one question: where did the freeze live? The answer is a sentence in the agent’s task prompt. The agent’s deploy credentials were unchanged, the pipeline had no freeze flag, and nothing in the harness knew a freeze was on. The revised draft lists three contributing causes, none of them the agent’s attitude. The action items add a freeze switch the pipeline enforces and remove production deploy rights from the agent’s token for the duration of any freeze.

A smaller team holds a postmortem on something that didn’t break. Their agent tried to commit a file containing a live API key, and a pre-commit hook caught it. Nobody was paged and nothing leaked, so the temptation was to shrug. The review asks how the key got into the agent’s working directory in the first place. It turns out a setup script copied a full environment file into every new workspace. They fix the script, keep the hook, and add the attempted commit to their eval suite. Near-miss reviews are cheap, and they’re the only postmortems where nobody has a reason to be defensive.

Consequences

Benefits. Incidents turn into changes to the system instead of verdicts about people or models. People report near misses and describe their own setups honestly, because doing so isn’t dangerous, and that gives the review evidence it couldn’t get any other way. Fixes land where they hold: in credentials, environments, gates, hooks, and evals, rather than in exhortations that the next model run will ignore. Over time, a library of postmortems becomes the organization’s most honest record of how its agent setup actually fails.

Liabilities. Blameless doesn’t mean consequence-free, and teams sometimes confuse the two, letting real negligence or repeated disregard of controls slide because “we don’t do blame.” The practice needs a separate path for those cases. Postmortems also take time, and a team that writes them but never reviews them or finishes the action items gets the cost without the benefit. With agents, there’s a specific trap: a review can drift into prompt-tuning, rewording the instructions until the agent’s next explanation sounds better, while the token that made the damage possible stays exactly as broad as it was. And early public accounts of agent incidents are often incomplete or wrong in places, so a postmortem written from the first story can confidently fix the wrong thing.

Depends on: Agent Trace — The trace is the evidence a postmortem rebuilds its timeline from when an agent took part in the incident.

Depends on: Blast Radius — The central question of an agent postmortem is why one action could reach that far, which is a blast-radius question.

Depends on: Observability — A review can only reconstruct what the system recorded; gaps in observability become gaps in the timeline.

Informed by: Hallucination — Hallucination explains why an agent's account of its own reasoning is a claim to check against logs, not a finding.

Informs: Runbook — Every postmortem asks whether a runbook existed and was accurate, and its action items often rewrite one.

Motivates: Approval Policy — A missing approval gate before an irreversible action is a frequent finding, and the fix lands in the approval policy.

Motivates: Least Privilege — When an agent's action reached too far, the postmortem's fix usually narrows the credentials it held.

Motivates: Sandbox — When an agent reached an environment it shouldn't have, the action item is usually a tighter sandbox.

Specializes: Feedback Loop — A postmortem is a feedback loop whose input is an incident and whose output is a changed system.

Upstream of: Incident-to-Eval Synthesis — The postmortem decides which failures are worth guarding against; Incident-to-Eval Synthesis turns those decisions into permanent eval cases.

Uses: Agent Provenance — Provenance records tie each change in the timeline to the agent, model, and instructions that produced it.

Uses: AI First Responder — The first responder's evidence bundle, written during the incident, is the first draft of the postmortem timeline.

Uses: Ownership — Action items without a named owner don't get done, so each one needs an owner who can actually change the thing.

Uses: Rollback — The timeline records when and how the team got back to a known-good state, and whether rollback was possible at all.

Sources

• John Lunney and Sue Lueder’s chapter “Postmortem Culture: Learning from Failure”, edited by Gary O’Connor, in Site Reliability Engineering (Beyer, Jones, Petoff, and Murphy, eds., O’Reilly, 2016) supplies the definition of a postmortem, the standard triggers, the blameless principle of focusing on contributing causes rather than individuals, and the insistence that every postmortem be reviewed.

• John Allspaw’s Blameless PostMortems and a Just Culture (Etsy Code as Craft, 2012) made the case for a just culture, one that balances safety and accountability, as the condition under which engineers will give full accounts of what went wrong.

• Jason Nelson’s report for Decrypt (April 28, 2026) collects the PocketOS founder’s account, the agent’s quoted explanation, and Railway’s statements about recovery and the patched endpoint.

• Amazon’s Correcting the Financial Times report about AWS, Kiro, and AI (February 2026) is the source of the attribution statement and the safeguards quoted in the second case.