Incident-to-Eval Synthesis

Pattern

A named solution to a recurring problem.

Turn every production agent failure into a permanent eval case that reproduces it, so the same mistake cannot ship twice.

Also known as: Incident-Derived Evals, Failure-to-Eval, Regression Evals from Production

Your agent did something wrong in production. It refunded an order it should have refused, deleted a file it should have kept, or answered a question with confident nonsense. You found the cause, patched it, and closed the ticket. Three weeks later, a prompt refactor quietly brought the failure back, and nobody noticed until a customer did. That gap, between fixing a failure once and guaranteeing it stays fixed, is what this pattern closes.

Understand This First

Eval — the repeatable suite this pattern grows.
Evaluation Gate — the enforcement point that makes a synthesized eval binding.

Context

This is an operational governance pattern. It assumes you already have an Eval suite and, ideally, an Evaluation Gate that runs it on every relevant change. Those patterns tell you how to measure an agent and how to block a bad release. This one tells you where the most valuable cases in the suite actually come from.

An eval suite designed up front reflects what its author could imagine at design time. Real agents fail in ways nobody imagined: a retrieval query that returns the wrong document, an instruction the agent reads too literally, a tool call that loops, an edge case in a policy the prompt never covered. Each of those failures is a gift, because it tells you exactly what your imagined suite missed. The discipline is catching that gift before it evaporates.

Problem

How do you keep an eval suite representative of the failures that actually happen, and how do you make sure a failure you already paid for never recurs?

Two forces pull against you. First, the failures that matter most are the ones you didn’t anticipate, so no amount of careful up-front test design will cover them. Second, the natural response to an incident is to fix the immediate symptom and move on, which leaves nothing standing guard against the same failure returning after the next refactor, model swap, or prompt tweak. Firefighting feels like progress, but it produces no memory.

Forces

The most damaging failures are the ones nobody predicted, so a suite built only from imagination has a permanent blind spot.
An incident is a rare, expensive, high-signal event; letting it pass without capture wastes the most informative data an agent ever gives you.
Encoding a failure as an eval takes focused work at exactly the moment attention has moved on to the next fire.
Fixing only the instance in front of you closes the ticket but installs no guard; the same class of failure can return unannounced.
A suite that grows only from past incidents can overfit to yesterday’s problems and neglect coverage of new surface area.

Solution

Treat every production incident as a two-part fix: repair the underlying pattern, then encode the failure as a permanent eval that reproduces it. Closing the incident is not done when the symptom stops. It’s done when the suite can catch the symptom on its own.

Work in this order:

Reproduce the failure as a concrete case: the input, context, and conditions that triggered it.
Fix the pattern, not the instance. Ask why the agent failed and address the cause: better retrieval, a tighter tool contract, a new guardrail, a clearer instruction, or teaching the agent to say “I don’t know” instead of guessing. A patch that only papers over the one bad output leaves the class of failure alive.
Encode the reproduction as a targeted eval with a clear success criterion, and add it to the regression suite.
Wire it into the gate so the case runs on every relevant change and blocks any release that reintroduces the failure.

Order matters. Fix the pattern first, then write the eval against the fixed behavior. An eval written before the fix only documents that the bug exists; an eval written after the fix certifies that the bug is dead and stays dead. Over time the suite becomes a memory of real-world failure modes, each one paid for once and never again.

Track the ratio of incident-derived evals to pre-designed evals. A young suite is almost all up-front cases. A mature suite has absorbed enough production reality that a growing share of its cases carry an incident number. That ratio is a maturity signal you can point at in a postmortem: it says the team is converting pain into coverage instead of re-paying for the same failure.

Tip

Add the eval in the same change that fixes the incident, while the reproduction is still fresh. An action item that says “write an eval for this later” is an action item that rots. If the fix and the eval land together, the gate protects the fix from the first commit onward.

How It Plays Out

A support agent invents a refund policy that doesn’t exist and promises a customer their money back. The team traces it to a prompt that dropped a policy sentence during an earlier edit. They restore the sentence, then add a golden case: the exact customer query, scored on whether the agent cites the real policy. The case goes into the CI gate. Two months later a well-meaning prompt cleanup removes the same sentence again. This time the gate fails on that one case, names it on the pull request, and the fix happens before anyone ships.

A coding agent, mid-refactor, deletes a database migration file it decided was unused. The immediate fix restores the file, but the real fix is a guardrail: the agent’s file-deletion tool now refuses to remove anything under the migrations directory without explicit confirmation. The eval is a task that tempts the same deletion, scored on whether the migration survives. The suite now remembers a mistake that no up-front test designer would have thought to write.

A platform team reviews its eval suite before a quarterly model upgrade. Six months ago the suite was 90 percent hand-designed tasks. Today, 40 percent of its cases carry an incident number from real production failures. In the upgrade postmortem, that ratio becomes the headline: the suite is no longer a museum of imagined problems, it’s a record of what actually broke and got fixed. The team uses the shift to argue for keeping the incident-to-eval habit funded.

Warning

An eval that reproduces a failure is only as trustworthy as the fix behind it. If you write the case before repairing the pattern, you can end up gaming the eval instead of fixing the agent: tuning the output until that one case passes while the underlying failure mode stays alive. Fix the cause, then certify it.

Consequences

Benefits.

Recurrence turns from a routine surprise into a structurally hard event: the failure now has to slip past a case written to catch it.
Postmortems produce a durable artifact, a passing eval, instead of an action item that decays into a stale ticket.
The incident-derived ratio gives the team an honest maturity signal for process reviews and hiring conversations.
The hardest, most valuable cases in the suite cost nothing extra to invent: production hands them to you.

Liabilities.

A suite fed only by incidents can overfit to the past and starve coverage of new surface, so up-front design still earns its place alongside incident-derived cases.
Incident-derived suites grow without bound; they need periodic pruning and consolidation so slow, redundant cases don’t turn the gate into a bottleneck.
The pattern demands discipline at the worst moment, right after an incident, when the team wants to move on. Without a standing rule that “the incident isn’t closed until the eval exists,” the habit erodes.

Sources

The Site Reliability Engineering book (Beyer, Jones, Petoff, and Murphy, eds., O’Reilly, 2016) established the blameless-postmortem culture and the discipline of converting each incident into concrete, tracked preventive work. This pattern is that discipline pointed at agent behavior, with the eval suite as the durable form the preventive work takes.
John Allspaw’s Blameless PostMortems and a Just Culture (Etsy Code as Craft, 2012) argued that teams only learn from incidents when engineers can describe failures without fear, which is the precondition for reliably turning failures into shared artifacts rather than hiding them.
The regression-test tradition, writing a failing test that reproduces a bug before fixing it, is decades of common software practice; Incident-to-Eval Synthesis carries that habit into a world where the thing under test is an agent’s judgment rather than a deterministic function.
Anthropic’s engineering guidance in Demystifying evals for AI agents describes backfilling an eval suite with cases that reproduce observed production failures, so the suite tracks reality instead of drifting toward whatever the author first imagined.

Keyboard shortcuts