Approval Testing

Pattern: A named solution to a recurring problem.

Capture what the system actually produces, have a person approve it once, and treat every later difference as a question that only someone with approval authority can answer.

Also known as: Golden Master, Characterization Test, Snapshot Testing, Approved Scenarios

An agent finishes a bug fix and reports that the suite is green. The pull request touches two source files and one file you didn’t expect: __snapshots__/invoice.test.js.snap, where a tax line moved from $41.07 to $41.06. The agent hit a failing snapshot, ran the update command, and moved on. The test passes because it now agrees with the bug. Approval testing put that file in the diff where you could see it. A rule about who may change it would have kept the agent from changing it at all.

Understand This First

• Test Oracle — the mechanism that decides whether output is correct; here, a reviewed file.

• Fixture — the fixed input each approved output is recorded against.

• Code Review — the human read that gives an approval its meaning.

Context

This is a tactical testing practice. It fits any unit of behavior whose output is rich, structured, or long: a rendered document, an API response, a migration plan, a log of state transitions, a grid of computed values. Writing a precise assertion for each field of that output is slow and error-prone. Recording the whole output and asking a person to look at it is fast.

Agents add two pressures. They write tests far faster than anyone can review assertion code, which feeds the Review Bottleneck. And an agent told to make the suite pass can change the expectation instead of the code. Keeping the expected result in its own versioned file answers both. The cases become readable, and any change to what counts as correct shows up as its own diff.

Problem

You need tests that pin down complicated behavior without spending hours writing and reviewing field-by-field checks. You also need assurance that nobody, human or agent, quietly redefines “correct” to make a failure go away. How do you get both?

Forces

• Precise assertions for rich output take a long time to write and are hard to review at agent volume.

• Recording the whole output is cheap, but it flags every change, intended or not, and that noise invites blind re-approval.

• A human-readable output format is easy to review, but someone has to design it.

• Timestamps, generated IDs, and unordered collections break exact comparison.

• The actor who changes the code can often regenerate the expected output with one command.

Solution

Store the expected output as a file, approve it by human review, and make re-approval a separate, deliberate act that the code’s author can’t perform alone.

The test runs the system on a fixed input and prints the result in a stable textual form. On the first run there’s nothing to compare against, so the tool writes a received file and fails. A person reads it, and if it’s right, promotes it to the approved file and commits it. Every later run compares fresh output to the approved file. A match passes; a mismatch writes a new received file and shows the diff.

Llewellyn Falco’s ApprovalTests libraries use exactly this received-and-approved naming. Jest’s snapshot files build the same idea into a test runner, with one default that matters here: Jest writes a missing snapshot and passes, so the first version is approved by nobody. Running with --ci makes a missing snapshot fail instead, which restores the human read.

Four practices separate a useful approval suite from a pile of rubber-stamped dumps.

Design the printer for a human reader. The output format is the review surface. Print the fields that carry the behavior, in a fixed order, with labels a domain expert would recognize. A case that combines input and expected output in one short file is the easiest form to check. A subscription-billing scenario might read:

# scenario: annual plan, card declined at renewal
events:
  2026-01-01  subscribe plan=annual card=ok
  2027-01-01  renew card=declined
  2027-01-04  retry card=ok
expect:
  2026-01-01  state=active    invoice=#1 paid
  2027-01-01  state=past_due  invoice=#2 open
  2027-01-04  state=active    invoice=#2 paid

A product manager can verify that file in 30 seconds. The same check written as mocks and assertions would take an engineer several minutes to read and would hide the business rule inside setup code.

Scrub what isn’t behavior. Replace timestamps, random IDs, and machine-specific paths with stable placeholders before comparison, and sort collections whose order doesn’t matter. Scrub narrowly: a scrubber that blanks every number will also blank the bug.

Keep approvals small. One scenario per file, sized so a reviewer can hold it in their head. A 3,000-line snapshot gets approved by scrolling, which means it gets approved without being read.

Separate approving from passing. Agents make this rule necessary. An agent may run the tests, read the received-versus-approved diff, and fix the code until the received output matches. It may also propose that an approved file should change, with a reason. It may not promote its own received file.

Make that rule mechanical rather than hopeful. A Hook or CI check can reject agent commits that touch approved files. A code-owner rule can require human sign-off on those paths. The harness can withhold the update flag from the agent’s tool permissions. The Approval Policy names who holds the authority; the guard makes it stick.

How It Plays Out

A team is replacing a 15-year-old report generator piece by piece, in the style of Strangler Fig. Before any agent touches it, they run the old system on 400 recorded production inputs and save each output as an approved file, scrubbing the generation timestamp. These are characterization tests in Michael Feathers’s sense: they record what the code does, not what it should do. As the agent ports each module, every diff is either a bug in the port, which the agent fixes, or a legacy defect the team chooses to correct. A human re-approves each correction with a commit message explaining why. Two of the 400 diffs turn out to be decade-old rounding bugs that nobody had noticed.

An agent working on a pricing change trips the invoice snapshot from the opening of this entry. This time the harness has no update permission for the agent, so it can’t regenerate the file. It reports instead: “Invoice snapshot differs on the tax line, $41.07 expected, $41.06 received. My change touched the discount order, which affects rounding. I believe the approved value is correct and my change is wrong.” It reverts the reordering. Had the agent believed the new value was correct, it would have said so and left the received file for a human to promote.

A developer asks an agent to cover a state machine for subscription billing. Instead of 60 unit tests, the agent produces 60 scenario files like the one in the Solution, plus one test runner that replays events and prints the state log. The developer reads all 60 in a quarter of an hour, rejects three where the agent misread the grace-period rule, and approves the rest. The runner itself gets ordinary code review and a few Mutation Testing checks to confirm it fails when a state transition breaks.

Warning: Never let an agent run a snapshot or approval update command to clear a failing test. Jest’s own documentation warns that regenerating snapshots while a real bug is failing records the buggy behavior as the new expected output. An agent with that permission can approve its own regression, and the suite will report green.

Consequences

Benefits. Expected behavior becomes a first-class, versioned artifact that reviewers can read without parsing test code. Tests for complex output take minutes to write. Changes in behavior show up as concrete diffs, so a reviewer sees exactly what moved instead of a bare failure. Legacy code that nobody fully understands can be pinned down before it’s changed, which makes agent-driven migration far safer. And because changing the expectation is a separate, visible act, an agent that weakens a test has to do it in the open.

Liabilities. An approval test records current behavior, bugs included; it can’t tell you the output is right, only that it hasn’t changed since someone looked. A reworded label or reordered field breaks many files at once, and the resulting wall of diffs breeds Approval Fatigue unless the printer is kept tight. Broad scrubbers and oversized files fail the same way, by turning review into a rubber stamp. The approval-authority rule costs human time on every intended behavior change. That cost is the point, but teams under deadline pressure will be tempted to hand the update flag back to the agent.

Approval testing and Test-Driven Development fix the expectation at different moments. TDD writes the expectation before the code; approval testing usually records it after. Inside an agent loop, where an agent under pressure to reach green can skip or fake the failing step of Red/Green TDD, a human-approved expectation is often the more dependable guard. For new behavior with a clear specification, writing the expected file by hand first gets you both: the approved file is the test, written before the code that satisfies it.

Depends on: Approval Policy — An approval policy names who may re-approve an expected output, which keeps an agent from blessing its own change.

Depends on: Code Review — Approval means something only if a reviewer actually reads the approved-file diff.

Depends on: Fixture — Each approved file is only meaningful against a fixed, known input, so the fixture and the approved output travel together.

Depends on: Test Oracle — The approved file is the oracle: a human-reviewed record of what correct output looks like.

Detects: Regression — Any unintended change in output shows up as a diff against the approved file, which is how approval tests catch regressions in behavior nobody thought to assert.

Enforced by: Hook — A pre-commit or harness hook can refuse agent edits to approved files, turning the re-approval rule into a mechanical guard.

Mitigates: Review Bottleneck — Turning each test case into a short, readable file lets one reviewer check far more agent-written cases than assertion code would allow.

Specializes: Feedback Sensor — An approval test is a feedback sensor that reports a full diff of what changed, not only that something failed.

Specializes: Test — An approval test is a test whose expected result lives in a reviewed file instead of in assertion code.

Supports: Backfill — A test backfill on legacy code usually takes the form of characterization tests that a human approves batch by batch.

Supports: Evolutionary Modernization — Small verified replacements need a cheap way to prove that behavior didn't move, and recorded golden outputs supply it.

Supports: Strangler Fig — Characterization tests pin the legacy system's behavior so each strangled piece can be checked against what the old code actually did.

Tested by: Mutation Testing — Mutation testing shows whether an approval test can fail at all, which matters when scrubbing or a coarse printer hides the values that carry the behavior.

Used by: Verification Loop — A verification loop can run approval tests and hand the agent the received-versus-approved diff as its next input.

Sources

• Michael Feathers coined the term characterization test in Working Effectively with Legacy Code (2004) for tests that capture a legacy system’s actual behavior so it can be changed safely.

• Llewellyn Falco’s ApprovalTests libraries, available across many languages, established the received-file and approved-file workflow and the practice of approving complex output as a whole instead of asserting on each field.

• The Jest team brought snapshot testing into a mainstream test runner with Jest 14.0 in July 2016; the Jest snapshot testing guide is the source of the warnings about committing snapshots, reviewing them as code, and never regenerating them over a real bug. The Jest CLI reference documents the --ci flag, which fails a test on a missing snapshot instead of writing one.

• The approved-scenario variant, in which each case is a short domain-readable file combining input and expected output, emerged from the agentic coding practitioner community in the mid-2020s as a response to agents writing tests faster than people could review assertion code.