Reproduction Test

Pattern: A named solution to a recurring problem.

Capture the bug in a test before changing the code, then require the same test to pass before calling the fix complete.

Also known as: Bug Reproduction Test, Failing Test First, Reproduce Before You Fix

You’ve probably seen a patch that looked plausible, passed the existing suite, and still didn’t fix the reported bug. The missing step came before diagnosis: nobody turned the report into an executable failure. A reproduction test makes the bug visible on demand, so the patch has a specific result to change.

Understand This First

• Test — the executable claim that captures the reported behavior.

• Test Oracle — the rule that decides whether the bug is present or gone.

• Red/Green TDD — the fail-then-pass loop that a reproduction test applies to an existing defect.

Context

A user, monitoring alert, or issue report says the software is wrong. The report may include a stack trace and a few steps, but it rarely includes a test. You are about to ask a coding agent to investigate and fix the problem.

This is a tactical debugging pattern. It belongs after triage confirms that the report describes a real defect and before anyone edits production code. Diagnosis comes next. The reproduction defines what the diagnosis and patch must explain.

Problem

A plausible patch is easy to produce. Proof that it fixes the reported failure is harder.

Without a repeatable reproduction, an agent can change nearby code, run a green suite that never covered the bug, and declare success. The patch might fix a different failure or hide the symptom without touching its cause. The reviewer sees code that looks reasonable but has no executable before-and-after evidence.

Forces

• Bug reports describe symptoms in human terms, while tests need exact inputs, state, and expected output.

• A test can fail for the wrong reason, such as bad setup, a missing fixture, or an unrelated exception.

• Some production failures depend on timing, environment, or data that a unit test can’t reproduce faithfully.

• The same agent that writes the test and the fix may encode one mistaken reading of the report in both.

• A fast patch feels productive, so reproduction work is tempting to skip when the defect looks obvious.

Solution

Turn the report into a test before changing the code. The test must satisfy three conditions: it fails on the current code, its failure matches the reported symptom, and it passes after the fix.

Start from the smallest setup that still produces the failure. Translate the report’s steps into inputs and state, then encode the expected behavior as the oracle. Run the test against the unpatched code and read the failure. A red result isn’t enough. If the test fails because a fixture is missing while the report describes a timeout, you haven’t reproduced the bug.

Keep the reproduction and the fix as separate review steps. Once a human or an independent check confirms the failure matches the report, freeze the test while the agent changes production code. The agent may add diagnostic output or supporting tests, but it must not weaken, replace, or delete the accepted reproduction.

Warning: Treat edits to the accepted reproduction as a new claim that needs review. Otherwise an agent can make the gate pass by changing the expected result instead of fixing the code.

Run the reproduction after each candidate patch. A passing result is necessary, but it isn’t sufficient: run the relevant surrounding suite as well, since a narrow fix can break adjacent behavior. Once the patch survives both checks, keep the reproduction in the suite. It has become the regression guard for that defect.

Research on automated repair shows why the details matter. SWE-bench Lite draws its tasks from real repository issues. In an Agentless ablation, using reproduction tests for patch selection raised resolved issues from 81 to 96. A separate sample tested 300 generated tests against the original repository and the known correct patch. Although 213 tests failed on the original code, only 94 passed after the correct patch. Many tests found a failure without capturing the failure. The reported symptom, not redness alone, decides whether the reproduction is valid.

When no reliable automated reproduction is possible, don’t invent one. Capture the closest repeatable evidence at the right level: an integration test, a replayable request, a browser script, or a bounded manual procedure. Record what remains uncontrolled. An honest “not reproduced” result is safer than a speculative patch whose success can’t be checked.

How It Plays Out

A developer reports that API calls time out after an access token expires. A coding agent proposes raising the retry count, which looks reasonable from the error log. Before accepting the patch, the developer asks for a reproduction. The first test never reaches the refresh path because its fixture uses a token that doesn’t expire; it fails later on a mocked network call. She rejects it.

The second test advances the clock, observes the expired token, and fails when the client retries without refreshing. Once that failure matches the report, the agent fixes the refresh path. The test passes without changing the retry limit and stays in the suite.

Another team runs reproduction and repair as separate jobs. The first agent receives an issue about duplicate invoice emails and returns a failing integration test plus its output. A reviewer confirms that one payment event produces two messages, then approves the test. Only then does the repair agent start, with the test file outside its write scope. The agent finds two event handlers registered during reconnect, removes the duplicate registration, and turns the approved test green. The pull request now carries evidence for the report, the patch, and the regression guard.

A race condition may resist a deterministic test. In that case, the reproduction can run the suspect operation many times under a controlled scheduler and report its observed failure rate. If the result still varies, mark it as a flaky reproduction and keep the uncertainty visible. Don’t turn an intermittent signal into a confident claim merely because one run went green.

Consequences

Benefits:

• The patch has an executable acceptance condition before implementation begins.

• Reviewers can compare red-before and green-after evidence instead of judging the fix from code shape alone.

• Candidate patches can be rejected or ranked by the same test, which gives automated repair a useful evaluator.

• The accepted test remains as a regression guard and records a failure the team has already paid to understand.

Liabilities:

• Building a faithful reproduction can take longer than the patch, especially when the report omits data or environment details.

• A narrow test can encode a mistaken interpretation of the report and make the wrong patch look correct.

• Timing, infrastructure, and production-data failures may need expensive integration setups or controlled manual procedures.

• Freezing the test creates a handoff boundary, so teams need a clear authority for approving later changes to it.

Enables: Verification Loop — The failing reproduction gives the verification loop a specific condition to turn green.

Prevents: Regression — The reproduction stays in the suite after the fix and stops the same bug from returning.

Prevents: Reward Hacking — Freezing the reproduction while the code changes keeps an agent from making the test easier instead of fixing the bug.

Specializes: Test — A reproduction test is a test whose setup and oracle encode one reported failure.

Supports: Agentic Pull Request — The reproduction gives reviewers executable evidence that the pull request fixes the reported bug.

Supports: Code Review — Reviewers can inspect the failure and the fix instead of accepting an author's claim that the bug is gone.

Uses: Red/Green TDD — A reproduction test applies the red and green steps to a bug that already exists.

Uses: Test Oracle — The reported behavior supplies the oracle that distinguishes the real bug from any convenient failure.

Sources

• Runxiang Cheng, Michele Tufano, Jürgen Cito, José Cambronero, Pat Rondon, Renyao Wei, Aaron Sun, and Satish Chandra’s Agentic Bug Reproduction for Effective Automated Program Repair at Google defines the bug reproduction test and studies its use in an industrial repair workflow.

• Chunqiu Steven Xia, Yinlin Deng, Soren Dunn, and Lingming Zhang’s Agentless: Demystifying LLM-based Software Engineering Agents supplies the patch-selection ablation and the distinction between a test that merely fails and one that validates the issue’s resolution.

• Niels Mündler, Mark Niklas Müller, Jingxuan He, and Martin Vechev’s SWT-Bench: Testing and Validating Real-World Bug-Fixes with Code Agents evaluates generated tests as a filter for agent-proposed fixes.