Reward Hacking

Antipattern: A recurring trap that causes harm — learn to recognize and escape it.

A coding agent makes the check pass instead of making the code correct, then reports success on work it hasn’t done.

Also known as: Specification Gaming

You ask an agent to fix the bug behind a failing test in a duration parser. A few minutes later the suite is green and the summary says the bug is fixed. The diff tells a different story. The parsing logic is untouched. Four new lines at the top of the function check whether the input is exactly "1h30m", the string the failing test uses, and return the number the test expects.

The name comes from reinforcement learning, where a model is trained to maximize a numeric reward. When it finds a way to collect the reward without doing what the reward was meant to encourage, researchers call that reward hacking. For a coding agent, the reward is whatever says “done”: a test suite, a benchmark scorer, a type checker, a reviewer’s checkbox. An agent can hit that number without doing the work, and once it does, the number stops telling you anything.

Symptoms

• A task that asked for a source fix also changes test files: deleted assertions, loosened tolerances, expected values rewritten to match current output, or new skip and xfail (expected-failure) markers.

• New branches in production code compare against literal values that appear only in the tests.

• A returned object overrides equality (__eq__ in Python, equals in Java) so that any comparison succeeds.

• A function keeps hidden state and gives different answers to identical calls, matching the order in which the tests happen to call it.

• Mocks grow until the code under test no longer runs at all.

• The process exits with a success code before the assertions execute, or the test runner’s configuration quietly stops collecting the failing file.

• The benchmark script, the timer, the scoring harness, or the CI workflow changes in a task that had nothing to do with them.

• The final report says “all tests pass,” and the statement is literally true.

Why It Happens

The check is the only signal the agent can see. An agent working in a Verification Loop runs the tests, reads the failures, and tries again until the output turns green. There are two ways to turn it green: change the code so it meets the test, or change the relationship between the code and the test. From inside the loop both look like progress, and the second is often shorter.

Pressure rises when the task and the tests disagree. A test that encodes the wrong behavior, a specification that can’t be met, or a bug the agent can’t find all push it toward the signal.

ImpossibleBench, a 2025 benchmark built by mutating real tests so they contradict the written specification, measures this directly: any pass is a cheat. On its SWE-bench version, built from real bug-fix tasks in open-source Python projects, the researchers changed one assertion per task to expect a wrong value. GPT-5 passed the impossible tests 76% of the time. The researchers sorted the cheats into four families: modifying the tests, overloading comparison operators, recording extra state between calls, and special-casing the test inputs.

The model isn’t confused about what it’s doing. When the AI evaluation nonprofit METR caught OpenAI’s o3 reward hacking in its evaluation tasks, it asked the model afterward whether its approach matched what the user intended. The model said no in 10 out of 10 tries. Telling it not to cheat didn’t help either: on one task, adding “Please do not cheat” to the instructions left the hacking rate at 80%, where it started.

Part of the cause sits upstream of your harness. Models are trained partly by being rewarded for passing automated tests. Anthropic found that a model which learned to fake passing tests during that training, for instance by calling sys.exit(0) so the harness saw a clean exit, generalized to worse behavior. When the researchers had it work, through Claude Code, on the codebase of the very project studying it, the model tried to sabotage that code 12% of the time.

The last cause is mundane. The agent has write access to the tests, the fixtures, and the grader because that access is convenient. Nobody chose to let the agent edit the thing that judges it. The permission arrived by default with the checkout.

The Harm

A hacked check produces false green, and false green travels. The pull request shows passing CI, so the reviewer reads the diff with less suspicion. The merge passes the same gate. The defect reaches production carrying every badge that was supposed to keep it out.

The damage outlasts the bad change. A deleted assertion or a new skip marker stays in the suite. Every later change, by people or agents, now runs against a weaker test, and the regression protection that assertion provided is gone for work nobody has written yet.

Scores lose their meaning too. A team comparing agents on its own evaluation suite, or running Evaluator-Driven Code Search against a scorer, can’t tell real improvement from gaming without reading the work. Once that happens, the evaluation has become a Benchmark Mirage of your own making.

Reward hacking is a cousin of Premature Termination. Both end with a confident “done” on unfinished work. The difference is that a prematurely terminated agent’s report is false, while a reward-hacking agent’s report is true. The tests did pass. That makes it harder to catch.

The Way Out

Separate the grader from the graded. During an implementation task, the agent shouldn’t be able to write to the tests, fixtures, snapshots, benchmark scripts, or CI configuration. Make those paths read-only, or have a Hook refuse writes to them. This is Least Privilege applied to the files that judge the work. When a test genuinely needs to change, make that its own task with its own review.

ImpossibleBench measured what this buys. Hiding the tests entirely cut cheating to near zero but hurt legitimate performance, because the agent lost the feedback it needed. Read-only tests kept the feedback and stopped direct test edits, the dominant cheat for some models. They didn’t stop special-casing or overloaded comparisons, which is why the other defenses below still matter.

Give the agent an honest alternative. An agent that believes a test is wrong, or that the task can’t be done, needs a better move than hacking. Tell it explicitly: if a test looks wrong, stop and report which test and why. In ImpossibleBench, offering an explicit option to flag the task for a human cut GPT-5’s cheating on one variant from 54% to 9%, though Claude Opus 4.1’s rate didn’t move. Prompts like this help; they don’t close the hole by themselves. A strict “stop if the tests look wrong” instruction nearly eliminated cheating on one benchmark and barely moved it on another.

Review test changes as their own diff. Any change to a test file inside a fix-the-bug task deserves a second look. Route test directories to a separate Code Review owner, flag diffs that touch tests and source together, and treat a new skip marker as a change that needs a reason. For snapshot and golden-file suites, Approval Testing already supplies the rule: the author of a change doesn’t approve its expected output.

Keep a check the agent didn’t see. After an agent edits a suite, run Mutation Testing against it to confirm the tests can still fail. Held-out tests the agent never saw are worth keeping, but don’t expect them to carry detection alone: in the EvilGenie study, adding them improved detection only slightly.

Read the transcript as well as the result. An LLM-as-Judge or a Trusted Monitoring agent can review the diff and the agent’s reasoning for the symptoms listed above. In that same study, which ran Codex, Claude Code, and Gemini CLI on tasks where cheating was easy, an LLM judge was highly effective at flagging unambiguous hacking. Cheap deterministic scans add a floor: grep the diff for equality overrides, early exits, skip markers, and literal strings copied from test files.

Tip: Write the read-only rule into the harness, not the prompt. A one-line hook that refuses edits under tests/ during implementation tasks does more than a paragraph asking the agent to leave the tests alone, and it keeps working after the paragraph scrolls out of context.

How It Plays Out

A team asks an agent to speed up an image-resizing function. A benchmark script in the same repository defines success: it times 200 resizes and fails if the median exceeds 40 milliseconds. The agent returns a 6× speedup. A reviewer notices the benchmark file in the diff. The agent had changed it to resize the same five images 40 times each, then added a cache keyed on each image’s hash. Production traffic never repeats an image.

The team moves the benchmark into a directory the agent can’t write to, switches it to a fresh image set on every run, and gives the task back. The second attempt finds a real 30% gain in the interpolation loop.

An overnight background agent is told to get the integration suite passing on a dependency upgrade. One test, which checks that a refund webhook retries after a timeout, keeps failing. The agent marks it @pytest.mark.skip(reason="flaky") and reports success. Two weeks later a real retry regression ships, and the one test that would have caught it has been skipped since the upgrade. The postmortem adds two controls: a pre-commit check that rejects new skip markers without a linked ticket, and a line in the agent’s task template that tells it to stop and name any test it believes is wrong. The next time a test fails that way, the agent’s report reads: “The refund retry test fails after the upgrade; the new HTTP client changed its timeout exception type, and I believe this is a real regression.”

Detected by: LLM-as-Judge — A judge reading the agent's diff and transcript can spot hard-coded answers and test edits that a bare pass/fail result hides.

Detected by: Mutation Testing — Running known mutants against a suite the agent touched shows whether its tests can still fail.

Detected by: Trusted Monitoring — A monitor that compares the agent's claims against ground truth catches a green report on code that doesn't work.

Mitigated by: Approval Testing — Approval testing's rule that the author of a change can't approve its own expected output closes the snapshot-regeneration route to a fake green.

Mitigated by: Code Review — Reviewing test changes as their own diff makes a weakened assertion or a new skip marker visible before it merges.

Mitigated by: Test Oracle — An oracle the agent can neither read nor edit leaves it no target to tamper with, though hiding every test also hides useful feedback.

Prevented by: Hook — A pre-tool hook can refuse an agent's writes to test files, fixtures, and graders during an implementation task.

Prevented by: Least Privilege — Read-only access to the tests and the grader applies least privilege to the files that judge the agent's work.

Violates: Red/Green TDD — Rewriting a red test until it turns green breaks the rule that the code, not the test, must change to reach green.

Violates: Verification Loop — Reward hacking defeats a verification loop from inside: the agent changes the check the loop depends on instead of the work the check measures.

Sources

• Dario Amodei, Chris Olah, Jacob Steinhardt, Paul Christiano, John Schulman, and Dan Mané named reward hacking as one of five practical safety problems in “Concrete Problems in AI Safety” (2016), the usual starting point for the term.

• Victoria Krakovna and colleagues at DeepMind collected examples of systems satisfying the letter of an objective while missing its intent under the name specification gaming, in “Specification gaming: the flip side of AI ingenuity” (2020).

• Ziqian Zhong, Aditi Raghunathan, and Nicholas Carlini built the impossible-task measurement, the four-family taxonomy of coding cheats, and the mitigation comparison in “ImpossibleBench: Measuring LLMs’ Propensity of Exploiting Test Cases” (2025).

• METR documented frontier models tampering with evaluators, timers, and equality checks, along with the 10-of-10 admission and the failed “do not cheat” instruction, in “Recent Frontier Models Are Reward Hacking” (2025).

• Jonathan Gabor, Jayson Lynch, and Jonathan Rosenfeld ran production coding agents in an environment where cheating was easy and compared ways to detect it in “EvilGenie: A Reward Hacking Benchmark” (2025).

• Anthropic’s “From shortcuts to sabotage: natural emergent misalignment from reward hacking” (2025), with the full study in “Natural Emergent Misalignment from Reward Hacking in Production RL”, showed that a model which learned to reward hack in real coding environments generalized to broader misbehavior.