Hallucination

Concept: Vocabulary that names a phenomenon.

A hallucination is model output that reads as fluent and confident but isn’t grounded in the task, the project, or the real world; in agents it extends to claims about actions that never happened.

You asked for a function and got one that calls client.upsert_many(). The name matches the library’s style, the arguments look right, and the method doesn’t exist. Anyone who codes with a model meets this early. The word is borrowed from psychiatry, where a hallucination is a perception with no stimulus behind it. Language-generation researchers took it up for summaries that stated things the source document never said. The fit is loose, since the model isn’t perceiving anything, but the term stuck.

Understand This First

• Model — a model’s fluency doesn’t depend on whether it’s right.

• Agent — the agentic forms of hallucination need a model acting in a loop with tools.

What It Is

A hallucination is output that’s plausible and ungrounded. The model isn’t lying, and nothing has malfunctioned. A model produces the most likely continuation of its input, and likely isn’t the same as true. When the input doesn’t contain the answer, the model still produces a confident one, built from patterns that usually hold. A method named upsert_many is likely because libraries like that one often have methods like it.

In code, hallucinations come from three gaps in grounding. Two peer-reviewed studies sorted hallucinations in generated code by hand. Their categories differ in detail, but both put the task and outside knowledge at the top. The study that worked at repository scale adds the project itself:

• The task. The output conflicts with what was asked. The ticket said cursor pagination and the code uses offsets, a stated constraint goes missing, or a requirement appears that nobody stated. This was the largest category in both studies: about 44 percent of the hallucinations Zhang and colleagues found in repository-level generation, and about 40 percent in Liu and colleagues’ study of function-level benchmarks.

• The world. The output names things that don’t exist or don’t work the way the code assumes: a method, a package, a command-line flag, a config key that belonged to the previous major version. Zhang’s team counted about 32 percent here, with wrong API knowledge the largest slice.

• The project. The output contradicts the repository it’s meant to live in: a helper the codebase never defined, a dependency missing from the manifest, a resource file at a path that was never created. That was the remaining quarter in Zhang’s study.

Give the model tools and a loop, and a fourth kind appears: hallucinations about the agent’s own work. The agent acts on a false premise, an unverified assumption about the task or environment such as which test command the repository uses. Or it reports fabricated success, claiming results it never produced. Zhao and colleagues annotated 1,794 failed and successful runs of command-line coding agents. In 57.9 percent of the decisive errors, the information the agent needed was already available and it ignored, forgot, or misread it. Errors from genuinely lacking the knowledge or ability came to about a third. False premise, at 30.7 percent, was the largest single type.

Loud and quiet hallucinations

Hallucinations in code behave differently from hallucinations in prose, and the difference decides where your checks belong. Simon Willison put it bluntly in 2025: “The moment you run LLM generated code, any hallucinated methods will be instantly obvious: you’ll get an error.” An agent that runs its own code sees that error and fixes it. Those are loud hallucinations. They cost a compile error or a stack trace.

Quiet hallucinations are the ones that run. A config loader that ignores unknown keys accepts the invented key and does nothing with it. A function implements a slightly different requirement than the one in the ticket and passes every test, because no test checks the difference. Liu’s data shows that a hallucination usually comes with a failing test: samples with no hallucination passed their tests 54.73 percent of the time, and samples with one passed only 6.26 percent of the time. That’s a correlation, not a law, and the few that pass are the quiet ones. They’re what review exists to catch.

Why It Matters

The folk version, “the AI makes things up,” leaves you with two bad responses. You distrust everything the model writes, which throws away most of its value, or you trust anything that compiles, which lets the quiet ones through. The mechanism view is more useful because each kind of grounding gap has a different check.

Task hallucinations are caught by comparing the output against the requirement, not against the agent’s summary of what it did. World hallucinations are caught by execution, type checks, and lookups: grep, the docs, --help, the package registry. Project hallucinations are best prevented: give the agent the real context first, through Retrieval, a Codebase Map, and Context Engineering, then let the compiler and tests confirm it. Hallucinations about the agent’s own actions yield only to evidence the agent didn’t write, such as the harness’s record of which commands ran and what they returned.

Grounding helps but doesn’t cure. Zhang’s team tried retrieval-augmented generation as a mitigation; it improved results for all six models they tested, and the authors call it a preliminary pilot rather than a fix. Vendor guidance points the same direction. Anthropic’s advice for reducing hallucinations starts with explicitly allowing the model to say it doesn’t know, and, for long documents, having it extract the relevant quotes before answering. Both lower the rate. Neither makes verification optional.

How to Recognize It

AI Smell covers the reviewer’s habits for spotting plausible-but-wrong output, including fabricated references. The signs below are the ones tied to each grounding gap.

• A name you can’t find. Search for the function, flag, or endpoint. If it isn’t there, or a similar but different thing is, you’ve found a world or project hallucination.

• An install that fails with “not found.” That’s a package hallucination. Treat it as a security event, not a typo: Slopsquatting exists because attackers register the names models invent.

• A setting that does nothing. The config change deployed cleanly and behavior didn’t move. Check whether the loader silently ignores unknown keys.

• A diff that answers a different question. Read the change against the ticket’s acceptance criteria. Requirement drift was the largest category in both studies, and it’s the one least likely to break anything visibly.

• A near miss late in a session. The agent names a field that’s almost one of yours. That’s often Context Rot: the right token was in the window and got too little attention.

• A verdict without evidence. The final message says tests pass, but the transcript shows no test command, or one whose output doesn’t match the claim.

Tip: When an agent reports success, ask for the evidence instead of the verdict: the exact command, its exit code, and the last lines of its output. Then check those against the harness’s own log of tool calls, which the model doesn’t write.

How It Plays Out

A developer asks an agent to speed up a slow build. The agent adds --parallel-jobs 8 to the build command, runs it, and gets “unknown option.” It reads the tool’s --help, finds that the flag is --jobs, and fixes the command on the next turn. The whole hallucination cost about 10 seconds. This is the cheap case, and it’s cheap because the agent ran the command instead of handing the developer an untested suggestion.

A team’s service started dropping requests during a database failover. The postmortem traced it to a change an agent had made a month earlier, adding a retry setting to the service’s YAML config. The key name was plausible, matched the style of the file, and came from an older release of the client library. The current loader ignored unknown keys, so the setting had never done anything. Code review passed it because it read correctly, and no test exercised failover. The team switched the loader to strict mode, which rejects unrecognized keys at startup, and three more dead settings surfaced the same day.

An agent is assigned a flaky integration test. It assumes the repository runs tests from the root with one command, though this monorepo runs them per package. The command it chooses collects no tests and exits. The agent edits a timeout, reruns the same command, and ends with “Fixed the flaky test; the full suite passes.” A trusted monitor reviewing the run compares the claim with the tool log, finds no test results anywhere in the output, and sends the work back.

The false premise came first, and the fabricated report followed from it. Zhao’s team found the same order in their data: fabricated success appeared in 26 percent of failed runs, and 84 percent of it began at or after the step where the run had become unrecoverable. By the time an agent invents a result, the run has usually already failed.

Consequences

Benefits. Naming the kind of grounding gap tells you which check to add. Loud hallucinations get handled by the Verification Loop and Feedback Sensors the agent already runs, so human attention can go to the quiet ones: requirement drift, silently ignored settings, and success claims. Teams that make this split stop spending review time catching invented method names that the compiler would have caught anyway.

Liabilities. The word covers a lot. A missed requirement, a stale API, and a fabricated test report need different remedies, and calling them all “hallucination” can hide that. Fabricated success in particular sits next to two antipatterns with other causes: Premature Termination, where the run stops before the work is done, and Reward Hacking, where the agent games the check instead of misreporting it. Sycophancy produces false agreement for a different reason again. The term also invites blaming the model for a context failure. If the agent never saw the API spec, the fix is in the input, not the model.

The rates quoted here describe specific models, benchmarks, and samples, and they’ll move as models change. Transluce’s audit of about 8,600 real coding sessions found severe overselling of results in 1.8 percent of the public sessions it studied and 0.1 percent of its internal ones. The cases included agents that claimed a review had passed before it finished and agents that invented a human’s approval. The percentages are small and come from particular populations, but they describe a behavior that exists in production. Measure your own agent before assuming any rate applies to it.

Depends on: Agent — False premises and fabricated success appear only once a model acts in a loop with tools.

Depends on: Model — Hallucination follows from a property of the model itself: its fluency doesn't depend on being right.

Detected by: Feedback Sensor — Compilers, type checkers, and linters report invented names back to the agent automatically.

Detected by: Formal Verification — A proof checker rejects a hallucinated proof step no matter how convincing it reads.

Detected by: Smell (AI Smell) — AI Smell gives reviewers the recognition habits, and its fabricated-reference smell is the classic hallucination shape.

Detected by: Trusted Monitoring — A monitor that checks claims against ground truth catches fabricated success that the agent's own summary hides.

Detected by: Verification Loop — Running the code and checking the result catches loud hallucinations within one iteration.

Enabled by: Context Rot — Context Rot breeds project-context hallucinations late in long sessions, when the right file is in the window but weakly attended.

Enables: Slopsquatting — Package hallucinations repeat, which gives attackers a list of invented names worth registering.

Informs: Premature Termination — Fabricated success often appears in the final report of a run that stopped short, so recognizing it points to the stop-condition remedies.

Mitigated by: Codebase Map — A codebase map tells the agent which files, symbols, and paths actually exist before it writes code that names them.

Mitigated by: Context Engineering — Choosing what the model sees decides how much of the task and the project it has to infer.

Mitigated by: Retrieval — Retrieval puts the real documentation in front of the model so it has less to guess.

Specializes: Failure Mode — Hallucination is the agent's version of Byzantine output: a confident answer that is wrong.

Sources

• Ziyao Zhang, Yanlin Wang, Chong Wang, Jiachi Chen, and Zibin Zheng, LLM Hallucinations in Practical Code Generation: Phenomena, Mechanism, and Mitigation (ISSTA 2025), built the repository-level taxonomy of task, knowledge, and project-context conflicts and tested retrieval as a mitigation.

• Fang Liu, Yang Liu, Lin Shi, Zhen Yang, Li Zhang, Xiaoli Lian, Zhongqi Li, and Yuchi Ma, Beyond Functional Correctness: Exploring Hallucinations in LLM-Generated Code (Transactions on Software Engineering), classified hallucinations in HumanEval and CoderEval output and measured how they relate to passing tests.

• Simon Willison, Hallucinations in code are the least dangerous form of LLM mistakes (March 2025), argued the asymmetry between hallucinations that fail when run and mistakes that run.

• Xiangxin Zhao, Han Li, Shuaiting Li, Tianyi Zhao, Earl T. Barr, Federica Sarro, and He Ye, Failure as a Process: An Anatomy of CLI Coding Agent Trajectories (2026), measured false premises and fabricated success across seven models and three agent scaffolds on Terminal-Bench.

• Transluce, Measuring coding agent misalignment in the wild (August 2026), audited about 8,600 real coding sessions for agents overselling their results.

• Joshua Maynez, Shashi Narayan, Bernd Bohnet, and Ryan McDonald, On Faithfulness and Factuality in Abstractive Summarization (ACL 2020), studied summarization models that hallucinate content unfaithful to their source, the sense of the word that code generation inherited.

• Ziwei Ji and colleagues, Survey of Hallucination in Natural Language Generation (ACM Computing Surveys), collected the research on hallucination across language-generation tasks before coding agents existed.

Further Reading

• Reduce hallucinations — Anthropic’s short list of prompting techniques, with examples of each.

• Hallucination by Code Generation LLMs: Taxonomy, Benchmarks, Mitigation, and Challenges — a 2025 survey that maps the research on code hallucinations, useful for finding benchmarks.