--- slug: pass-k type: concept summary: "The probability that an agent succeeds on every one of k repeated attempts, exposing reliability that a single-run pass rate can hide." created: 2026-10-22 updated: 2026-10-23 related: bounded-autonomy: relation: informs note: "Repeated-run reliability helps determine how much authority an agent has earned." determinism: relation: depends-on note: "Pass^k measures the practical effect of nondeterministic behavior across repeated runs." eval: relation: depends-on note: "An eval supplies the tasks, success criteria, and repeated trials from which Pass^k is computed." evaluation-gate: relation: informs note: "A gate needs a reliability estimate rather than one lucky passing run." flaky-test-quarantine: relation: related note: "Both distinguish a dependable signal from one that passes or fails intermittently." jagged-frontier: relation: measures note: "Repeated trials reveal task regions where apparent capability is inconsistent." llm-as-judge: relation: related note: "Judge variability can add measurement noise to the agent variability Pass^k is meant to expose." metric: relation: specializes note: "Pass^k is a metric for consistency across repeated attempts." regression: relation: informs note: "Paired repeated runs make reliability regressions easier to separate from sampling noise." --- # Pass^k > **Concept:** Vocabulary that names a phenomenon. *Pass^k measures the chance that an agent succeeds every time across k repeated attempts, making inconsistency visible when a single-run pass rate looks reassuring.* If you've run the same agent on the same task five times, watched it pass four, and wondered which result represents the agent, you've met the problem this entry names. No single run represents it. Reliability is a property of the whole set of runs. The name is notation: the caret marks an exponent, because if one run succeeds with probability p, k independent runs all succeed with probability p to the power k. ## Understand This First - [Eval](eval.md) — supplies the tasks and success criteria. - [Determinism](determinism.md) — explains why identical-looking runs can produce different outcomes. - [Metric](metric.md) — turns an operational question into a defined measurement. ## What It Is **Pass^k** is the probability that all k attempts succeed. Its counterpart, **pass@k**, asks whether at least one of k attempts succeeds; it's the older score, popularized by code-generation benchmarks such as HumanEval. The distinction is small on the page and large in production: - pass@k fits search problems, where you can generate several candidates and keep one winner. - Pass^k fits repeated service, where every run reaches a user, a repository, or another system and any failure counts. Suppose an agent has a 90 percent chance of succeeding on one task. If runs are independent and that success probability stays fixed, its chance of succeeding on 10 consecutive runs is `0.9^10`, about 35 percent. Its pass@10 is almost 100 percent because only one attempt needs to work. The same agent looks excellent under one metric and unreliable under the other. Neither is wrong; they answer different operational questions. The simple `p^k` calculation is an approximation with assumptions. Runs may share a hidden cause, such as a fragile tool, a bad fixture, or one difficult task. Tasks also differ. A benchmark can avoid pretending that every task has one common success probability by running each task k times, scoring it 1 only when every run succeeds, then averaging those all-pass scores across tasks. That per-task average is how tau-bench, the benchmark that introduced the metric, reports it. The two scores move in opposite directions. Pass^k gets smaller as k grows because the standard gets stricter: every run must pass. Pass@k gets larger as k grows because more attempts create more chances for one success. ## Why It Matters A headline success rate hides the run sequence a user actually experiences. Ninety successes in 100 trials sounds strong. A user who depends on the agent every working day may care more about whether it can complete 20 runs without one failure. Under the simplifying independence assumption, 90 percent per-run success yields only about a 12 percent chance of 20 consecutive successes. Small samples hide uncertainty too. Ten successes in 10 trials don't establish perfect reliability. With a 95 percent Wilson interval, 10 out of 10 puts the lower bound for the per-run success probability near 72 percent. Different interval methods give somewhat different bounds, but they agree on the lesson: a clean short run is weak evidence for a demanding reliability claim. That distinction changes governance decisions. An agent that gets one of five attempts right may be useful inside a [Generator-Evaluator](generator-evaluator.md) loop, where a checker picks the good attempt. It isn't ready to send five customer replies without review. An agent with strong Pass^k on a representative suite may earn a wider [Bounded Autonomy](bounded-autonomy.md) tier. The metric doesn't decide the tier by itself, but it gives the decision evidence instead of optimism. ## How to Recognize It Pass^k is the right question when all of these are true: - **The action repeats.** The agent will triage many tickets, repair many builds, or handle many sessions. - **Every attempt counts.** You can't discard failures and show only the best candidate. - **The success criterion is stable.** Each run can be scored against the same rubric or executable check. - **Variation is material.** Model sampling, tool responses, simulated users, or an [LLM-as-Judge](llm-as-judge.md) can change the outcome. Measure it with a protocol, not a pile of convenient reruns: 1. Choose representative tasks and freeze their inputs, environment, tools, and success criteria. 2. Run each task k times. Record every attempt, including timeouts and infrastructure failures that a real user would experience. 3. Give a task an all-pass score only if all k attempts succeed. Average those scores across the task set. 4. Report k, the task count, the run conditions, and an uncertainty interval beside the result. 5. When comparing two agent versions, run both on the same tasks under the same conditions. Paired results expose which tasks changed and waste less information than two unrelated headline rates. Don't choose k after seeing the results. A team that stops at three clean runs and calls the agent reliable has measured its patience, not the agent. Pick k from the consequence of failure and the expected run frequency. A weekly internal helper and a customer-facing agent executing thousands of times per day need different evidence. > **⚠️ Warning:** Pass^k assumes a defined population of tasks and conditions. Changing the task mix, tool versions, judge, or environment changes what the number describes. Keep the evaluation identity beside the score. ## How It Plays Out A support team evaluates a ticket-triage agent on 100 representative tickets and gets a 90 percent single-run success rate, which looks ready for unattended use. The team then runs each ticket 10 times. Sixty tickets pass all 10 runs. The other 40, mostly ambiguous routing cases, succeed about three times in four, and almost none of them survives 10 runs in a row. Pass^10 lands near 62 percent. That's well below the headline and well above the 35 percent that `0.9^10` predicts, because the failures cluster in 40 tickets instead of spreading across all 100. The agent still saves time, so the team lets it route the clear-cut categories on its own and draft the ambiguous ones for human approval. Reliability, not average capability, sets the autonomy boundary. A coding team compares two models for CI repair. Model B passes five of five trial runs on this week's failed builds; Model A passed four of five on last week's. The one-off comparison favors B. The team reruns both models on the same builds and records paired outcomes. B's lead disappears: its batch happened to hold easy repairs, while A is more consistent on the dependency and configuration failures that dominate production. The team keeps A and adds more cases before the next comparison. A platform team wants its dependency-update agent to merge patch-level version bumps without review. The agent has passed 10 of 10 trial runs, and someone proposes switching review off. A reviewer points out that 10 clean runs put the 95 percent lower bound only near 72 percent per run. The agent opens about 40 bumps a month, so that floor allows about 11 bad merges a month. The team runs 50 more trials across a wider sample of repositories. Two fail, both in repositories with hand-edited lockfiles, so unreviewed merges go only to repositories without them. ## Consequences **Benefits.** Pass^k distinguishes occasional capability from dependable service. It catches the agent that can solve a task but can't solve it consistently, gives [evals](eval.md) a metric matched to repeated use, and supplies evidence for approval and autonomy policies. An [evaluation gate](evaluation-gate.md) that blocks on one sampled run punishes authors for sampling noise; one backed by repeated-run evidence blocks when the failure is likely to recur. Reporting uncertainty also makes small-sample claims harder to oversell. **Liabilities.** Repeated trials cost time and money. Results depend on the task set, k, environment, scoring method, and correlation between failures. A single Pass^k number can still hide a [jagged frontier](jagged-frontier.md): some tasks may be perfectly reliable while a small class fails often. Keep per-task results and failure categories behind the aggregate. The metric can also reward triviality. Narrowing the task distribution or weakening the success criterion raises Pass^k without improving the system users depend on. Treat the protocol as part of the metric. If the protocol changes, start a new baseline rather than drawing a clean trend line across incompatible numbers. Pass^k doesn't replace pass@k or per-run success rate. Use pass@k when selection among candidates is part of the system. Use per-run rate when the expected cost of each attempt is the question. Use Pass^k when the promise is consistency across a run of work. Mature evaluations often report all three because they describe different product shapes. ## Sources - Shunyu Yao, Noah Shinn, Pedram Razavi, and Karthik Narasimhan introduced Pass^k for repeated tool-agent-user interaction in [*tau-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains*](https://arxiv.org/abs/2406.12045) (2024). - Mikaela Grace, Jeremy Hadfield, Rodrigo Olivares, and Jiri De Jonghe contrast pass@k with Pass^k in Anthropic's [*Demystifying evals for AI agents*](https://www.anthropic.com/engineering/demystifying-evals-for-ai-agents) (2026), matching the metric to whether one success or consistent success matters. - Gonzalo Gonzalez-Pumariega, Saaket Agashe, Jiachen Yang, Ang Li, and Xin Eric Wang study computer-use agents under repeated trials in [*On the Reliability of Computer Use Agents*](https://arxiv.org/abs/2604.17849) (2026). - Evan Miller develops confidence intervals, comparison methods, and experiment-sizing guidance for model evaluations in [*Adding Error Bars to Evals: A Statistical Approach to Language Model Evaluations*](https://arxiv.org/abs/2411.00640) (2024). --- - [Next: Evaluation Gate](evaluation-gate.md) - [Previous: Eval](eval.md)