--- slug: agent-readiness-assessment type: pattern summary: "Measure how well a repository supports a defined agent workflow, fix the limiting condition, and repeat the same assessment." created: 2026-11-01 updated: 2026-11-01 related: harnessability: relation: measures note: "The assessment turns harnessability from a broad property into evidence about a particular workflow." harness-engineering: relation: informs note: "Assessment findings identify which harness changes are worth making first." verification-loop: relation: uses note: "A working verification loop is one of the capabilities the assessment exercises directly." instruction-file: relation: uses note: "The assessment checks whether repository instructions are discoverable, accurate, and useful in practice." evaluation-gate: relation: informs note: "Repeated assessment findings can become durable evaluation gates once their checks are stable." agentops: relation: uses note: "Task traces and outcome data help test whether readiness findings predict agent performance." --- # Agent Readiness Assessment > **Pattern:** A named solution to a recurring problem. *Measure how well a repository supports a defined agent workflow, fix the limiting condition, and repeat the same assessment.* *Also known as: Agentic Readiness Review, Repository Readiness Baseline* Repository readiness becomes visible when an agent tries to work. A setup command exists but fails on a clean machine. The advertised check passes while skipping integration tests. A permissions policy exists, but the agent can't find it. An assessment turns those surprises into a repeatable diagnosis instead of a vague judgment about whether the repository looks prepared. ## Understand This First - [Harnessability](harnessability.md) — the codebase quality that the assessment examines. - [Harness Engineering](harness-engineering.md) — the configuration and tooling work that often follows the assessment. ## Context A team wants agents to perform real work in an existing repository. Tests, continuous integration, contribution instructions, linters, and architectural documentation are useful signals, but their presence doesn't show that an agent can discover the right commands, run them from a clean checkout, understand a failure, or complete a representative task without unsafe guesswork. Readiness is always readiness for something. A repository can be ready for small dependency updates but not database migrations. It can support supervised code changes while remaining unsuitable for unattended releases. The assessment therefore starts with a bounded workflow and a set of representative tasks, not with a universal score. ## Problem How can a team decide what prevents reliable agent work, without confusing files and tool names with working capabilities? Simple checklists reward visible artifacts. A repository gets credit because it contains a CI file, a test directory, or an instruction document. Yet the CI job can be stale, the tests can depend on an undocumented service, and the instructions can name a command that no longer exists. This proxy can do more than overstate readiness. It can invert the ranking: a repository with a broken but elaborate control surface scores above a simpler repository whose one documented command works. An unbounded trial has the opposite weakness. Asking an agent to "try some tasks" produces anecdotes that can't be compared after remediation. If the task, agent, model, or starting state changes between trials, an apparent improvement may have nothing to do with the repository change. ## Forces - **Coverage versus cost.** A broad assessment may find more problems, but an expensive assessment won't be repeated often enough to guide improvement. - **Standardization versus local truth.** Common dimensions make assessments comparable, but the target workflow determines which dimensions matter and how much. - **Presence versus operation.** File presence is cheap to detect. Executing a capability yields stronger evidence but takes time and may require a controlled environment. - **Scores versus diagnosis.** A single number is easy to report. It can hide the one failed capability that blocks the whole workflow. - **Stable comparison versus changing systems.** Repeating the same trial supports causal comparison, while agents, models, dependencies, and tasks continue to change. ## Solution Run a repeatable assessment cycle around a defined agent workflow. Gather evidence at three levels: repository claims, direct capability checks, and benchmark-task outcomes. Prefer executable evidence over presence checks, identify the limiting condition, repair it, and rerun the same checks and tasks. ### 1. Define the workflow and benchmark tasks State the work the repository is expected to support. Include the allowed authority and the completion test. "Agent-ready" is too vague; "an agent can make a bounded application change, run the required checks, and prepare a reviewable patch without external setup knowledge" is testable. Choose a small benchmark set that samples the actual work. Include at least one ordinary change and one task that crosses a meaningful boundary, such as a schema change, interface update, or test-fixture revision. Keep the task descriptions, starting revisions, and success criteria so the set can be rerun. ### 2. Inventory candidate capabilities Inspect setup instructions, task entry points, tests, static checks, architectural guidance, permission boundaries, and observability. Treat this inventory as a map of claims, not proof that the capabilities work. A file-presence check is useful for locating the next direct check. It isn't a readiness result. Label every presence-based result as a proxy so it can't quietly acquire the authority of an executed test. ### 3. Exercise capabilities directly Start from a clean environment and follow only the information available to the agent. Run setup, discover the prescribed task commands, execute validation, and inspect the resulting diagnostics. Record whether each capability is absent, present but untested, verified once, or verified repeatedly. Use a table like this to keep the evidence types distinct: | Dimension | Direct check | Weak proxy | Outcome signal | | --- | --- | --- | --- | | Environment setup | A clean checkout reaches a working development state from documented steps | A setup script exists | Benchmark tasks start without operator repair | | Task discovery | The agent finds the correct build, test, and formatting commands from repository guidance | A README or instruction file exists | The agent chooses the correct commands without guessing | | Validation | The documented check detects an injected or known failure and returns a failing status | Test and CI directories exist | Defects are caught before handoff | | Architecture | A representative change can be placed using current boundaries and examples | Architecture documents exist | The patch touches the expected modules and avoids unrelated churn | | Authority | A disallowed action is refused while permitted work remains possible | A permissions policy exists | The task completes without unauthorized access or needless escalation | | Observability | Tool failures expose an actionable cause and preserve diagnostic output | Log configuration exists | The agent recovers from ordinary failures without operator interpretation | The direct check is the primary evidence. The outcome signal tests whether the capability matters during actual work. The proxy is a lead for investigation, not a substitute for either. ### 4. Run representative agent tasks Run the benchmark tasks under a recorded agent and harness configuration. Capture task completion, validation results, retries, operator interventions, and unsafe attempts. Record elapsed time and cost when decisions depend on them. A score that rises while task failures remain unchanged hasn't demonstrated improvement. When repeated trials are affordable, use them. One successful run can be luck, especially when the workflow includes nondeterministic model behavior. Keep repository failures separate from agent failures: an undocumented dependency is a repository-readiness finding, while a model misreading clear instructions may not be. ### 5. Find the limiting condition Summarize each dimension with its evidence and confidence. A small local scale can help: - **0:** absent or unusable for the target workflow - **1:** present but not exercised - **2:** verified in the current assessment - **3:** repeatedly verified and associated with successful benchmark outcomes This is an assessment scale, not a maturity model. Don't average away a blocking zero. If every task depends on setup and setup fails, improving documentation style or adding another linter won't make the workflow ready. Prioritize remediation by the failures a finding explains, the number of tasks it affects, the cost and reversibility of the repair, and whether another improvement depends on it. Start with the smallest change that removes the current constraint and makes later evidence easier to collect. ### 6. Repair and remeasure Make one coherent remediation, then rerun the same direct checks and benchmark tasks. Hold the repository starting point, task definitions, agent, model, harness, and authority constant where practical. Record any change you couldn't hold constant, because it limits the comparison. Keep both the readiness evidence and the task outcomes. If a direct check improves but task outcomes don't, the assessed capability may not be the limiting one. If outcomes improve while the score doesn't, the rubric may be missing an important capability. Once a check is stable and consequential, move it into an [Evaluation Gate](evaluation-gate.md) or another automated control. The periodic assessment can then concentrate on capabilities that still require judgment or representative task execution. ## How It Plays Out Consider a Python service whose repository appears well prepared. It has a setup guide, a project configuration file, a CI workflow, unit tests, and a command named `make check`. A presence checklist gives it high marks. The direct assessment starts from a clean checkout. Setup stops because a native library isn't named in the guide. After an operator supplies it, `make check` passes, but inspection shows that the command skips integration tests. In six benchmark tasks, two agents produce patches that pass the advertised check but fail when a database migration is applied. The assessment records setup and validation as the limiting conditions, despite the repository's abundant readiness artifacts. The team adds a preflight that reports the missing library, updates the setup guide, and makes `make check` start the integration service and run the migration tests. It then repeats the same six tasks with the same configuration. Clean setup succeeds, the advertised check catches a deliberately invalid migration, and the migration tasks no longer reach handoff with that defect. The result isn't proof that the repository is ready for every agent workflow. It is evidence that a specific constraint was removed for the assessed workflow. The next cycle can now investigate the remaining failures instead of polishing the old proxy score. ## Consequences An agent readiness assessment gives a team a defensible baseline and a ranked repair list. It makes hidden setup knowledge, misleading validation commands, weak diagnostics, and unclear authority visible before they become repeated task failures. The pattern also creates maintenance work. Benchmark tasks need care, controlled runs consume time, and changing agent configurations complicate comparisons. A rigid rubric can become another proxy if the target workflow changes while the checks do not. The assessment is most useful as a loop, not a certification. Its claim should stay narrow: for this workflow, under these conditions, these capabilities were exercised, this condition limited performance, and this repair changed the measured result. ## Related Articles {{#related}} --- - [Next: Bounded Autonomy](bounded-autonomy.md) - [Previous: Harnessability](harnessability.md)