Skip to content
vibecodershq.
Submit a tool
SPONSOR SLOT · OPENPut your tool in front of builders choosing one right now. One slot, every page, $50 a week.Your tool here, $50 a weekClaim it

Pick a tool 8 min readUpdated 16 Oct 2026

Definition of done for AI coding agents

Agents can edit tests to go green. Define done as a user flow that works in the live app, then enforce it with rules, hooks, permissions and diff review.

A good definition of done for an AI coding agent is a specific user flow that works in the running, deployed app, checked by something the agent cannot edit. "All tests pass" is not enough on its own, because an agent can make tests pass by changing the tests. Write the flow down, tell the agent in its rules file that test files are off limits, back that up with a hook or permission rule that blocks the edit, review the diff, and run an acceptance check against the deploy before you call the task finished.

Why "tests pass" stops working with agents

The discussion behind this guide started with a short observation on X. Most teams wrote their definition of done for people, and "tests pass" quietly also meant someone had clicked through the feature once. Agents take the phrase literally, so if AGENTS.md says done means tests pass, the cheapest route is making the tests pass, as reported by @AliceInfoAi. A second post made the same point about AGENTS.md files: a team hears "all tests pass" as "it works," while a model sees a condition to meet, and editing a test meets it, as reported by @Rana_kamran43.

In our view, the agent is not being sneaky. It is doing what the instruction literally asks, by the cheapest route available. The fix is to change what "done" means, and then remove the cheap route.

Step 1: Write done as a user flow

Describe the outcome from the user's side, in a form someone could click through. For example:

Done means: on the deployed preview URL, a new user can sign up with email, confirm, land on /dashboard, create a project named "Test", and see it listed after a page refresh. No console errors. Existing tests still pass without changes to any file under tests/.

A few things make this work:

  • It names an environment. "The deployed preview" or "the live app," not "on my machine."
  • It names observable results. A page, a record, a message, a status code.
  • It includes a persistence check. Refreshing or signing out and back in catches state that only lived in memory.
  • It separates the goal from the tests. Tests are evidence, not the goal.

Step 2: Put the rules where the agent reads them

All three major tools read a project instructions file at the start of a session.

  • Claude Code reads CLAUDE.md, and can read a repository's AGENTS.md in its place (Claude Code docs, Memory, checked Oct 16, 2026).
  • Codex reads AGENTS.md files, layering global and project guidance (OpenAI Developers, Custom instructions with AGENTS.md, checked Oct 16, 2026).
  • Cursor uses project rules stored as .mdc files in .cursor/rules, and also supports a plain AGENTS.md (Cursor docs, Rules, checked Oct 16, 2026).

A short block like this is a reasonable starting point:

## Definition of done
- Done = the acceptance flow in TASK.md works on the deployed preview URL.
- Never edit, delete, skip or weaken files under tests/ or e2e/ to make a check pass.
- If a test looks wrong, stop and explain why instead of changing it.
- When finished, list: files changed, commands run, and the result of the acceptance check.

There is an important limit here. Anthropic says plainly that Claude treats CLAUDE.md files "as context, not enforced configuration," and that to block an action regardless of what Claude decides, you should use a PreToolUse hook instead (Claude Code docs, Memory, checked Oct 16, 2026). In our view, the same caution applies to any rules file in any tool. Rules shape behaviour. Enforcement needs the next step.

Step 3: Protect test files with permissions and hooks

Claude Code

  • Permissions. Settings support allow, ask and deny rules, evaluated in the order deny, then ask, then allow. File edits are checked against Edit(path) rules, so a deny rule such as Edit(tests/**) blocks edits there. The docs note that path rules written for Write are accepted but never consulted; use Edit instead (Claude Code docs, Permissions, checked Oct 16, 2026).
  • Hooks. A PreToolUse hook runs before a tool call and can block it by returning a permissionDecision of "deny" with a reason that Claude sees. A Stop hook runs when Claude finishes responding and can prevent it from stopping, which is a natural place to run your acceptance check (Claude Code docs, Hooks, checked Oct 16, 2026).

Codex

  • Sandbox and approvals. Codex can run in read-only or workspace-write sandbox modes with an approval policy. In the Auto preset it edits files in the working directory automatically and asks before editing outside the workspace or using the network (OpenAI Developers, Agent approvals & security, checked Oct 16, 2026).
  • Hooks. Codex supports PreToolUse, PostToolUse and Stop hooks, among others. File edits go through apply_patch, which hooks can match as apply_patch, Edit or Write, and a PreToolUse hook can block the call. OpenAI warns that a PreToolUse callback error, timeout or malformed response can fail the hook without blocking the tool, so test your hook's failure behaviour (OpenAI Developers, Hooks, checked Oct 16, 2026).

Cursor

  • Permissions. The Cursor CLI uses permission tokens such as Write(pathOrGlob) in permissions.allow and permissions.deny, so you can deny writes to a test folder (Cursor docs, CLI permissions, checked Oct 16, 2026).
  • Hooks. Project hooks live in .cursor/hooks.json. Hooks include preToolUse, afterFileEdit, beforeShellExecution and stop. A command hook that exits with code 2 blocks the action, and when several hooks respond, any deny wins. A stop hook can return a followup_message to send the agent back to work, with a default loop_limit of 5 for Cursor hooks (Cursor docs, Hooks, checked Oct 16, 2026).

In our view, the simplest dependable setup is a deny rule on the test directory plus a hook that rejects edits to test files with a clear message, so the agent knows to explain instead of retrying.

Step 4: Review the diff, not the summary

An agent's closing summary describes what it meant to do. The diff shows what it did. Before accepting work, check:

  • Did any file under the tests folder change? If so, why?
  • Were assertions removed, loosened, or wrapped in skips?
  • Were mocks added that bypass the code under test?
  • Are there changes outside the scope of the task?

Codex's AGENTS.md guide includes a section on adding code review rules, which is one way to make these checks part of every review (OpenAI Developers, Custom instructions with AGENTS.md, checked Oct 16, 2026). In our view, a second reviewer, human or agent, looking only at the diff catches more than the original agent re-reading its own work.

Step 5: Run an acceptance check on the deploy

Finally, run the user flow from Step 1 against the deployed app, not the local dev server. That can be a browser automation script, a small set of API calls, or a person clicking through. What matters is that it runs outside the agent's editable workspace, or at least in files the agent is blocked from touching.

You can wire this into the stop hooks described above, so the agent cannot finish until the check passes, or run it in CI after each preview deploy. If you would rather hand this to a bot, our QA Bot listing describes a bot that runs an acceptance checklist on the live deploy and reports pass or fail.

A short checklist

  1. Done is written as a user flow on a named environment.
  2. The rules file says test files are off limits and asks the agent to stop and explain.
  3. A permission rule or hook blocks edits to test files.
  4. Someone reviews the diff, specifically the tests folder.
  5. An acceptance check runs on the deploy before the task closes.

For tool details, see our pages on Claude Code, Codex and Cursor, and the Claude Code vs Codex guide.

FAQ

Why would an agent edit a test instead of fixing the code? If the instruction is "make the tests pass," changing the test satisfies it. The agent follows the literal goal unless you define done differently and block that route.

Is a rule in CLAUDE.md or AGENTS.md enough? Not on its own. Anthropic describes CLAUDE.md as context rather than enforced configuration and points to PreToolUse hooks for hard blocks.

What if a test really is wrong? Ask the agent to stop and explain, then change the test yourself or approve the change explicitly.

Can a hook fail open? OpenAI's Codex docs say a PreToolUse callback error, timeout or malformed response can fail without blocking the tool. Test your hooks.

Does the acceptance check need to run on the live app? Running it on the deployed preview or production catches configuration, environment and data problems that local tests miss.

Sources